JP7654949B2 - Fundus image processing program and fundus photographing device - Google Patents

Description

The present disclosure relates to a fundus image processing program for processing a fundus image of a test eye, and a fundus imaging device.

Non-mydriatic fundus cameras have been widely used as fundus imaging devices. Most non-mydriatic fundus cameras capture fundus images with an angle of view of about 45°. As disclosed in Patent Document 1, a method is also known in which a wide range of the fundus is represented in a single panoramic image by panoramic synthesis of multiple fundus images with different fixation positions.

On the other hand, in recent years, devices that can capture wider-angle fundus images have been attracting attention. For example, Patent Document 1 discloses a fundus imaging device that captures color fundus images with a viewing angle of 90 degrees or more.

JP 2016-123456 A

Fundus images taken with a fundus photography device usually contain distortion. This distortion becomes greater toward the periphery of the image. Also, the wider the angle of view, the more noticeable the distortion becomes on the periphery.

The distortion of the fundus image is caused by both the optical system of the device and the shape of the fundus. Of these, a correction formula for the distortion caused by the optical system of the device can be obtained in advance by photographing a chart or the like with a known shape. However, because the optical system of the eye varies from person to person, at least the distortion caused by the fundus shape cannot be corrected in the same way as the distortion caused by the optical system of the device.

In addition, when synthesizing fundus images into a panorama with a field angle of about 45°, distortion caused by the fundus shape of the subject eye was not considered to be a particular problem.

The present disclosure has been made in response to problems in the prior art, and aims to provide a fundus image processing program and a fundus photography device that can obtain fundus images in which distortion caused by the fundus shape is effectively suppressed.

A fundus image processing program according to a first aspect of the present disclosure is a fundus image processing program executed by a computer that processes data of a fundus image obtained by photographing the fundus of a test eye, and when the fundus image processing program is executed by a processor of the computer, the following steps are executed by the computer: an acquisition step of acquiring two or more fundus images photographed at different photographing positions via an imaging optical system, the two or more fundus images partially overlapping each other; a correction step of correcting distortion of some or all of the images contained in the two or more fundus images based on image information of the two or more fundus images partially overlapping each other; and a synthesis step of generating a panoramic fundus image by synthesizing the two or more fundus images whose distortions have been corrected; and further, the correction step includes a correspondence point setting step of setting a plurality of corresponding points between the two or more fundus images partially overlapping each other, and a coordinate transformation step of calculating a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points, and applying the coordinate transformation to the fundus images in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

A fundus image processing program according to a second aspect of the present disclosure is a fundus image processing program executed by a computer that processes data of a fundus image obtained by photographing the fundus of a test eye, and when the fundus image processing program is executed by a processor of the computer, an acquisition step of acquiring two or more fundus images photographed via an imaging optical system at different shooting positions, the two or more fundus images partially overlapping each other, and a correction step of correcting distortion of some or all of the images contained in the two or more fundus images based on image information of the two or more fundus images partially overlapping each other , and further, the correction step includes a correspondence point setting step of setting a plurality of corresponding points between the two or more fundus images partially overlapping each other, and a coordinate transformation step of calculating a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points, and applying the coordinate transformation to the fundus images in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

The fundus imaging device according to the third aspect of the present disclosure includes the imaging optical system and an image processing device that executes the fundus image processing program according to the first or second aspect.

According to the present disclosure, it is possible to obtain a fundus image in which distortion caused by the fundus shape is effectively suppressed.

1 is a diagram showing a schematic configuration of a fundus imaging device 1. FIG. FIG. 2 is a diagram showing a schematic configuration of a photographing optical system 2. FIG. 4 is a diagram showing the processing contents of each step of fundus image processing.

"overview"
First, a first embodiment that is an exemplary embodiment of the present disclosure will be described.

In the first embodiment, a method for generating a panoramic fundus image from multiple fundus images is described. In the first embodiment, data of the fundus image of the subject eye is processed by a computer. The fundus image processing program according to the first embodiment is executed by the processor of the computer, thereby performing an acquisition step, a correction step, and a synthesis step.

<Acquisition steps>
In the acquisition step, two or more fundus images are acquired via an imaging optical system. The two or more acquired fundus images partially overlap each other. The two or more fundus images may be acquired at different fixation positions, resulting in two or more fundus images partially overlapping each other. Each fundus image is stored in a memory accessible by the processor.

The fundus image acquired in the acquisition step may be a front image (two-dimensional image) of the fundus. However, this is not necessarily limited to this, and the fundus image may be a three-dimensional image (volume data) in which depth information is added to the front image (two-dimensional image).

Each fundus image may be captured at an angle of view greater than 45°, which is the angle of view of a standard fundus camera.

<Correction step>
In the correction step, based on the image information of two or more fundus images that overlap each other, distortion of some or all of the images included in the two or more fundus images is corrected. That is, according to this embodiment, image distortion is appropriately corrected without necessarily requiring information on the optical system of the device and information on the fundus shape other than the fundus image. That is, the influence of not only the field curvature caused by the photographing optical system but also the distortion that differs for each subject eye caused by the curved shape of the fundus is suppressed.

The correction step may further include a corresponding point setting step and a coordinate conversion step, as follows:

<Corresponding point setting step>
In the corresponding point setting step, a plurality of corresponding points are set between two or more fundus images. In other words, a plurality of corresponding points are set in overlapping portions between two or more fundus images. The corresponding points may be set at positions of features common to the two or more fundus images. In the fundus images, the corresponding points may be set at structural features of tissues, such as branching of blood vessels.

Here, the method of setting multiple corresponding points between two or more fundus images can be selected as appropriate. For example, the processor detects multiple points (feature points) on the tissue for finding corresponding points from within the image area of one of the two or more fundus images (referred to as the "target image"). For detecting the feature points, for example, points suitable for alignment may be detected using the good features to track function of opencv, or they may be set at equal intervals. Next, from within the image area of an image other than the target image (referred to as the "source image") among the two or more fundus images, corresponding points that correspond to the feature points of the target image are detected. For example, template matching, block matching, etc. can be used to detect the corresponding points of the source image.

<Coordinate conversion step>
Next, the coordinate conversion step will be described. For convenience, the fundus image to be processed in the coordinate conversion step may be referred to as an input image, and the processing result may be referred to as an output image.

In the coordinate transformation step, the processor determines a coordinate transformation that matches the positional relationships of multiple corresponding points between multiple fundus images, each of which has multiple corresponding points set. For example, in the first embodiment, the processor determines a coordinate transformation (coordinate transformation formula) that minimizes the distance between multiple corresponding points in two or more input images under constraints that take into account the field curvature of the imaging optical system and the curvature of the fundus. The coordinate transformation (coordinate transformation formula) may be expressed by adding together a distortion correction model and a parallel translation (see the embodiments for a specific example).

Then, the calculated coordinate transformation is applied to two or more input images. In other words, the transformed coordinates for each pixel of the input image are calculated based on the coordinate transformation formula obtained under the constraints. As a result, an output image is obtained by providing the pixel values of each pixel of the input image to the transformed coordinates. At this time, the application of the coordinate transformation causes each input image to be deformed and moved. As a result, distortion is corrected in each fundus image, and alignment is also performed.

In this embodiment, the constraint condition may be determined in advance regardless of the actual degree of curvature of the field and the curvature of the fundus. Here, the image center of each fundus image is considered to have less distortion caused by the curvature of the fundus than the image peripheral portion. In addition, the image center has less distortion caused by the field curvature of the photographing optical system than the image peripheral portion. Therefore, for each fundus image to be corrected for distortion, the constraint condition may be that "the amount of coordinate change in the image center is suppressed more than the amount of coordinate change in the image peripheral portion." More preferably, the constraint condition may be that "there is no change in coordinates in the image center due to the distortion correction model." The image center may be an area in the center of each fundus image, whose position, shape, and size are determined in advance.

In addition, in a fundus image captured with the visual axis tilted relative to the optical axis of the imaging optical system, it is considered that the region in which distortion caused by the curvature of the fundus is the least displaces from the center of the image according to the amount of tilt between the optical axis and the visual axis. For this reason, within the range of field angle and tilt angle in which the distortion in the center of the image is sufficiently small compared to the distortion in the peripheral areas of the image, the constraint that "the center of the image does not change in coordinates due to the distortion correction model" may be used. Alternatively, the image region in which "there is no change in coordinates due to the distortion correction model" is the constraint may be changed for each image according to the field angle and tilt angle.

<Synthesis Steps>
In the synthesis step, a panoramic fundus image is generated by synthesizing (combining) two or more fundus images in which the distortion has been corrected. If the coordinate transformation step has been performed in advance, two or more fundus images to which the coordinate transformation has been applied are synthesized. The pixel value of each pixel may be set by averaging or weighting the overlapping portion. In the case of weighting the average, the closer to the edge of the fundus region in each fundus image, the smaller the weighting coefficient is set, so that the seam between the images is less noticeable. As a result of the synthesis, a panoramic fundus image having a wider angle than the fundus image on which the synthesis is based can be obtained. Although the fundus image on which the synthesis is based has a larger distortion in the peripheral portion of the image, the distortion is appropriately corrected by the correction step, so that the discontinuous connection of the fundus structure at the seam between the images in the panoramic fundus image is appropriately reduced. That is, the application of coordinate transformation that makes the positional relationship of the multiple corresponding points coincide with each other between the multiple fundus images in which the multiple corresponding points are set, suppresses discontinuity at the seam between the images. In addition, distortion is corrected by coordinate transformation that minimizes the distance between multiple corresponding points in two or more input images under constraints that take into account the field curvature of the photographing optical system and the curvature of the fundus, so that a panoramic fundus image that makes it easy to observe peripheral tissue can be obtained.

Second Embodiment
The coordinate transformation in the correction step of the first embodiment is premised on obtaining a panoramic fundus image. If it is only necessary to correct the distortion of a part or all of the images included in the fundus image, it is not necessarily required to obtain a coordinate transformation (coordinate transformation formula) that "minimizes" the distance between multiple corresponding points in two or more input images. If it is only necessary to correct the distortion of a part or all of the images included in the fundus image, it is considered sufficient to obtain a coordinate transformation (coordinate transformation formula) that minimizes the variation in the distance between multiple corresponding points in two or more input images, and apply the obtained coordinate transformation to a part or all of the two or more fundus images in which corresponding points are set.

Even in the case where a panoramic fundus image is not necessarily generated and distortion correction is performed in this way, some or all of the descriptions in <> in the first embodiment can be appropriately applied.
<Example>
An example will be described below. In the embodiment, the fundus photographing device 1 itself photographs the fundus image and processes the photographed fundus image. That is, the fundus photographing device 1 of this embodiment functions as a computer that processes the fundus image. Therefore, the fundus photographing device 1 can process the photographed fundus image while photographing the fundus image. However, the device that can function as a computer is not limited to the fundus photographing device 1. For example, a personal computer, a server, a mobile terminal, or a smartphone that can acquire data of the fundus image photographed by the fundus photographing device 1 may function as a computer that performs the fundus image processing described below. In addition, the control units of multiple devices may cooperate to perform the fundus image processing.

The schematic configuration of the fundus photographing device 1 will be described with reference to Figs. 1 and 2. As shown in Fig. 1, the fundus photographing device 1 includes an optical unit 1A and a control unit 1B. The optical unit 1A stores a photographing optical system 2 (see Fig. 2). The control unit 1B includes a processor (CPU) 70 and a storage device (memory) 75. The processor 70 is a control unit that controls the fundus photographing device 1. The storage device 75 stores a fundus image processing program. The control unit 1B is connected to an operation unit 85 and a monitor 90. The operation unit 85 may be a pointing device such as a mouse or a touch panel, or may be another user interface. The monitor 90 can display various images such as a photographed fundus image. A personal computer (PC) or the like may be used as the control unit 1B. In addition, the optical unit 1A and the control unit 1B may be housed in the same housing or in separate housings.

The photographing optical system 2 will be described with reference to FIG. 2. The photographing optical system 2 illustrated in FIG. 2 is a scanning type optical system. In detail, the photographing optical system 2 in this embodiment is a two-dimensional scanning type optical system. In a two-dimensional scanning type optical system, the illumination light is formed in a spot shape on the fundus and is scanned two-dimensionally on the fundus. However, a line scanning type (or slit scanning type) optical system may be adopted as the scanning type optical system. In this case, the illumination light is formed in a line shape or a slit shape on the fundus and is scanned in a direction intersecting the cross-sectional longitudinal direction of the light beam of the illumination light. In addition, the photographing optical system does not have to be a scanning type optical system. For example, the photographing optical system may capture a two-dimensional front image of the fundus by simultaneously irradiating the illumination light to a two-dimensional area on the fundus. The photographing optical system 2 includes an illumination optical system 10 and a light receiving optical system 20.

The following describes the irradiation optical system 10. The irradiation optical system 10 includes a light source 11, a beam splitter 13, a lens 14, a scanning unit 16, and an objective lens 17.

The irradiation optical system 10 of this embodiment scans the illumination light focused into a point shape on the fundus by the scanning unit 16. The scanning unit 16 scans the illumination light in two different directions. For convenience of explanation, the following will exemplify a case in which the illumination light is scanned by raster scanning. In this case, the scanning unit 16 includes a first optical scanner 16A for main scanning and a second optical scanner 16B for sub-scanning. However, it is also possible to change the configuration of the scanning unit 16. For example, the scanning unit may be a two-degree-of-freedom device (e.g., MEMS, etc.). The optical scanner may be at least one of a galvanometer mirror, a polygon mirror, a resonant scanner, a MEMS, and an acousto-optical element (AOM), etc., as appropriate.

The irradiation optical system 10 irradiates the illumination light emitted by the light source 11 onto the fundus Er of the subject eye E. The irradiation optical system 10 may be capable of simultaneously irradiating light in multiple wavelength ranges (multiple illumination lights having different wavelengths), or may be capable of selectively switching between light in multiple wavelength ranges to irradiate. The irradiation optical system 10 of this embodiment is capable of switching between visible light and invisible light (infrared light in this embodiment) to irradiate the fundus Er. It is also capable of simultaneously irradiating the fundus Er with visible light in multiple wavelength ranges.

The beam splitter 13 combines and separates the optical path of the irradiation optical system 10 and the optical path of the light receiving optical system 20. The beam splitter 13 may be a perforated mirror or another member. The lens 14 is a focusing lens, and is displaced by a drive unit (not shown). The focus (visual acuity) is adjusted by displacing the lens 14 along the optical axis.

The objective lens 17 emits illumination light (laser light in this embodiment) scanned by the scanning unit 16 toward the fundus Er of the subject's eye E. The objective lens 17 forms a rotation point P at the position of the exit pupil. At the rotation point P, the illumination light that has passed through the scanning unit 16 is rotated. As a result, the illumination light is scanned on the fundus Er. When multiple illumination lights with different wavelengths are emitted simultaneously, the multiple illumination lights are irradiated onto the fundus Er along the same optical path that passes through the pupil of the subject's eye E. Fundus reflection light of the illumination light is emitted from the pupil as parallel light.

The light receiving optical system 20 will now be described. In this embodiment, the light receiving optical system 20 shares with the illumination optical system 10 each component arranged on the optical path from the objective lens 17 to the beam splitter 13. Light from the fundus Er (e.g., fundus reflection of illumination light, or fluorescence, etc.) is guided back along the optical path of the illumination optical system 10 to the beam splitter 13. The beam splitter 13 guides the light from the fundus Er to an independent optical path of the light receiving optical system 20.

The independent optical path of the light receiving optical system 20 includes a lens 21, a pinhole plate 23, and a light separation unit 30. The pinhole plate 23 is disposed on the fundus conjugate plane, and removes light from positions other than the focal point or focal plane of the fundus Er, and guides light from the focal point or focal plane to the light separation unit 30. The light separation unit 30 separates light from the fundus Er. In this embodiment, the light separation unit 30 selectively separates light from the fundus Er in a wavelength-selective manner. That is, the light separation unit 30 branches the optical path of the light receiving optical system 20 according to the wavelength of light. As shown in FIG. 2, the light separation unit 30 of this embodiment may include multiple dichroic mirrors, including two dichroic mirrors 31 and 32 having different optical separation characteristics (wavelength separation characteristics). The optical path of the light receiving optical system 20 is branched into three or more optical paths by multiple dichroic mirrors. One of the light receiving elements 25, 27, and 29 is disposed on each of the multiple branched optical paths.

The light separation unit 30 in this embodiment separates the light from the fundus Er by wavelength, and causes each of the three light receiving elements 25, 27, and 29 to receive light in a different wavelength range. In detail, each of the three light receiving elements 25, 27, and 29 receives one of the three colors of light: blue, green, and red. Therefore, the fundus imaging device 1 can simultaneously capture multiple (three in this embodiment) front fundus images of the fundus Er captured with illumination light of each of the blue, green, and red wavelengths. It is also possible to generate a color front fundus image from multiple front fundus images captured simultaneously. Note that a front fundus image is a two-dimensional image of the fundus viewed from the optical axis direction of the illumination light.

As described above, the irradiation optical system 10 of this embodiment is capable of switching between visible light and infrared light to irradiate the fundus Er. The fundus imaging device 1 can capture an infrared image (two-dimensional front image) of the fundus Er by irradiating the fundus Er with infrared light and receiving the infrared light reflected from the fundus Er with a light receiving element. Since the subject does not feel glare while the infrared image is being captured, there is little possibility that the quality of the infrared image will be degraded by blinking, etc.

The fundus imaging device 1 further has a fixation presenting optical system. In this embodiment, the irradiation optical system 10 also serves as the fixation presenting optical system. The fixation optical system presents a fixation target to the subject's eye. More specifically, visible light irradiated by the light irradiating unit 11 is used as the fixation target. In this case, the light emitting unit 10 may turn on the visible light at a specified scanning position and turn it off at other scanning positions to induce fixation of the subject's eye. The position of the fixation target can be changed by controlling the positions of the optical scanners 16A and 16B.

With reference to Figure 3, the fundus image processing performed by the fundus imaging device 1 will be described. As described above, the fundus imaging device 1 is an example of a computer. Therefore, the device that performs the fundus image processing illustrated in Figure 3 is not limited to the fundus imaging device 1. If the fundus image processing described below is performed by a device other than the fundus imaging device 1, the process of "capturing and acquiring" the image will be interpreted as a process of acquiring the data of the image captured by the fundus imaging device 1. In the fundus image processing of this embodiment, a panoramic fundus image is generated by combining multiple fundus images captured by the imaging optical system 2.

For simplicity of explanation, the number of fundus images used to construct a panoramic fundus image will be described as the minimum of two. However, this is not necessarily limited to this, and a panoramic fundus image may be generated based on three or more fundus images.

As shown in FIG. 3, when fundus image processing is started, the processor 70 of the fundus imaging device 1 captures fundus images at multiple imaging positions on the fundus of the same subject eye as the imaging target, and acquires image data (S1). This results in the acquisition of multiple fundus images captured at mutually different imaging positions.

For example, the photographing position is changed by changing the position of the fixation target presented to the subject's eye. Here, two fundus images (5A, 5B) are acquired by photographing at two photographing positions different in the left-right direction. At this time, the photographing positions of the two fundus images 5A and 5B are set so that the images partially overlap each other. In addition, in this embodiment, as an example, the fundus image 5A is an image photographed in a state where the optical axis of the photographing optical system and the visual axis are aligned (i.e., an image in which the fovea is located approximately at the center of the image), and the fundus image 5B is an image photographed at a position displaced in the left-right direction from the fundus image 5A. However, this is not necessarily limited to this. In the following description, for the sake of simplicity, unless otherwise specified, it is assumed that the vertical deviation of the photographing positions is not taken into consideration (does not occur) when photographing the two photographed images 5A and 5B.

Then, the processor 70 executes a correction process (S2). In the correction process (S2), the distortion of the two fundus images 5A, 5B that partially overlap each other is corrected based on the image information of the fundus images. As a result of the correction process (S2) in this embodiment, not only are the distortions of the two fundus images 5A, 5B that partially overlap each other corrected, but alignment is also performed. However, this is not necessarily limited to this, and the distortion correction and alignment may be performed in stages by separate processes.

The correction process (S2) will be described in detail below. In this embodiment, based on the image information of two fundus images 5A and 5B taken at two different photographing positions in the left-right direction, the distortion of each image, which is the distortion component in the left-right direction, is corrected by the correction process (S2).

First, the processor 70 sets multiple corresponding points 6a, 6b on the two fundus images 5A, 5B. Here, the fundus image 5A is treated as the target image, and the fundus image 5B is treated as the source image. First, the processor 70 detects multiple points (feature points) on the tissue to obtain the corresponding points from within the image area of the fundus image 5A, which is the target image. As an example, the well-known opencv good features to track function or the like is used to detect the feature points. As a result, multiple feature points are detected approximately uniformly throughout the image area of the fundus image 5A. In detail, in this embodiment, multiple feature points are arranged so that the number of feature points in a unit area is within a predetermined range.

Next, the processor 70 detects corresponding points 6b that correspond to the feature points of the fundus image 5A, which is the target image, from within the image area of the frontal fundus image 5B, which is the source image. To detect the corresponding points 6b, for example, template matching or block matching methods can be used. Of the multiple feature points set in the fundus image 5A, some that have corresponding points 6b in the fundus image 5B are treated as corresponding points 6a in the fundus image 5A.

Next, a coordinate transformation equation that minimizes the distance between corresponding points on the two fundus images 5A and 5B (i.e., the distance between corresponding points 6a and 6b that are set to the same feature) is found. In this embodiment, the coordinate transformation equation is expressed by adding together an equation representing a distortion correction model and an equation representing parallel translation. The equation representing the distortion correction model in this embodiment is the distortion correction equation. The coordinate transformation equation for point i is shown as [Equation 1].

In Equation 1, "x _ai (1+k _a1 r _ai ² +k _a2 r _ai ⁴ +k _a3 r _ai ⁶ )" represents the distortion correction model. Also, x_shift _a represents parallel translation. Note that x _ai is the x coordinate of point i in fundus image 5A, x _ai ' is the x coordinate of point i in fundus image 5A' after transformation, and r _ai is the distance from the origin of point i in fundus image 5A. Note that the origin here is the center of the image. The coordinate transformation equation for fundus image 5B can also be expressed in a format similar to [Equation 1].

In this embodiment, the coefficients of the above coordinate transformation equation are solved using the least squares method. Minimizing the distance between corresponding points can be expressed as [Equation 2]. By deriving the coefficients of [Equation 2], it is possible to obtain a coordinate transformation equation that minimizes the distance between corresponding points on the two fundus images 5A and 5B.

However, simply minimizing the distance between corresponding points may result in the two fundus images 5A and 5B being aligned inappropriately with respect to the actual distortion. Therefore, in this embodiment, the coefficients are derived by solving the constrained least squares method based on the following constraints:
Constraint 1: The coordinates of any one image do not change when translated.
Constraint 2: The coordinates of the center of the image do not change due to the distortion correction model.

<Regarding constraint 1>
Constraint condition 1 is derived from the need to determine an appropriate image as a reference when aligning a plurality of images. In this embodiment, by excluding translation from the coordinate transformation of the fundus image 5A, the origin of the fundus image 5A is fixed as the origin of the transformed image.

<Regarding constraint 2>
Near the optical axis of the photographing optical system, it is considered that there is almost no curvature of field of the photographing optical system and almost no displacement of the fundus surface in the direction of the optical axis, regardless of individual differences in the shape of the subject eye and the angle between the optical axis of the photographing optical system and the visual axis, etc. When the photographing position is changed by changing the positional relationship between the optical axis of the photographing optical system and the visual axis, as in this embodiment, the image center of each fundus image 5A, 5B is the area with the least distortion in each image, and thus constraint condition 2 is derived.

Constraints 1 and 2 can be expressed, for example, by the following equations. Based on constraint 1, [Equation 3] is derived. Furthermore, according to constraint 2, the coordinates do not move near the center (x=0.1) of each fundus image 5A and 5B due to distortion correction, so [Equation 4] and [Equation 5] are derived, respectively.

The coordinates of each pixel after transformation can be obtained by applying the coordinate transformation formula obtained by solving [Equation 2] to [Equation 5] to each pixel of each fundus image 5A, 5B. In this way, the pixel values of each pixel in the input image (fundus images 5A, 5B before transformation) are assigned to the transformed coordinates, respectively, to obtain an output image. Here, for pixels in the output image to which no pixel value has been assigned, the pixel value can be obtained by interpolating the surrounding pixels. .

Image distortion occurs not only in the left-right direction (x direction), which is the direction connecting the photographing positions of the two fundus images 5A and 5B, but also in the up-down direction (y direction), which is perpendicular to the left-right direction. A transformation formula for the y direction can be obtained in a similar manner, but when dealing with only images photographed at different positions in the left-right direction as in this embodiment, it is difficult to appropriately correct the distortion in the up-down direction. Therefore, for example, in one fundus image (preferably, fundus image 5A photographed with the optical axis of the photographing optical system and the visual axis aligned; an image in which the fovea is located approximately at the center of the image), it is possible to assume that the distortion in the x direction and the distortion in the y direction are equivalent, and to obtain a coordinate transformation in the y direction for each y coordinate of fundus images 5A and 5B.

In this embodiment, each pixel in the fundus images 5A and 5B is moved based on the coordinate transformation equation obtained under the constraints.

In detail, the distortion in each of the fundus images 5A and 5B is corrected based on the equation of the distortion correction model. As described above, the center of the image is transformed to minimize the distance between corresponding points of the fundus images 5A and 5B under the condition that the coordinates of the center of the image do not change due to the distortion correction model (condition 1), and the distortion that increases the further away from the center of each image is appropriately suppressed. At this time, the peripheral parts of the transformed fundus images 5A and 5B are elongated compared to the center of the fundus images 5A and 5B before the transformation. In this way, in this embodiment, the distortion in the fundus images 5A and 5B can be appropriately corrected without necessarily requiring either the information on the optical system of the device or the information on the fundus shape. That is, not only the effect of the field curvature caused by the photographing optical systems 10 and 20, but also the effect of the distortion that differs for each subject eye caused by the curved shape of the fundus is suppressed. Furthermore, alignment is performed based on the equation of parallel movement.

After the correction process is performed as described above, the fundus images 5A and 5B are combined (S3). This results in a panoramic fundus image based on the two fundus images 5A and 5B. The fundus images 5A and 5B on which the synthesis was based had larger distortions in the peripheral areas of the images, but the distortions have been appropriately corrected by the correction step (S2), so that the unnatural appearance of the seams between the images in the panoramic fundus image is appropriately reduced. In addition, the converted fundus images 5A and 5B on which the synthesis was based have larger distortions in the peripheral areas stretched compared to the images before the transformation, so that a panoramic fundus image in which the tissue in the peripheral areas is easy to observe can be obtained.

<Modification>
The techniques disclosed in the above embodiments are merely examples, and therefore the techniques exemplified in the above embodiments can be modified.

For example, in the above embodiment, a case has been described in which distortion correction is performed based on two fundus images 5A and 5B, and a panoramic fundus image is generated. However, this is not necessarily limited to this, and for example, distortion correction may be performed based on three or more fundus images, and a panoramic fundus image may be generated.

For example, in addition to the two fundus images 5A and 5B shown in the above embodiment, at least a third fundus image 5C may be used. The third fundus image 5C may be a fundus image captured at a different capturing position in the y direction from the two fundus images 5A and 5B, and may be an image that partially overlaps with at least one of the two fundus images 5A and 5B.

When three fundus images 5A, 5B, and 5C are present as input images, the coordinate transformation of the three images may be solved simultaneously. That is, in addition to minimizing the distance between corresponding points in fundus images 5A and 5B as shown in the above embodiment, the coordinate transformation coefficients of each image that satisfy the minimization of the distance between corresponding points in fundus images 5A and 5C are found. Here, in addition to the constraints shown in the above embodiment, the coordinate transformation coefficients of each image are found under the constraint that there is no distortion near the center in fundus image 5C as well. By performing this for each of the x and y directions, it is possible to find the coordinate transformation equations used to correct the distortion of 5A, 5B, and 5C and to align them.

In this method, the correction of distortion in the y direction is performed based on coordinate transformation using image information from a third fundus image 5C, which is taken at a different shooting position in the y direction from the two fundus images 5A and 5B, making it easier to properly correct distortion in accordance with the actual shape of the subject's eye.

1 Fundus photography device

Claims (4)

A fundus image processing program executed by a computer that processes fundus image data obtained by photographing a fundus of a subject's eye,
The fundus image processing program is executed by the processor of the computer,
acquiring two or more fundus images captured at different photographing positions through an imaging optical system, the two or more fundus images being partially overlapped with each other;
a correction step of correcting distortion of a part or all of images included in two or more fundus images that partially overlap each other based on image information of the two or more fundus images;
a synthesis step of generating a panoramic fundus image by synthesizing two or more fundus images in which the distortion has been corrected, the synthesis step being executed by the computer;
Furthermore, the correction step
a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other;
a coordinate transformation step of determining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points based on coordinate information of the plurality of corresponding points, and applying the coordinate transformation to the fundus image in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points. A fundus image processing program executed by a computer that processes fundus image data obtained by photographing a fundus of a subject's eye,
The fundus image processing program is executed by the processor of the computer,
acquiring two or more fundus images captured at different photographing positions through an imaging optical system, the two or more fundus images being partially overlapped with each other;
a correction step of correcting distortion of a part or all of images included in two or more fundus images that partially overlap each other based on image information of the two or more fundus images;
is executed by the computer,
Furthermore, the correction step
a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other;
a coordinate transformation step of determining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points based on coordinate information of the plurality of corresponding points, and applying the coordinate transformation to the fundus image in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points. A fundus image processing program as described in claim 1 or 2, wherein in the correction step, the coordinate transformation is performed under the constraint that, for each fundus image to be corrected for distortion, the amount of coordinate change in the center of the image is suppressed more than the amount of coordinate change in the peripheral part of the image. The photographing optical system;
A fundus photographing apparatus comprising: an image processing apparatus that executes the fundus image processing program according to claim 1 .