JP7623682B2 - Image processing device, image processing program, and image processing method - Google Patents

Description

The present invention relates to an image processing device, an image processing program, and an image processing method.

For early detection of vascular disorders in the brain, medical devices that visualize the inside of the human body, such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging), are used. However, these medical devices are expensive. This means that the burden of capital investment is large, and only a limited number of hospitals have these medical devices. In addition, medical costs are high, which means that patients bear a large medical expense burden.

The brain is the same organ as the eye in terms of development. Therefore, the state of blood vessels in the brain can be inferred from the state of retinal blood vessels at the fundus. Therefore, fundus examinations, which do not require the aforementioned medical equipment, have been used to diagnose the risk of vascular disorders in the brain. In recent years, with the revision of the third stage of specific health checkups, fundus examinations have been incorporated into internal medicine examination items. However, fundus examinations depend heavily on the experience and skill of the examining physician. For this reason, it is difficult for internists who are not specialists to perform accurate fundus examinations. Therefore, there is a demand for the automation of fundus examinations. To automate fundus examinations, image processing technology is required to accurately extract blood vessels from images of the fundus.

Fundus examinations can be used not only to prevent and detect eye diseases early, but also to estimate the progression of arteriosclerosis in blood vessels in the brain. Arteriosclerosis does not progress at a constant rate, but at an accelerating rate. Therefore, early treatment of arteriosclerosis can be achieved by observing the degree of arteriosclerosis over time and identifying patients whose arteriosclerosis is accelerating.

Among the indices for evaluating the degree of progression of arteriosclerosis, the ratio of the venous diameter at the intersection of an artery and a vein (diameter of the part influenced by the artery (V2) / diameter of the part not influenced by the artery (V1): V2/V1 ratio) is known as an index that is not affected by vascular pulsation (see, for example, Patent Document 1).

To automate fundus examinations using the V2/V1 ratio, a technology is needed to automatically detect the intersections of retinal arteries and veins using image processing. Here, a method using a well-known CNN (Convolutional Neural Network) requires a large amount of training data for training the CNN. However, it is difficult to prepare a large amount of training data that captures images of the intersections of retinal blood vessels, which is also medical data.

Previously, technologies have been proposed to automatically detect vascular intersections as an alternative to CNN (see, for example, Patent Document 2).

The technology disclosed in Patent Document 2 generates a thinned blood vessel image in which blood vessels are thinned from a fundus image, generates a thinned blood vessel image with blood vessel diameter calculated by a blood vessel diameter measurement means, and extracts blood vessel intersections from the thinned blood vessel image. In extracting these intersections, for the thinned branching points, the angles and other branching points/intersections/end points that connect to them are measured, pairs of branching points with close distances between them are detected, and branching point pairs that satisfy specified conditions based on predetermined criteria are regarded as intersections. As a result, this technology can detect intersections that appear as two branching point pairs by thinning blood vessels.

However, this technology requires many items as judgment criteria, and the specific calculation method for each item is unclear. In addition, the comparison target for the specified conditions that must be met is unclear. As a result, this technology lacks objectivity and cannot be used as is to detect blood vessel intersections.

JP 2008-079682 A JP 2009-189586 A

The present invention aims to detect the intersections between arteries and veins from captured images of blood vessels.

The image processing device according to the present invention is an image processing device that identifies vascular intersections based on an image in which blood vessels are captured, and includes a storage unit that stores a thinned image of blood vessels included in the captured image, a detection unit that detects a plurality of detection points corresponding to any of the intersections, branching points, and end points of the thinned blood vessels based on the thinned image, a group information identification unit that identifies, based on the thinned image, group information indicating a pair of detection points connected by a thinned blood vessel out of the plurality of detected detection points, and a blood vessel graph generation unit that generates a blood vessel graph in which each of the two detection points constituting a pair is connected by a line segment based on the identified group information. The system includes a connection information acquisition unit that acquires connection information for each of the detected detection points based on the vascular graph, and a crossing point identification unit that identifies a detection point that corresponds to a crossing point from the detected detection points based on the connection information, and the crossing point identification unit extracts a plurality of branch detection points that are connected to three line segments from the detected detection points based on the connection information, extracts two branch detection points that are connected to a common line segment from the extracted branch detection points based on the connection information as a detection point pair, and identifies the branch detection points that constitute the detection point pair and that satisfy a predetermined condition as the detection point that corresponds to the crossing point.

According to the present invention, it is possible to detect the intersections between arteries and veins from captured images of blood vessels.

1 is a network connection diagram showing an embodiment of an image processing apparatus according to the present invention; 1 is a functional block diagram showing an embodiment of an image processing apparatus according to the present invention; 3 is a schematic diagram showing an example of a captured image processed by the image processing device of FIG. 2; 1 is a flowchart illustrating an embodiment of an image processing method according to the present invention. 5 is a flowchart of a blood vessel graph generation process included in the image processing method of FIG. 4 . 6 is a schematic diagram showing an example of an extracted image generated in the blood vessel graph generation process of FIG. 5 . 6 is a schematic diagram showing an example of a thinned image generated in the blood vessel graph generation process of FIG. 5 . FIG. 3 is a schematic diagram showing an example of a filter stored in a storage unit included in the image processing device of FIG. 2 . 9 is a schematic diagram showing an example of a state in which each filter pattern in FIG. 8 is applied to the thinned image in FIG. 7 . 6 is a flowchart of a group identification process included in the vascular graph generation process of FIG. 5 . 11 is a schematic diagram showing an example of a blood vessel search in the group identification process of FIG. 10 . 6 is a schematic diagram showing an example of a blood vessel graph generated by the blood vessel graph generation process of FIG. 5 . FIG. 5 is a flowchart of a link information acquisition process included in the image processing method of FIG. 4 . 14 is a schematic diagram showing an example of detection points from which link information is acquired in the link information acquisition process of FIG. 13 . 14 is a schematic diagram showing a method of calculating an angle between line segments in the connection information acquisition process of FIG. 13. 14 is a schematic diagram showing a method of calculating an aperture in the connection information acquisition process of FIG. 13. 5 is a flowchart of a crossing point detection process included in the image processing method of FIG. 4 . 18 is a schematic diagram showing detection points that can be detected as detection points corresponding to crossing points in the crossing point detection process of FIG. 17 . FIG. 5 is a schematic diagram showing the success rate and the failure rate when blood vessel intersections are detected from an actual fundus image by the image processing method of FIG. 4 .

Below, with reference to the drawings, we will explain embodiments of an image processing device (hereinafter referred to as "the device"), an image processing program (hereinafter referred to as "the program"), and an image processing method (hereinafter referred to as "the method") according to the present invention.

The present invention detects a plurality of detection points based on a thinned image in which blood vessels included in an image in which blood vessels are captured are thinned, and identifies group information of the detection points. The present invention also generates a blood vessel graph based on the group information, obtains connection information for each of the detection points based on the blood vessel graph, and detects detection points corresponding to intersections based on the connection information.

In the embodiment described below, the present invention will be explained using a fundus image in which a vein and an artery are captured as an example of a captured image. The captured image includes a portion where an artery and a vein cross each other (hereinafter referred to as a "crossing point").

"Captured image" includes, for example, fundus images captured by a fundus camera, X-ray images captured by an X-ray imaging device, CT images captured by a CT device, as well as images captured by a normal optical camera. In the present invention, the captured image includes at least blood vessels as the subject. As described above, in this embodiment, the captured image is a fundus image.

"Blood vessels" includes not only human blood vessels but also blood vessels of animals other than humans. In this embodiment, the blood vessels are blood vessels in the fundus of a human eye.

A "detection point" is a point (pixel) that corresponds to either an intersection, branching point, or end point of a thinned blood vessel. Here, an intersection refers to a portion in the captured image where an artery and a vein cross, a branching point refers to a portion in the captured image where one blood vessel splits into two blood vessels (branch vessels), and an end point refers to the end of a blood vessel in the captured image.

"Group information" is information that indicates a pair of two detection points that are connected by a thinned blood vessel among a plurality of detection points. In other words, group information is information that indicates a detection point that is connected to another detection point among a plurality of detection points. Group information includes pixel information (coordinates, number of pixels) of the blood vessels that are connected to the two detection points that make up the pair.

A "vascular graph" is a graph of blood vessels captured in a captured image, with each detection point as a node and each line segment connecting multiple detection points as an edge. In other words, a vascular graph is made up of multiple detection points and multiple line segments connecting the detection points.

"Connection information" is information related to objects (other detection points, line segments, blood vessels) connected to a detection point. Connection information includes, for example, the number of line segments connected to the detection point (hereinafter also referred to as "order"), the angle between the line segments connected to the detection point, other connection points connected to the detection point via line segments, the length of the blood vessel connected to the detection point (blood vessel connected to two detection points), and the caliber of the blood vessel connected to the detection point.

●Image processing device●
First, an embodiment of the present device will be described.

FIG. 1 is a network connection diagram showing an embodiment of the present device.
The figure shows that the device 1 is connected to an imaging device 2 via a network (communication line) N that uses a wired communication method or a wireless communication method.

The device 1 identifies the intersections of blood vessels based on the captured image i1 in which the blood vessels are captured. The specific configuration and operation of the device 1 and the captured image i1 will be described later.

The imaging device 2 captures the captured image i1. The imaging device 2 is, for example, a fundus camera.

The network N is, for example, a communication network such as the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

Configuration of the Image Processing Device FIG. 2 is a functional block diagram showing an embodiment of the device 1. As shown in FIG.

The present device 1 is realized, for example, by a personal computer. The present program runs on the present device 1, and the present program cooperates with the hardware resources of the present device 1 to realize the present method.

Here, by having a computer (not shown) execute this program, the program can cause the computer to function in the same manner as the device 1, and cause the computer to execute the method.

The device 1 includes a communication unit 11, a memory unit 12, and a control unit 13.

The communication unit 11 acquires the captured image i1 from the imaging device 2 via the network N. The communication unit 11 is composed of, for example, a communication module and an antenna. The captured image i1 is stored in the storage unit 12.

The storage unit 12 stores information necessary for the device 1 to execute the method described below. The storage unit 12 is composed of, for example, a recording device such as a hard disk drive (HDD) or solid state drive (SSD) provided in the device 1, a volatile memory such as a random access memory (RAM), and/or a portable storage medium such as a flash memory.

The control unit 13 controls the operation of the entire device 1 and executes the method described below. The control unit 13 is composed of, for example, a CPU (Central Processing Unit) provided in the device 1, a volatile memory such as a RAM that functions as a working area for the CPU, and a non-volatile memory such as a ROM (Read Only Memory) that stores various information such as the program. The control unit 13 includes an extracted image generating unit 131, a thinned image generating unit 132, a detection unit 133, a group information identifying unit 134, a vascular graph generating unit 135, a connection information acquisition unit 136, and an intersection identifying unit 137.

The extracted image generating unit 131 generates an extracted image i2 based on the captured image i1. The specific operation of the extracted image generating unit 131 will be described later.

"Extracted image i2" is an image in which the blood vessel area contained in captured image i1 is extracted.

The thinned image generating unit 132 generates a thinned image i3 based on the extracted image i2. The specific operation of the thinned image generating unit 132 will be described later.

"Thinned image i3" is an image in which the blood vessel area included in extracted image i2 has been thinned to a width of one pixel. In other words, thinned image i3 is an image in which the blood vessels included in captured image i1 have been thinned.

The detection unit 133 detects multiple detection points corresponding to any of the intersections, branching points, and end points of the thinned blood vessels based on the thinned image i3. The specific operation of the detection unit 133 will be described later.

The group information identification unit 134 identifies group information indicating a pair of two detection points connected by a thinned blood vessel from among the multiple detected detection points based on the thinned image i3. The specific operation of the group information identification unit 134 will be described later.

Based on the identified pair information, the blood vessel graph generating unit 135 generates a blood vessel graph i4 in which each of the two detection points constituting the pair is connected by a line segment. The specific operation of the blood vessel graph generating unit 135 will be described later.

The connection information acquisition unit 136 acquires connection information for each of the detected detection points based on the extracted image i2, the thinned image i3, and the blood vessel graph i4. The specific operation of the connection information acquisition unit 136 will be described later.

The intersection point identification unit 137 identifies a detection point that corresponds to an intersection point among the multiple detected detection points based on the connection information. The specific operation of the intersection point identification unit 137 will be described later.

In addition, the extracted image generating unit, thinned image generating unit, detection unit, group information identifying unit, vascular graph generating unit, connection information acquiring unit, and intersection identifying unit in the present invention may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of a CPU.

In addition, the extracted image generating unit, thinned image generating unit, detection unit, group information identifying unit, vascular graph generating unit, connection information acquiring unit, and intersection identifying unit in the present invention may be configured using common hardware (such as a CPU) or may be configured using separate hardware.

●Image processing method (operation of image processing device)●
Next, the operation of the present device 1, that is, an embodiment of the present method executed by the present device 1, will be described with reference to FIG.

In the following description, it is assumed that the device 1 acquires an image i1 from the imaging device 2 via the network N and stores it in the storage unit 12. It is also assumed that the image i1 used in the following description captures an intersection point where a part of a vein and an artery cross each other.

FIG. 3 is a schematic diagram showing an example of a captured image i1.
The figure shows that part of an artery branches into two, one of the branched arteries has an end point, and parts of the artery and vein cross, with part of the vein being obscured by part of the artery and not imaged.

Figure 4 is a flowchart illustrating an embodiment of the method.

This method includes a vascular graph generation process (S1), a connection information acquisition process (S2), and an intersection point detection process (S3).

Blood Vessel Graph Generation Process First, the device 1 executes a blood vessel graph generation process (S1).

Figure 5 is a flowchart of the vascular graph generation process (S1).

The "blood vessel graph generation process (S1)" is a process for generating a blood vessel graph i4 from a captured image i1.

First, the extracted image generating unit 131 reads out the captured image i1 from the storage unit 12, and performs known image processing on the captured image i1 (for example, processing using a Convolutional Neural Network (CNN) using learning, or processing using a Black-Top-Hat (BTH) transformation, which is a type of morphological operation) to extract the blood vessel area included in the captured image i1, and generate an extracted image i2 (S101). That is, for example, the extracted image generating unit 131 binarizes the blood vessel area included in the captured image i1 to white and the area other than the blood vessel area (background area) to black, thereby generating an extracted image i2 in which the captured image i1 has been binarized. The extracted image i2 is stored in the storage unit 12.

FIG. 6 is a schematic diagram showing an example of an extracted image i2.
This figure shows that the blood vessel region extracted from the captured image i1 shown in FIG. 1 is represented in white (pixel value "1"), and the background region is represented in black (pixel value "0").

Return to Figure 5.
Next, the thinned image generating unit 132 performs a known thinning process on the extracted image i2 to thin the blood vessel region included in the extracted image i2 to a width of one pixel, thereby generating a thinned image i3 (S102). The thinned image i3 is stored in the storage unit 12.

The "known thinning process" is an image process that thins the white areas of a binarized image, and is, for example, the process described in "Zhang T. Y., and Ching Y. Suen, "A fast parallel algorithm for thinning digital patterns", Communications of the ACM, (USA), 1984, Vol. 27, No. 3, pp. 236-239."

FIG. 7 is a schematic diagram showing an example of a thinned image i3.
This figure shows that the blood vessel region of the extracted image i2 shown in FIG. 2 has been thinned to a width of one pixel.

Return to Figure 5.
Next, the detection unit 133 applies each pattern of the filter F to each pixel of the thinned image i3 to identify pixels that are detection points (S103). That is, the detection unit 133 executes a filter process using the filter F on the thinned image i3.

"Filter F" is a filter that is applied to the thinned image i3 so that the detection unit 133 can identify vascular intersections, branching points, end points, and passing points from each pixel of the thinned image i3. Filter F includes multiple patterns that correspond to vascular intersections, branching points, end points, and passing points. Filter F is stored in advance in the storage unit 12.

FIG. 8 is a schematic diagram showing an example of the filter F stored in the storage unit 12. As shown in FIG.
This figure shows that filter F includes multiple patterns each consisting of 3x3 pixels. This figure shows that filter F includes a first group in which a different pattern appears when rotated, and a second group in which no other pattern appears when rotated. This figure shows that each pixel in each pattern is binarized (white: pixel value "1", black: pixel value "0").

Specifically, the detection unit 133 selects one pixel from among the pixels constituting the thinned image i3 as a target pixel, and refers to a detection range of 3 x 3 pixels including eight pixels surrounding the target pixel. Next, the detection unit 133 superimposes each pattern on the detection range to determine whether or not there is a pattern that matches the detection range. When there is a pattern that matches the detection range, the detection unit 133 identifies the pixel at the center of the detection range (i.e., the target pixel) as a pixel that is the detection point. Next, the detection unit 133 calculates the inner product of the pixel value ("0" or "1") of the pixel in the detection range and the pixel value ("0" or "1") of the pixel that constitutes the pattern that matches the detection range. Next, when the inner product is "2", the detection unit 133 identifies the target pixel as an end point, when the inner product is "4" or more, the detection unit 133 identifies the target pixel as a branch point (including a crossing point), and when the inner product is "3", the detection unit 133 identifies the target pixel as a pass point. At this point, no distinction is made between branch points and intersection points. The coordinate information of the identified detection points (end points, branch points) is associated with the thinned image i3, for example, and stored in the storage unit 12.

In addition, the detection unit in the present invention may identify the target pixel as either an end point or a branch point (including an intersection point) based on the sum of the number of pixels with a pixel value of "1" in the detection range, instead of the dot product.

FIG. 9 is a schematic diagram showing an example of a state in which each pattern of the filter F is applied to each pixel of the thinned image i3.
In the figure, "a" indicates that the inner product is "4 (branch point),""b" indicates that the inner product is "3 (passing point),""c" indicates that the inner product is "2 (end point)," and "d" indicates that the inner product is "5 (crossing point included in the branch point)."

Return to Figure 5.
Next, the group information identifying unit 134 executes a group identifying process (S104).

FIG. 10 is a flowchart of the group identification process (S104).
FIG. 11 is a schematic diagram showing an example of blood vessel search in the group identification process (S104).
11, for ease of explanation, the central pixel is labeled "x1", and the eight pixels surrounding the central pixel are labeled "x2" to "x9". In addition, some pixels in a second region described below are labeled "x10" to "x12". In the following explanation, the pixels surrounding the central pixel are the pixels adjacent to the central pixel diagonally above, below, left, and right, and the pixels adjacent to the central pixel are the pixels adjacent to the central pixel above, below, left, and right.

The "pair identification process (S104)" is a process for identifying pairs of two detection points connected by a thinned blood vessel among the multiple detected detection points by searching for thinned blood vessels connected to each detection point based on the thinned image i3 and the coordinate information of the detection points associated with the thinned image i3 from the storage unit 12. In other words, the pair identification process (S104) identifies other detection points connected to a certain detection point via a blood vessel.

First, the group information identification unit 134 selects one of the detected detection points as the starting point of the search (S104-1). Next, the group information identification unit 134 recognizes a 3 x 3 pixel region with the detection point as the first center pixel as the first region (S104-2). Next, the group information identification unit 134 references eight pixels surrounding the first center pixel in the first region as reference pixels (S104-3). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-4). Here, the group information identification unit 134 performs the determination in the order of, for example, "x2" to "x9", and detects the pixel that is first determined to be a pixel that corresponds to a specific pixel as a specific pixel in the first region (S104-5).

"Specific pixels" are pixels that represent thinned blood vessels. "Non-specific pixels" are pixels that represent things other than thinned blood vessels (i.e., the background).

Then, the group information identification unit 134 recognizes a 3×3 pixel region with the detected specific pixel as the second center pixel as the second region (S104-6). Next, the group information identification unit 134 references, as reference pixels, five pixels among the surrounding pixels surrounding the second center pixel in the second region, excluding the first center pixel and the pixel adjacent to the first center pixel (S104-7). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-8). Here, the group information identification unit 134 performs the determination in the same order as in the process (S104-5), and detects the pixel that is first determined to be a pixel corresponding to a specific pixel as a specific pixel in the second region (S104-9). Next, the group information identification unit 134 determines whether the detected specific pixel is a detection point other than the detection point that is the starting point (S104-10).

When the detected specific pixel is not another detection point ("N" in S104-10), the group information identification unit 134 recognizes a 3 x 3 pixel region with the detected specific pixel as the third center pixel as the third region (S104-11). Next, the group information identification unit 134 refers to five pixels, excluding the second center pixel and the pixel adjacent to the second center pixel, out of the eight pixels surrounding the third center pixel in the third region as reference pixels (S104-12). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-13). Here, the group information identification unit 134 performs the determination in the same order as in the process (S104-5), and detects the pixel that is first determined to be a pixel corresponding to a specific pixel as a specific pixel in the third region (S104-14). Next, the group information identification unit 134 determines whether the detected specific pixel is another detection point (S104-15). When the detected specific pixel is not another detection point ("N" in S104-15), the pair identification process (S104) returns to process (S104-6). On the other hand, when the detected specific pixel is another detection point ("Y" in S104-10, 15), the pair information identification unit 134 identifies the pair information (information indicating the coordinate information of the two detection points and the number of specific pixels between the two detection points (hereinafter referred to as "number of pixels between detection points")) of the detection point that is the starting point and the other detection point that is the last detected specific pixel as a pair of detection points (S104-16). The pair information of the detection points is stored in the storage unit 12, for example, in association with the number of pixels between the detection points.

Return to Figure 5.
Next, the blood vessel graph generating unit 135 generates a blood vessel graph i4 by connecting two detection points constituting each pair with a line segment based on the group information (S105). At this time, the blood vessel graph generating unit 135 assigns an identification number to each detection point based on, for example, a predetermined condition in order to identify each detection point. The blood vessel graph i4 is stored in the storage unit 12 in association with, for example, the group information and the number of pixels between the detection points.

FIG. 12 is a schematic diagram showing an example of a blood vessel graph i4.
In the figure, the numbers given to each detection point are identification numbers of the respective detection points.

In this way, the device 1 executes the vascular graph generation process (S1) on the captured image i1 to generate a vascular graph i4 that represents blood vessels using points (detection points) and straight lines (line segments). By representing blood vessels using a vascular graph, the device 1 can easily obtain the connectivity information described below.

Linkage Information Acquisition Process Next, the device 1 executes linkage information acquisition process (S2: see FIG. 4).

Figure 13 is a flowchart of the connection information acquisition process (S2).

The "connection information acquisition process (S2)" is a process for acquiring connection information based on the extracted image i2, the thinned image i3, and the blood vessel graph i4.

First, the connection information acquisition unit 136 acquires, based on the vascular graph i4, the detection points connected to each detection point, the number (order) of detection points (i.e., line segments) connected to each detection point, and the angles between the line segments connected to each detection point as connection information (S201).

FIG. 14 is a schematic diagram showing an example of detection points from which link information is obtained.
The figure shows an enlarged portion of the blood vessel graph i4. The figure shows that the detection points connected to the detection point "10" are "11", "14", "25", and "30", and that the degree of the detection point "10" is "4".

FIG. 15 is a schematic diagram showing a method for calculating an angle between line segments.
The figure shows that when an arbitrary detection point is "P", a detection point connected to the detection point "P" by a line segment "a" is "Q", and a detection point connected to the detection point "P" by a line segment "b" is "R", the angle "θ" between the line segments "a" and "b" can be calculated using the following formula (1).

θ=cos ^-1 (a・b/|a|・|b|) (1)

Return to Figure 13.
Next, the connection information acquisition unit 136 acquires the length of the blood vessel connecting the two detection points (hereinafter referred to as "blood vessel length") as connection information based on the number of pixels between the detection points stored in association with the blood vessel graph i4 (S202). Here, the number of pixels between the detection points is the number of specific pixels that exist between the two detection points in the thinned image i3. In other words, the number of pixels between the detection points is the blood vessel length. The number of pixels between the detection points includes the detection points.

Note that the number of pixels between detection points does not have to include the detection points.

Next, the connection information acquisition unit 136 calculates the diameter of the blood vessel connecting the two detection points as connection information based on the extracted image i2 and the thinned image i3 (S203).

FIG. 16 is a schematic diagram showing a method for calculating the aperture.
In the figure, detection points are indicated by "●".

Specifically, the connection information acquisition unit 136 extracts an image (hereinafter referred to as a "selected blood vessel image") including blood vessels (hereinafter referred to as "selected blood vessels") connected only to two arbitrarily selected detection points from the thinned image i3 (S203-1). Next, the connection information acquisition unit 136 performs a known expansion process on the selected blood vessel image to generate an expanded blood vessel image in which the selected blood vessel is expanded (S203-2). Next, the connection information acquisition unit 136 performs a mask process on the expanded blood vessel image using the region of the extracted image i2 corresponding to the expanded blood vessel image as a mask to define a pseudo blood vessel region (S203-3). That is, for example, the connection information acquisition unit 136 processes pixels that correspond to blood vessels in the expanded blood vessel image and that protrude from the masked extracted image i2 as a background. Next, the connection information acquisition unit 136 calculates the blood vessel area (total number of pixels showing blood vessels) of the pseudo blood vessel region (S203-4). Next, the connectivity information acquisition unit 136 calculates the diameter of the selected blood vessel using the following formula (2) (S203-5).

Diameter "w" = (vascular area between two detection points) / (distance between two detection points) (2)

The acquired connection information is associated with the corresponding detection points and line segments and stored in the memory unit 12.

Note that process (S201) may be executed after process (S202) and process (S203), or may be executed in parallel with process (S202) and process (S203).

Crossing Point Detection Process Next, the device 1 executes the crossing point detection process (S3: see FIG. 4).

FIG. 17 is a flowchart of the crossing point detection process (S3).
Fig. 18 is a schematic diagram showing detection points that may be detected as detection points corresponding to crossing points in the crossing point detection process (S3). In Fig. 18, blood vessels are shown with dashed lines, line segments are shown with solid lines, and detection points are shown with "◯".

The "intersection detection process (S3)" is a process for detecting a detection point corresponding to an intersection among a plurality of detection points. Most of the branching points of blood vessels are three-way intersections where one blood vessel branches into two blood vessels. Therefore, in the blood vessel graph i4 based on the thinned blood vessels, a detection point with an order of "4" can be regarded as a detection point corresponding to a point where two blood vessels (artery, vein) intersect (vascular intersection). On the other hand, a detection point with an order of "3" can be regarded as a detection point corresponding to a blood vessel branching point. However, an intersection may appear as two adjacent branching points. In other words, a detection point with an order of "3" can correspond not only to a branching point but also to an intersection. The intersection detection process (S3) is a process for determining whether such a detection point with an order of "3" corresponds to a branching point or to an intersection.

First, the intersection identification unit 137 extracts a detection point with degree "4" from among the multiple detection points based on the vascular graph i4 and the connection information (S301), and identifies the detection point with degree "4" as the detection point that corresponds to the intersection (S302).

Next, the intersection point identification unit 137 extracts, from among the multiple detection points, a detection point with degree "3" as a branch extraction point based on the vascular graph i4 and the connection information (S303).

Next, the intersection point identification unit 137 extracts, from among the branch detection points, two detection points that are connected to one common line segment (hereinafter referred to as the "common line segment") as a detection point pair (S304). That is, the intersection point identification unit 137 identifies, as a detection point pair, a detection point that is connected to one end of one common line segment (hereinafter referred to as the "first detection point") and a detection point that is connected to the other end of the common line segment (hereinafter referred to as the "second detection point"). In other words, the detection point pair includes the first detection point and the second detection point that is connected to the first detection point by the common line segment.

Next, the intersection point identification unit 137 acquires the length (blood vessel length) "d" of the blood vessel connecting the first detection point and the second detection point, and the diameter "w" of the blood vessel connecting the first detection point and the second detection point based on the connection information (S305). Next, the intersection point identification unit 137 determines whether the blood vessel length "d" is smaller than the diameter "w" (S306). The diameter "w" is an example of a first threshold value in the present invention.

When the blood vessel length "d" is smaller than the caliber "w"("Y" in S306), the intersection point identification unit 137 acquires an angle "θ ₁ " between two of the three line segments connected to the first detection point, excluding the common line segment (hereinafter referred to as the "first angle"), and an angle "θ ₂ " between two of the three line segments connected to the second detection point, excluding the common line segment (hereinafter referred to as the "second angle") (S307). Next, the intersection point identification unit 137 calculates the absolute value of the difference between the first angle "θ ₁ " and the second angle "θ ₂ ", and determines whether the absolute value is smaller than a predetermined second threshold "V _th " (S308). The second threshold "V _th " is, for example, 20° and is stored in the storage unit 12 in advance.

When the absolute value is smaller than the second threshold value " _Vth "("Y" in S308), the crossing point identification unit 137 identifies the first and second detection points constituting the connection point pair as detection points corresponding to the crossing point (S309). Next, the crossing point identification unit 137 calculates the coordinates of the midpoint between the coordinates of the first detection point and the coordinates of the second detection point, and identifies the coordinates as the coordinates of the detection point pair corresponding to the crossing point (S310).

On the other hand, when the blood vessel length "d" is greater than or equal to the caliber "w"("N" in S306), or when the absolute value is greater than or equal to the second threshold value " _Vth "("N" in S308), the intersection point identification unit 137 identifies each of the first and second detection points constituting the detection point pair as branch points (S311).

Next, the crossing point identification unit 137 determines whether there are any unprocessed detection point pairs (S312). If there are any unprocessed detection point pairs ("Y" in S312), the crossing point identification unit 137 executes the process (S304) and subsequent processes. On the other hand, if there are no unprocessed detection point pairs ("N" in S312), the crossing point identification unit 137 stores the coordinates of the detection points (detection point pairs) corresponding to all the identified crossing points in the memory unit 12 in association with information indicating the crossing points (S313).

In this way, the device 1 determines whether the intersection pair satisfies two conditions: a length condition in which the blood vessel length "d" is compared with a first threshold (diameter "w"), and an angle condition in which the absolute value of the difference between the angles is compared with a second threshold "Vth", and identifies the intersection point that appears as two adjacent branch points. When two blood vessels cross and overlap, the state in which the two blood vessels are locally parallel at the overlapping portion hardly occurs, and the width of the blood vessel at the overlapping portion of the two blood vessels is approximately the same as the diameter of the blood vessel. Therefore, by setting the diameter of the blood vessel corresponding to the common line segment as the first threshold, it is possible to determine whether the detection point pair corresponds to the intersection point. In addition, by varying the first threshold according to the diameter (thickness) of the blood vessel that forms the intersection point, the accuracy of determining whether the detection point pair corresponds to the intersection point is improved compared to the case where a constant value is used as the threshold. Furthermore, when two blood vessels cross, the blood vessel near the intersection point can be approximated to a state in which two line segments cross when viewed locally. Therefore, the first angle can be considered to be approximately the same as the second angle. This method uses a blood vessel graph in which blood vessels are replaced with detection points and line segments, and the first angle and the second angle can be easily calculated, but a certain degree of difference is likely to occur between the first angle and the second angle. Therefore, by setting the second threshold as an empirical threshold, it is possible to determine whether a pair of detection points corresponds to a crossing point.

FIG. 19 is a schematic diagram showing the success rate and the failure rate when blood vessel intersections are detected from actual fundus images by this method.
In the figure, "number of correct answers" is the number of crossing points visually counted in 40 fundus images. "Correct answer rate" is a percentage of the value obtained by dividing the number of detected crossing points by the number of correct crossing points detected. "Incorrect answer rate" is a percentage of the value obtained by dividing the number of incorrect detections by the total number of detections. The second threshold is 20°. As shown in FIG. 19, the correct answer rate of this method was about 97%, and the incorrect answer rate was about 10%.

Summary According to the embodiment described above, the present device 1 includes a detection unit 133, a group information specification unit 134, a blood vessel graph generation unit 135, a connection information acquisition unit 136, and a crossing point specification unit 137. The detection unit 133 detects a plurality of detection points based on the thinned image i3. The group information specification unit 134 specifies group information based on the thinned image i3. The blood vessel graph generation unit 135 generates a blood vessel graph i4 based on the specified group information. The connection information acquisition unit 136 acquires connection information for each of the detected plurality of detection points based on the blood vessel graph i4. The crossing point specification unit 137 extracts a plurality of branch detection points based on the connection information, extracts two branch detection points that are connected to a common line segment from the extracted plurality of branch detection points as a detection point pair, and specifies a branch detection point that constitutes a detection point pair that satisfies a predetermined condition (length condition, angle condition) as a detection point corresponding to a crossing point. According to this configuration, the present device 1 can easily acquire stable connection information based on a blood vessel graph. As a result, the device 1 can accurately detect the intersection points between arteries and veins from the captured images.

According to the embodiment described above, the connection information acquisition unit 136 acquires the angle between the multiple line segments connected to the detection point based on the blood vessel graph i4, acquires the blood vessel length of the blood vessel connecting the two detection points based on the thinned image i3, and acquires the caliber of the blood vessel connecting the two detection points based on the thinned image i3. Normally, when acquiring the blood vessel length based on the captured image i1 and the extracted image i2, it is not easy to identify the start point or end point of the blood vessel from the branch point or intersection point because multiple blood vessels are involved at the branch point or intersection point. However, according to this configuration, since the blood vessel length is acquired using the thinned blood vessel, even if the detection point is a branch point or intersection point, the detection point is represented by one pixel. Therefore, the present device 1 can easily identify the start point and end point of the blood vessel connecting the two detection points. Also, in calculating the caliber of the blood vessel, the area of the blood vessel at each of the start point and end point of the blood vessel is easy to specify, and the connection information acquisition unit 136 can easily calculate the caliber of the blood vessel based on the area of the blood vessel.

Furthermore, according to the embodiment described above, the crossing point identification unit 137 identifies a detection point pair as a detection point corresponding to a crossing point when the blood vessel length between the first detection point and the second detection point is smaller than a predetermined first threshold value and the absolute value of the difference between the first angle and the second angle is smaller than a predetermined second threshold value. In other words, the crossing point identification unit 137 identifies a detection point pair as a detection point corresponding to a crossing point when the detection point pair satisfies a length condition and an angle condition. According to this configuration, the present device 1 can accurately detect the crossing point between an artery and a vein from a captured image by performing a logical sum of the judgments based on the two threshold values.

Furthermore, according to the embodiment described above, the first threshold is the diameter of the blood vessel connecting the first detection point and the second detection point. With this configuration, the device 1 can accurately determine whether or not a detection point pair corresponds to an intersection. In addition, by varying the first threshold according to the diameter (thickness) of the blood vessel that forms the intersection, the accuracy of determining whether or not a detection point pair corresponds to an intersection is improved compared to when a fixed value is used as the threshold.

Furthermore, according to the embodiment described above, the connection information acquisition unit 136 acquires the caliber as connection information based on the thinned image i3 and the extracted image i2. With this configuration, as described above, the device 1 can easily identify the start and end points of the blood vessel based on the thinned image i3, and can accurately identify the area and width of the blood vessel based on the extracted image i2.

Furthermore, according to the embodiment described above, the connection information acquisition unit 136 extracts thinned blood vessels that connect two detection points that form a pair from the thinned image i3, and acquires the diameter of the extracted blood vessels by performing a dilation process and a filter process using the extracted image i2 as a filter. With this configuration, the device 1 can easily identify the start point and end point of the blood vessel from which the diameter is to be acquired, and since the extracted image, which is an image from which the blood vessel is extracted, is used as a filter, the area of the blood vessel and the diameter based on the area can be calculated extremely stably.

Furthermore, according to the embodiment described above, the detection unit 133 detects detection points by performing a filter process using a filter F on the thinned image. This configuration reduces the processing load required to detect detection points.

Furthermore, according to the embodiment described above, the group information identification unit 134 detects a specific pixel from the eight pixels surrounding the first center pixel in the first region. Next, the group information identification unit 134 repeats a process of detecting a new specific pixel from five pixels excluding the first center pixel and the pixels adjacent to the first center pixel among the eight pixels surrounding the second center pixel in the second region, and a process of detecting a new specific pixel from five pixels excluding the second center pixel and the pixels adjacent to the second center pixel among the eight pixels surrounding the third center pixel in the third region, thereby identifying group information. According to this configuration, the group information identification unit 134 can identify group information (thinned blood vessels connecting two detection points) at high speed and with a small processing load.

In the embodiment described above, the crossing point identification unit 137 identifies a pair of detection points that satisfies both the length condition and the angle condition as a pair of detection points that correspond to a crossing point. Alternatively, the crossing point identification unit 137 may identify a pair of detection points that satisfies only the length condition as a pair of detection points that correspond to a crossing point.

In the embodiment described above, the device 1 includes the extracted image generating unit 131 and the thinned image generating unit 132. Alternatively, the device may not include extracted image generation, or may not include the extracted image generating unit and the thinned image generating unit. In this case, the device does not generate extracted images and thinned images, but instead acquires extracted images and thinned images generated by other devices via a network, for example, and stores them in the storage unit.

Furthermore, in the embodiment described above, the first threshold value was the diameter of the blood vessel corresponding to the common line segment. Alternatively, the first threshold value in the present invention may be fixed to a predetermined value.

Furthermore, the second threshold in the present invention is not limited to 20°. That is, for example, the second threshold may be adjusted and set to an appropriate value while referring to the accuracy rate and the error rate.

Furthermore, the group information identification unit in the present invention may detect specific pixels using known image processing.

Furthermore, the memory unit in the present invention may store filter patterns corresponding to the endpoints.

Furthermore, the connection information acquisition unit in the present invention may determine the blood vessel length based on the blood vessel graph. That is, for example, the connection information acquisition unit in the present invention may determine the blood vessel length as the length of a line segment.

1 Image processing device 12 Storage unit 133 Detection unit 134 Group information identification unit 135 Blood vessel graph generation unit 136 Connection information acquisition unit 137 Intersection point identification unit

Claims (12)

1. An image processing device that identifies crossing points of blood vessels based on an image of the blood vessels, comprising:
a storage unit that stores a thinned image in which the blood vessels included in the captured image are thinned;
a detection unit that detects a plurality of detection points corresponding to any of the crossing points, branching points, and end points of the thinned blood vessels based on the thinned image;
a group information specifying unit that specifies group information indicating a pair of two detection points connected by the thinned blood vessel, among the plurality of detection points detected based on the thinned image;
a blood vessel graph generating unit that generates a blood vessel graph by connecting two of the detection points that constitute the pair by a line segment based on the specified pair information;
a connectivity information acquisition unit that acquires connectivity information of each of the detected detection points based on the blood vessel graph;
a crossing point identification unit that identifies a detection point corresponding to the crossing point among the plurality of detection points detected based on the connection information;
and
The intersection point identification unit is
extracting a plurality of branch detection points connected to three of the line segments from the plurality of detected detection points based on the connection information;
extracting, as a detection point pair, two branch detection points that are connected to a common line segment from among the extracted branch detection points based on the connection information;
identifying the branch detection points constituting the detection point pair as the detection points corresponding to the crossing point based on the connection information of the two branch detection points constituting the detection point pair;
13. An image processing device comprising: The link information acquisition unit
acquiring angles between the plurality of line segments connected to the detection point as the connection information based on the blood vessel graph;
acquiring, as the connection information, a length of the blood vessel connecting the two detection points based on the thinned image;
acquiring, as the connection information, a diameter of the blood vessel connecting the two detection points based on the thinned image;
2. The image processing device according to claim 1. The detection point pair is
A first detection point;
a second detection point connected to the line segment common to the first detection point;
Including,
The intersection point identification unit is
when the length of the blood vessel connecting the first detection point and the second detection point is smaller than a predetermined first threshold, the detection point pair is identified as the detection point corresponding to the intersection point.
3. The image processing device according to claim 2. The intersection point identification unit is
when an absolute value of a difference between the angle between two of the three line segments connected to the first detection point, excluding the common line segment, and the angle between two of the three line segments connected to the second detection point, excluding the common line segment, is smaller than a predetermined second threshold value, the detection point pair is identified as the detection point corresponding to the intersection point.
4. The image processing device according to claim 3. the first threshold value is the diameter of the blood vessel connecting the first detection point and the second detection point;
4. The image processing device according to claim 3. the storage unit stores an extracted image in which the blood vessels included in the captured image are extracted;
the connection information acquisition unit acquires the aperture as the connection information based on the thinned image and the extracted image;
3. The image processing device according to claim 2. The link information acquisition unit
extracting, from the thinned image, the thinned blood vessels that connect the two detection points that form the set;
performing a dilation process on the extracted blood vessel and a filtering process using the extracted image as a filter to obtain the diameter;
7. The image processing device according to claim 6 . The linking information is
Another detection point connected to the detection point;
The number of the line segments connected to the detection point; and
Including,
3. The image processing device according to claim 2. the storage unit stores a plurality of filters corresponding to the crossover points, the branch points, and the end points,
the detection unit detects the detection points by performing a filtering process on the thinned image using the filter.
2. The image processing device according to claim 1. The thinned image is
A specific pixel indicating the thinned blood vessel;
non-specific pixels indicating a background other than the thinned blood vessels;
It is composed of
The group information identification unit
In a first region of 3×3 pixels having the detection point as a first center pixel, the specific pixel is detected from eight pixels surrounding the first center pixel, and then:
a process of detecting a new specific pixel from five pixels, excluding the first central pixel and a pixel adjacent to the first central pixel, among eight pixels surrounding the second central pixel in a second region of 3×3 pixels having the detected specific pixel as a second central pixel;
a process of detecting a new specific pixel from five pixels, excluding the second center pixel and a pixel adjacent to the second center pixel, among eight pixels surrounding the third center pixel in a third region of 3×3 pixels having the detected specific pixel as a third center pixel;
By repeating the above, the group information is identified.
2. The image processing device according to claim 1. A computer is caused to function as the image processing device according to claim 1.
2. An image processing program comprising: 1. An image processing method executed by an image processing device for identifying crossing points of blood vessels based on an image of the blood vessels, the method comprising:
The image processing device includes:
a storage unit that stores a thinned image in which the blood vessels included in the captured image are thinned;
Equipped with
The image processing device,
detecting a plurality of detection points corresponding to any of the crossing points, branching points, and end points of the thinned blood vessels based on the thinned image;
specifying, based on the thinned image, group information indicating a group of two of the detection points that are connected by the blood vessel, among the plurality of detection points detected;
generating a blood vessel graph by connecting each of the two detection points constituting the pair with a line segment based on the identified pair information;
acquiring connectivity information for each of the detected detection points based on the blood vessel graph;
extracting a plurality of branch detection points connected to three of the line segments from among the plurality of detected detection points based on the connection information;
extracting, as a detection point pair, two of the branch detection points that are connected to a common line segment from among the extracted branch detection points based on the connection information;
identifying the branch detection points constituting the detection point pair as the detection points corresponding to the crossing point based on the connection information of the two branch detection points constituting the detection point pair;
Including,
13. An image processing method comprising:

Images