JP6964162B2 - Image processing equipment and water droplet removal system - Google Patents

Description

The present invention relates to an image processing apparatus and a water droplet removal system.

Conventionally, a camera as an imaging unit that is mounted on a vehicle and captures an image of the surroundings of the vehicle is known, and the image captured by the camera is provided to, for example, a driver.

Water droplets such as raindrops and snow may adhere to the lens of the camera described above due to the influence of rain or snowfall, for example. Therefore, in order to remove water droplets, a water droplet removing device has been proposed in which the driver manually operates the device to eject compressed air or the like to the lens of the camera (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-037239

However, when manually operating the water droplet removing device, it is very difficult to constantly remove water droplets from the camera. Therefore, there is a demand for a technique that can automatically determine the adhesion of water droplets to a camera based on a captured image. In particular, in a system that constantly requires recognition of captured images, such as a system that detects white lines and marks on the road with a camera and recognizes the position of the vehicle with high accuracy, if water droplets adhere to the camera, it will self. There is a risk that the recognition accuracy of the vehicle position will decrease.

One aspect of the embodiment is made in view of the above, and an object of the present invention is to provide an image processing device and a water droplet removing system capable of determining adhesion of water droplets to an image pickup unit based on an image captured image. And.

In one aspect of the embodiment, the image processing apparatus includes a detection unit, a calculation unit, and a determination unit. The detection unit detects a candidate region that is a candidate for the water droplet region affected by the water droplets adhering to the imaging unit, based on the intensity of the edge of each pixel of the captured image acquired by the imaging unit. The calculation unit calculates the orientation of the edge of each pixel in the candidate region. The determination unit determines whether or not the candidate region is the water droplet region based on the orientation of the edge.

According to one aspect of the embodiment, it is possible to determine the adhesion of water droplets to the image pickup unit based on the captured image.

FIG. 1 is a diagram showing a determination mechanism of the image processing apparatus according to the embodiment. FIG. 2 is a block diagram showing a configuration of an image processing device and a water droplet removing system according to the first embodiment. FIG. 3 is a diagram showing an outline of image processing according to the first embodiment. FIG. 4 is a flowchart showing a processing procedure executed by the image processing apparatus according to the first embodiment. FIG. 5 is a diagram showing an outline of image processing according to a modified example of the first embodiment. FIG. 6 is a flowchart showing a processing procedure executed by the image processing apparatus according to the modified example of the first embodiment. FIG. 7 is a diagram showing an outline of image processing according to another modification of the first embodiment. FIG. 8 is a diagram showing an outline of image processing according to still another modification of the first embodiment. FIG. 9 is a block diagram showing the configuration of the image processing apparatus and the water droplet removing system according to the second embodiment. FIG. 10 is a diagram showing an outline of image processing according to the second embodiment. FIG. 11 is a flowchart showing a processing procedure executed by the image processing apparatus according to the second embodiment. FIG. 12 is a diagram showing an outline of image processing according to a modified example of the second embodiment. FIG. 13 is a flowchart showing a processing procedure executed by the image processing apparatus according to the modified example of the second embodiment.

Hereinafter, embodiments of the image processing apparatus and the water droplet removing system disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments shown below. Further, in the following, a case where the camera is arranged at the rear of the vehicle and determines the water droplets adhering to the camera that images the rear of the vehicle will be described as an example.

Further, in the following, the determination mechanism of the image processing apparatus according to the embodiment will be described with reference to FIG. 1, and then the specific configuration of the image processing apparatus and the water droplet removal system according to the embodiment will be described with reference to FIGS. 2 and later. I decided to.

(Determination mechanism of embodiment)
FIG. 1 is a diagram showing a determination mechanism of the image processing apparatus according to the embodiment. FIG. 1 shows an example of an captured image 100 in which the surroundings of a vehicle are imaged with water droplets attached to the camera. The captured image 100 includes a water droplet region (image of water droplets) that is blurred due to the influence of water droplets adhering to the lens of the camera. The water droplet region includes a gray-blurred substantially circular region and a black-blurred substantially arc-shaped region surrounding the gray-blurred region.

The gray, blurry, substantially circular region is due to the thin central portion of the water droplet, and the black, blurry, substantially arcuate region is due to the thick edge portion of the water droplet.

By the way, in the conventional water droplet removing device, it is necessary to manually operate the device. Therefore, in a system that detects white lines and marks on the road with a camera and recognizes the position of the vehicle with high accuracy, if the captured image becomes unclear due to the inclusion of the water droplet area in this way, a water droplet removal device is manually used. White lines and marks on the road cannot be detected until you operate. As a result, the recognition accuracy of the position of the own vehicle may decrease.

Therefore, in the image processing apparatus according to the embodiment, it is determined whether or not a water droplet region is included in the captured image obtained by the camera, and it is determined whether or not water droplets are attached to the camera. And said. Further, the water droplet removal system according to the embodiment is configured so that water droplets can be automatically removed based on the determination result of the image processing device. Hereinafter, the determination mechanism of the present embodiment will be described in detail.

First, a well-known edge detection process is performed on the above-mentioned captured image 100 to detect the edge intensity (magnitude of change in brightness) of each pixel. Then, by extracting a set of pixels having an edge intensity in a specific range, a candidate region 1 that is a candidate for a water droplet region is detected. As shown in the lower part of FIG. 1, the candidate region 1 corresponds to the above-mentioned arc-shaped black and blurred region. This is because it has been found by experiments and the like that the edge strength of the pixels included in the black and blurred region is within a specific range of a predetermined first threshold value or more and a predetermined second threshold value or less. Then, the circle 1a corresponding to the candidate region 1 can be estimated based on the plurality of points of interest included in the detected candidate region 1.

Then, if the intensity of the edge of the central region 2a within a predetermined range from the center 2 of the estimated circle 1a becomes lower than the predetermined threshold value, the candidate region 1 is a water droplet region (to be exact, a part of the water droplet region). It is determined that there is. This is because the change in brightness in the gray and blurred region described above is relatively small, and the frequency region is relatively low.

Further, the black and blurred region of the water droplet region has a width equal to or larger than a predetermined length due to the shape of the thick edge portion of the water droplet. The edges of the pixels in the black and blurred region are oriented so as to converge with each other toward the center of the water droplet region. Therefore, it is also possible to determine whether or not the candidate region is a water droplet region based on the orientation of the edges of the pixels included in the candidate region 1.

The orientation of the edges is indicated by a vector based on the intensity of the edges of the two axes orthogonal to each other along the horizontal and vertical directions in the captured image. For example, the direction of the edge is a positive vector from the one with the lower brightness to the one with the higher brightness.

In the first embodiment shown below, a case of determining whether or not water droplets are attached to the camera will be described based on the strength of the edge inside the circle 1a estimated from the candidate region 1 described above. Further, in the second embodiment shown below, a case of determining whether or not water droplets are attached to the camera will be described based on the orientation of the edges of the pixels in the candidate region 1. In any of the embodiments, it is possible to satisfactorily determine whether or not water droplets are attached to the camera based on the captured image.

(First Embodiment)
Hereinafter, a specific configuration of the embodiment will be described. FIG. 2 is a block diagram showing a configuration of an image processing device and a water droplet removing system according to the first embodiment.

The water droplet removing system 11 shown in FIG. 2 includes an image processing device 20 and a water droplet removing device 30.

The image processing device 20 determines whether or not water droplets are attached to the camera 10 based on the captured image 100 acquired by the camera 10 which is an imaging unit, and sends the determination result to the water droplet removing device 30. The detailed configuration of the image processing device 20 will be described later.

The water droplet removing device 30 automatically removes water droplets adhering to the camera 10 which is an imaging unit based on the determination result sent from the image processing device 20. The water droplet removing device 30 includes, for example, an air compression unit, a hose, and a nozzle, and ejects compressed air generated by the air compression unit from the nozzle toward the camera 10 via the hose to remove water droplets. The water droplet removing device 30 is not limited to this, and for example, the washer fluid may be ejected toward the camera 10 or the camera 10 may be wiped with a camera wiper.

The water droplet removing system 11 according to the first embodiment determines whether or not water droplets are attached to the camera 10 based on the captured image 100, and automatically removes the water droplets attached to the camera 10 based on the determination result. Can be removed. Therefore, water droplets can be constantly removed from the camera 10.

The camera 10 from which water droplets are removed by the water droplet removal system 11 may be, for example, an optical sensor that acquires an image through a lens or information on a target around a vehicle. Specifically, various optical sensors such as a front camera that images the front of the vehicle, a side camera that images the side of the vehicle, and a back camera that images the rear of the vehicle can be targeted.

The image processing device 20 includes a control unit 21 and a storage unit 25. The control unit 21 performs edge detection processing of the captured image 100, estimation processing based on the detection result, determination processing of whether or not water droplets are attached, and the like. The storage unit 25 stores in advance various processing conditions and various parameters used when performing detection processing, estimation processing, determination processing, and the like in the control unit 21, and appropriately sends the stored various processing conditions and various parameters to the control unit 21. .. Further, the storage unit 25 sends the detection result and the estimation result performed in the control unit 21 from the control unit 21, and stores the detection result and the estimation result.

The control unit 21 includes a detection unit 22, an estimation unit 23, and a determination unit 24.

The detection unit 22 receives the captured image 100 acquired by the camera 10 which is an imaging unit. Then, the detection unit 22 performs edge detection processing on the captured image 100 to derive the edge strength of each pixel. Then, the detection unit 22 detects a set of pixels in which the intensity of the derived edge is in a specific range as a candidate region 1 which is a candidate for a blurred water droplet region under the influence of water droplets. When the candidate region 1 is a water droplet region, the candidate region 1 has a substantially arc shape. Therefore, it is desirable to detect only the arcuate region having a substantially arc shape as the candidate region 1. Then, the detection unit 22 stores the derivation result of the edge strength and the detection result of the candidate area 1 in the storage unit 25. The candidate area 1 stored in the storage unit 25 is a determination target by the determination unit 24.

The estimation unit 23 receives the detection result of the candidate region 1. Then, the estimation unit 23 estimates the corresponding circle 1a so as to inscribe the candidate region 1 based on the detection result of the candidate region 1. Then, the estimation unit 23 stores the estimation result of the circle 1a in the storage unit 25.

The determination unit 24 receives the edge strength derivation result, the detection result of the candidate region 1, the estimation result of the circle 1a, and the like stored in the storage unit 25. Then, the determination unit 24 determines whether or not the candidate region 1 stored as the determination target in the storage unit 25 is a water droplet region based on the above-mentioned extraction result, detection result, estimation result, and the like, and determines whether or not the candidate region 1 is a water droplet region. Judge the adhesion of. Then, the determination result is sent to the water droplet removing device 30.

Hereinafter, the image processing in the control unit 21 will be described more specifically with reference to FIGS. 3 and 4.

FIG. 3 is a diagram showing an outline of image processing according to the first embodiment. FIG. 4 is a flowchart showing a processing procedure executed by the image processing apparatus according to the first embodiment.

The detection unit 22 receives the captured image 100 acquired by the camera 10 which is an imaging unit. Next, the detection unit 22 detects an edge by using a well-known edge detection method such as differentiating the brightness of each pixel with respect to the received captured image 100. Then, the detection unit 22 derives the intensity of the edge of each pixel of the captured image 100. In this example, the edge is detected using the brightness of each pixel, but the edge may be detected for each color component value (R value, G value, B value, etc.) of each pixel. .. In this case, the added value or the maximum value of the edge strength for each color component value may be the edge strength of the pixel.

Next, the detection unit 22 sets a set of pixels in a specific range in which the strength of the derived edge is equal to or higher than a predetermined first threshold value and equal to or lower than a predetermined second threshold value as a candidate region 1 as a candidate for the water droplet region. Detect (step S101). At this time, it is desirable to detect only the arcuate region having a substantially arcuate shape as the candidate region 1.

Next, the estimation unit 23 estimates the corresponding circle 1a so that the candidate area 1 is inscribed based on the detected candidate area 1 (step S102). In step S102, for example, the following method can be used as a method for estimating the circle 1a.

As shown in FIG. 3, the estimation unit 23 includes a plurality of points of interest 3a to 3d (hereinafter, collectively referred to as points of interest 3), which are intersections of the rectangle 1b inscribed with the detected candidate area 1 and the candidate area 1. ) Is extracted. Here, as the rectangle 1b, for example, a rectangle whose sides extend in the horizontal direction and the vertical direction can be selected. In that case, the points of interest 3 are the smallest point in the horizontal direction (point of interest 3a), the largest point in the vertical direction (point of interest 3b), and the largest point in the horizontal direction (point of interest 3b) in the candidate area 1. 3c) and the smallest point in the vertical direction (point of interest 3d) are extracted, respectively. The rectangle 1b does not necessarily have to be a rectangle whose sides extend in the horizontal and vertical directions, and a rectangle whose sides extend in an inclined direction from the horizontal and vertical directions may be selected.

Then, if the center 2 located at substantially equidistant distances from the extracted plurality of points of interest 3 is estimated, the circle 1a is estimated as a circle passing through the plurality of points of interest 3 centered on the center 2. The center 2 estimated here does not have to be completely equidistant from the plurality of points of interest 3. For example, when points that are completely equidistant are located between pixels, a plurality of pixels included in the periphery may be estimated as the center 2. Further, the estimated circle 1a does not have to pass through all the points of interest 3, and for example, some points of interest 3 may pass in the vicinity thereof.

Next, the determination unit 24 determines whether or not the candidate region is a water droplet region based on the estimated edge strength in the circle 1a. For example, as shown in FIG. 3, the determination unit 24 determines the strength of the edge of the central region 2a within a predetermined range from the center 2 (step S103).

Here, when the average edge strength in the central region 2a is lower than a predetermined threshold value (step S103, Yes), it is determined that the candidate region 1 is a water droplet region (more accurately, a part of the water droplet region). (Step S104). That is, the determination unit 24 determines that water droplets are attached to the camera 10.

Further, when the average edge strength in the central region 2a is higher than a predetermined threshold value (step S103, No), it is determined that the candidate region 1 is not a water droplet region (step S105).

Then, the determination unit 24 outputs the determination result to the water droplet removing device 30 (step S106), and ends the process.

The image processing apparatus 20 according to the first embodiment determines the adhesion of water droplets by utilizing the fact that the edge strength of the central region 2a is low among the changes in the brightness in the circle 1a.

As the central region 2a in the first embodiment, for example, a circle having a radius smaller than the radius 4 of the circle 1a can be used with the center 2 as the center. For the central region 2a, it is preferable to use a circle smaller than 1/2 of the radius 4 with the center 2 as the center, and more preferably to use a circle smaller than 1/4 of the radius 4.

In the first embodiment, the average edge strength in the central region 2a is determined, but only the edge strength of one pixel corresponding to the center 2 may be determined instead of the central region 2a. .. In this case, the determination unit 24 determines that the candidate region 1 is a water droplet region when the strength of the edge of the center 2 is lower than the predetermined threshold value, and when the strength of the edge of the center 2 is higher than the predetermined threshold value. May determine that the candidate region 1 is not a water droplet region.

In step 103 of the first embodiment, the determination unit 24 makes a determination using the strength of the edge derived by the detection unit 22. However, the determination unit 24 may make a determination using a parameter different from the edge intensity derived by the detection unit 22 (for example, a parameter indicating a substantially edge intensity such as a spatial frequency).

Here, it is assumed that deposits other than water droplets adhere to the camera 10. In this case, the candidate region 1 as a set of pixels whose edge strength is in a specific range does not have an arc shape, but has a rectangular shape or a circular shape, for example. Further, the edge strength does not decrease at the position corresponding to the central region 2a of the candidate region 1. Therefore, when the candidate region 1 has a rectangular shape or a circular shape, or when the strength of the edge of the central region of the candidate region 1 is high, it can be determined that it is not water droplets adhering to the camera 10.

Therefore, according to the image processing apparatus according to the first embodiment, it is possible to determine with high accuracy whether or not the deposits adhering to the camera are water droplets.

(Modification example 1)
The image processing apparatus 20 according to the first modification of the first embodiment will be described with reference to FIG.

In the embodiments shown below, components different from the embodiments already shown will be mainly described, and the same reference numerals will be given to the configurations common to the embodiments already shown, and the description thereof will be omitted. There is.

In step S102 in the first embodiment, the circle 1a is estimated. In the first modification, the determination unit 24 determines whether or not the radius 4 of the estimated circle 1a is equal to or less than a predetermined reference radius. The reason is as follows.

When the size of the water droplets adhering to the camera 10 becomes a predetermined size or more, the water droplets tend to fall due to gravity, vibration of the vehicle, or the like. Therefore, a candidate region having a predetermined size or larger can be regarded as, for example, a region related to another deposit, not a water droplet region.

Then, when the radius 4 of the circle 1a is equal to or less than the reference radius, the determination unit 24 proceeds to step S103, and the above-described processing is performed. Therefore, even in this case, if the strength of the edge in the central region 2a is lower than the predetermined threshold value, the determination unit 24 determines that the candidate region 1 is a water droplet region.

On the other hand, when the radius 4 of the circle 1a exceeds the reference radius, it is determined that the candidate region 1 is not a water droplet region.

The above-mentioned reference radius value is calculated from the size of the lens of the camera 10, the size and resolution of the image sensor, and the like, based on the physical size of the deposit.

According to the image processing apparatus according to the first modification, the water droplets adhering to the camera 10 can be determined with higher accuracy.

(Modification 2)
The image processing apparatus 20 according to the second modification of the first embodiment will be described with reference to FIGS. 5 and 6.

FIG. 5 is a diagram showing an outline of image processing according to the second modification of the first embodiment. FIG. 6 is a flowchart showing a processing procedure executed by the image processing apparatus according to the second modification of the first embodiment.

In the second modification, the validity of the candidate region 1 is easily determined by utilizing the fact that the black and blurred region (see FIG. 1) of the water droplet region has a width equal to or larger than a predetermined length.

In the second modification, the detection unit 22 selects the point of interest 3 which is the intersection of the rectangle 1b inscribed with the candidate area 1 and the candidate area 1 (step S201). For example, in FIG. 5, points of interest 3a and 3b are selected.

Next, the detection unit 22 moves the selected point of interest 3 toward the center 2 by a predetermined reference distance 51 (step S202). In FIG. 5, the points where the points of interest 3a and 3b are moved by the reference distance 51 toward the center 2 are the points 61a and 61b, respectively.

Next, the detection unit 22 determines whether or not the moved point is included in the candidate area 1 (step S203).

Here, when the moved point is included in the candidate region 1 (step S203, Yes), the detection unit 22 determines the candidate region 1 because the detected candidate region 1 has a high probability of being a water droplet region. It is held in the storage unit 25 as a determination target of the unit 24 (a target for determining whether or not it is a water droplet region) (step S204). For example, in FIG. 5, since both the moved points 61a and 61b are included in the candidate area 1, the candidate area 1 is held as a determination target of the determination unit 24.

Further, when the moved point is not included in the candidate region 1 (step S203, No), the detection unit 22 sets the candidate region 1 because the detected candidate region 1 is unlikely to be a water droplet region. It is deleted from the storage unit 25 and excluded from the determination target of the determination unit 24 (step S205). Then, the process ends.

In the second modification, the value of the reference distance 51 may be set by obtaining the width of the black and blurred region (see FIG. 1) of the standard water droplet region by an experiment or the like.

In the second modification, a region appropriate as the candidate region 1 that is a candidate for the water droplet region can be easily selected. Therefore, it is possible to reduce the processing amount of the entire control unit 21.

Therefore, according to the image processing device 20 according to the second modification, the water droplets adhering to the camera 10 can be determined at a higher speed.

(Modification example 3)
In the image processing apparatus 20 according to the second modification, the detection accuracy of the candidate region 1 can be improved by adding a point that moves from the point of interest 3. This modification 3 will also be described with reference to FIG.

In the third modification, the candidate region 1 is detected by utilizing the fact that the edge strength of the inner and outer pixels of the black and blurred region (see FIG. 1) of the water droplet region is not in the specific range.

In the third modification, the detection unit 22 moves the selected point of interest 3 toward the center 2 by a predetermined reference distance 52 (points 62a, 62b). Further, the detection unit 22 moves the selected point of interest 3 by a predetermined reference distance 53 so as to move away from the center 2 (point 63a, point 63b).

Next, the detection unit 22 determines whether or not the moved point is included in the candidate area 1. Here, when the moved point is not included in the candidate area 1, the detection unit 22 holds the detected candidate area 1 in the storage unit 25 as a determination target of the determination unit 24. For example, in FIG. 5, since the moved points 62a, 62b, 63a, and 63b are not included in the candidate area 1, the candidate area 1 is held as a determination target of the determination unit 24.

In the third modification, when the plurality of moved points are not all included in the candidate area 1, the detection unit 22 holds the detected candidate area 1 in the storage unit 25 as a determination target of the determination unit 24. There is. On the other hand, even if all of the moved plurality of points are not included in the candidate area 1, for example, when a majority of the moved points are not included in the candidate area 1. The detection unit 22 may hold the detected candidate area 1 in the storage unit 25 as a determination target of the determination unit 24.

In the third modification, the value of the reference distance 52 may be set by obtaining the width of the black and blurred region (see FIG. 1) of the standard water droplet region and the size of the central region thereof by an experiment or the like. .. Further, in the modification example 3, the value of the reference distance 53 may be set so that the point after movement is sufficiently separated from the black and blurred region.

In the modified example 3, the detection accuracy of the candidate region 1 can be improved by increasing the number of points to be moved from the point of interest 3 as compared with the modified example 2. Therefore, even when the region caused by noise is detected as the candidate region 1, only the candidate region 1 caused by water droplets can be accurately selected as the determination target of the determination unit 24.

Therefore, according to the image processing device 20 according to the modification 3, the water droplets adhering to the camera 10 can be determined with higher accuracy.

(Modification example 4)
The image processing apparatus 20 according to the fourth modification of the first embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing an outline of image processing according to the fourth modification of the first embodiment.

Since water droplets are often attracted downward by gravity to form a thick edge, the shape of the candidate area 1 corresponding to the black and blurred area of the water droplet area caused by the edge of the water droplet is approximately symmetrical. There are many. Therefore, in the modified example 4, it is evaluated whether or not the shape of the candidate region 1 is a shape close to left-right symmetry.

In the modification 4, the estimation unit 23 selects three points of interest 3 which are intersections of the rectangle 1b inscribed with the candidate area 1 and the candidate area 1. For example, in the case of the downwardly convex candidate region 1 as shown in FIG. 7, the point (3a) where the horizontal direction is the minimum, the point (3b) where the vertical direction is the maximum, and the horizontal direction are the maximum. The point (3c) can be selected as the point of interest 3.

Further, the candidate region 1 may have a shape that is convex upward. This is a case where, for example, water droplets adhere to the lower part of the captured image 100 and the black and blurred region is imaged only on the upper side. In the case of the candidate region 1 which is convex upward as described above, the point where the horizontal direction is the minimum, the point where the horizontal direction is the maximum, and the point where the vertical direction is the minimum can be selected as points of interest.

In any of the above cases, in order to evaluate whether or not the shape is nearly symmetrical, the three points adjacent to each other along the candidate region 1 arranged along the outer edge in the direction in which the candidate region 1 protrudes are the points of interest 3. You can select as.

Next, the detection unit 22 is the same as the first distance 54, which is the distance between the attention points 3a and the attention points 3b adjacent to each other along the candidate region 1 among the three selected points of interest 3a, 3b, and 3c. It is determined whether or not the ratio of the second distance 55, which is the distance between the points of interest 3b and the points of interest 3c adjacent to each other along the candidate region 1, is out of a predetermined range.

Here, when the ratio of the first distance 54 and the second distance 55 is within a predetermined range, it can be determined that the candidate region 1 has a shape close to left-right symmetry. In that case, the detection unit 22 holds the detected candidate area 1 in the storage unit 25 as a determination target of the determination unit 24.

On the other hand, when the ratio of the first distance 54 to the second distance 55 is out of the predetermined range, the detection unit 22 deletes the area detected as the candidate area 1 from the storage unit 25, and the determination unit 24 Exclude from the judgment target of.

In the modified example 4, as in the modified example 2, a region appropriate as the candidate region 1 that is a candidate for the water droplet region can be easily selected. Therefore, it is possible to reduce the processing amount of the entire control unit 21.

Therefore, according to the image processing apparatus according to the modified example 4, the water droplets adhering to the camera 10 can be determined at a higher speed.

(Modification 5)
The image processing apparatus 20 according to the fifth modification of the first embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an outline of image processing according to the fifth modification of the first embodiment.

As shown in FIG. 8, based on the detected candidate region 1, the corresponding circle 1a can be estimated so as to be inscribed in the candidate region 1. When the candidate region 1 has an arc shape, the outer edge portion of the candidate region 1 is located in the vicinity of the circle 1a. Therefore, in the modified example 5, it is evaluated whether or not the outer edge portion of the candidate region 1 is located in the vicinity of the circle 1a.

In the modified example 5, the detection unit 22 selects the above-mentioned point of interest 3. In FIG. 8, points of interest 3a, 3b, and 3c are selected. In the modified example 5, as shown in FIG. 8, three points of interest may be selected, or one or two points may be selected. It is preferable to select three points of interest 3 because a wider range of outer edges of the candidate area 1 can be evaluated.

Next, the detection unit 22 moves the selected point of interest 3 along the outer edge of the candidate region 1 by a predetermined distance unit. For example, in FIG. 8, the three selected points of interest 3 are moved clockwise, respectively. Then, the points where the attention point 3a is moved are the points 64a and 64b, the points where the attention point 3b is moved are the points 64c and 64d, and the points where the attention point 3c is moved are the points 64e and 64f.

Next, the detection unit 22 determines whether or not the ratio of the distance 56 between the moved point 64 and the center 2 and the radius 4 of the circle 1a is out of a predetermined range. For example, in FIG. 8, it is determined whether or not the ratio of the distances 56a to 56f corresponding to the moved points 64a to 64f and the radius 4 is out of the predetermined range, respectively.

Here, when the ratio of the distance 56 between the moved point 64 and the center 2 and the radius 4 is within a predetermined range, the detection unit 22 stores the detected candidate area 1 as the determination target of the determination unit 24. It is held in the portion 25. When the ratio of the distance 56 to the radius 4 is out of the predetermined range, the detection unit 22 deletes the area detected as the candidate area 1 from the storage unit 25, and determines the determination unit 24. Exclude from the target.

In the modified example 5, when a region having a collapsed shape due to deposits other than water droplets is detected as the candidate region 1, only the candidate region 1 due to the water droplets is accurately selected as the determination target of the determination unit 24. It becomes possible to do.

Therefore, according to the image processing apparatus according to the modified example 5, the water droplets adhering to the camera 10 can be determined with higher accuracy.

Further, the estimation accuracy of the circle 1a corresponding to the candidate region 1 can be improved by using the point moved from the attention point 3 along the outer edge of the candidate region 1 in the modification 5. Hereinafter, a specific method of this image processing will be described.

The estimation unit 23 extracts center point candidates at substantially equidistant distances from the selected plurality of points of interest 3a, 3b, and 3c. Further, another center point candidate located at substantially equidistant distances from the plurality of points 64a, 64c, 64e moved by a predetermined distance from the points of interest 3a, 3b, and 3c is extracted. Further, another center point candidate located at substantially equidistant distances from the plurality of points 64b, 64d, 64f moved by another predetermined distance from the points of interest 3a, 3b, and 3c is extracted. In this way, a plurality of center point candidates are extracted based on the points moved from the attention point 3 along the outer edge of the candidate region 1.

Next, the estimation unit 23 estimates the center 2 based on the extracted plurality of center point candidates. For example, the point at the center of gravity of the plurality of center point candidates may be detected as the center 2, or the point with the highest number of estimations among the plurality of center point candidates may be estimated as the center 2.

By estimating the center 2 based on a plurality of center point candidates in this way, it is possible to more accurately detect the position of the center 2 even when the position of the center 2 is displaced due to noise or the like. It becomes.

(Second Embodiment)
Subsequently, the configuration of the image processing device 20a according to the second embodiment will be described. In the second embodiment, among the determination mechanisms shown in FIG. 1, the determination of the adhesion of water droplets to the camera 10 is performed based on the direction of the edge of each pixel in the candidate region 1.

FIG. 9 is a block diagram showing the configuration of the image processing device 20a and the water droplet removing system 11a according to the second embodiment. Unlike the first embodiment, in the second embodiment, the control unit 21a includes a calculation unit 26 instead of the estimation unit 23.

The control unit 21a according to the second embodiment includes a detection unit 22, a calculation unit 26, and a determination unit 24. The detection unit 22 is the same as that of the first embodiment.

The calculation unit 26 receives the detection result of the candidate region 1. Then, the calculation unit 26 calculates the direction of the edge of each pixel in the candidate region 1. Then, the calculation unit 26 stores the calculation result of the edge orientation of each pixel in the candidate area 1 in the storage unit 25.

The determination unit 24 receives the result of deriving the strength of the edge, the result of detecting the candidate region 1, the result of calculating the direction of the edge, and the like. Then, the determination unit 24 determines the adhesion of water droplets on the camera 10 based on the above-mentioned extraction result, detection result, calculation result, and the like, and sends the determination result to the water droplet removing device 30.

Hereinafter, the image processing in the control unit 21a will be described more specifically with reference to FIGS. 10 and 11.

FIG. 10 is a diagram showing an outline of image processing according to the second embodiment. FIG. 11 is a flowchart showing a processing procedure executed by the image processing apparatus 20a according to the second embodiment.

Similar to the first embodiment, the detection unit 22 derives the intensity of the edge of each pixel in the captured image 100 acquired by the camera 10, and detects the candidate region 1 based on the strength of the derived edge (step). See S101).

Next, the calculation unit 26 calculates the direction of the edge of each pixel in the candidate region 1 based on the intensity of the edge of the captured image 100. For example, the calculation unit 26 determines the edge of each pixel in the attention point 3 which is the intersection of the rectangle 1b inscribed with the candidate area 1 and the candidate area 1 and the peripheral region within a predetermined range from the attention point 3. The orientation is calculated (step S301). For example, as shown in FIG. 10, with the attention point 3a, which is the smallest point in the horizontal direction in the candidate area 1, as a base point, a peripheral area within plus or minus 5 pixels from the attention point 3a in the candidate area 1 is selected. The direction of the edge of the pixel of the point of interest 3a and the peripheral region thereof is calculated.

Further, the calculation unit 26 evaluates the variation in the direction of the edge of each pixel of the point of interest 3 and the peripheral region thereof (step S302).

Next, the determination unit 24 determines whether or not water droplets are attached to the camera 10 based on the orientation of the edges of each pixel in the candidate region 1. For example, in the second embodiment, the determination unit 24 determines whether or not the variation in the direction of the edge of each pixel of the point of interest 3 and the peripheral region thereof is within a predetermined range (step S303).

When the detected candidate region 1 corresponds to a black and blurred substantially arc-shaped region (edge of a water droplet) among the attached water droplet regions, the direction of the edge of each pixel within a predetermined range is constant. It tends to go in the direction (about the direction of the center of the circle). For example, as shown in FIG. 10, when the attention point 3a, which is the smallest point in the horizontal direction, is used as the base point, the direction of the edge of the peripheral region tends to be positive in the horizontal direction.

Therefore, in the determination unit 24, when the variation in the direction of the edge of each pixel of the attention point 3 and the peripheral region thereof is within a predetermined range (step S303, Yes), the candidate region 1 is a water droplet region (to be exact). , Part of the water droplet region) (step S304). That is, the determination unit 24 determines that water droplets are attached to the camera 10.

Further, when the variation in the edge orientation of each pixel is out of the predetermined range (step S303, No), it is determined that the candidate region 1 is not a water droplet region (step S305). Then, the determination unit 24 outputs the determination result to the water droplet removing device 30 (step S306), and ends the process.

In the second embodiment, the area within the range of plus or minus 5 pixels from the point of interest 3a in the candidate area 1 is selected as the peripheral area. However, the peripheral area according to the second embodiment is not limited to the range of plus or minus 5 pixels. The peripheral area may be selected so that the variation in the direction of the edges is within a predetermined range. Further, although the attention point 3a is selected as the base point of the peripheral region, another attention point may be selected, or the pixels in the candidate region 1 other than the attention point may be selected.

Further, in the second embodiment, the variation in the direction of the edge in the point of interest 3 and the peripheral region thereof may be calculated by calculating the variance of the direction of the edge of each pixel.

In the second embodiment, it is possible to determine the adhesion of water droplets without estimating the circle 1a or the center 2 as in the first embodiment. Therefore, the processing amount of the entire control unit 21a can be reduced.

Therefore, according to the image processing device 20a according to the second embodiment, water droplets adhering to the camera 10 can be determined at high speed.

(Modification 6)
The image processing apparatus 20a according to the sixth modification of the second embodiment will be described with reference to FIGS. 12 and 13.

FIG. 12 is a diagram showing an outline of image processing according to the sixth modification of the second embodiment. FIG. 13 is a flowchart showing a processing procedure executed by the image processing apparatus 20a according to the sixth modification of the second embodiment.

In the modification 6, it is determined whether or not the candidate region 1 is a water droplet region by utilizing the fact that the edges of the pixels included in the black and blurred region of the water droplet region are oriented so as to converge with each other. ..

In the modification 6, the calculation unit 26 selects one point in the candidate area 1 as a starting point (step S401). For example, as shown in FIG. 12, the pixel 70a, which is the starting point, is selected. Then, the calculation unit 26 calculates the direction 71a of the edge of the pixel 70a, which is the starting point (step S402).

Next, the calculation unit 26 performs an end point detection process for detecting an end point located at a predetermined position with respect to the start point (step S403). For example, as shown in FIG. 12, the pixel 70b, which is located at a predetermined interval 72 from the pixel 70a which is the start point and is perpendicular to the edge direction 71a of the pixel 70a (start point), is detected as the end point. Then, the direction 71b of the edge of the pixel 70b, which is the end point, is calculated.

As shown in FIG. 12, in the modification 6, the pixel 70b as the end point is selected from the 3 o'clock position of the pixel 70a as the start point, but it may be selected from the 9 o'clock position. In the repeated end point detection process described later, the end point at the same clock position as the previous time may be selected for the new start point.

Next, the determination unit 24 performs an end point determination process for determining whether or not the direction of the edge of the start point and the direction of the edge of the end point are oriented so as to converge with each other (step S404). Specifically, as shown in FIG. 12, the edge orientation 71b of the pixel 70b located at the 3 o'clock position is in a positive direction counterclockwise with respect to the edge orientation 71a of the pixel 70a, and is at a predetermined angle. When it is within the difference, it can be determined that the directions of the edges of each other are converged.

In the end point detection process, when the end point is selected from the 9 o'clock position with respect to the start point, the direction of the edge of the end point is positive in the clockwise direction with respect to the direction of the edge of the start point, and a predetermined angle difference is obtained. When it is inside, it can be determined that the directions of the edges of each other are converged.

Then, when the direction of the edge of the end point with respect to the direction of the edge of the start point is within a predetermined angle difference (step S405, Yes), the determination unit 24 increases the number of processes by 1 and stores it in the storage unit 25 (step). S406).

Next, the determination unit 24 determines whether or not the number of processes stored in the storage unit 25 is equal to or greater than a predetermined number (step S407). Then, when the number of treatments is equal to or greater than a predetermined number (step S407, Yes), it is determined that the candidate region 1 is a water droplet region (step S408).

Further, in step S407, when the number of processes is less than a predetermined number (steps S407, No), the determination unit 24 processes the calculation unit 26 as the end point in the previous processes (here, the process). Starting from the point 70b), an instruction is given to perform the end point detection process again.

In the modified example 6, as shown in FIG. 12, for example, when the pixel 70a is set as the start point, the pixel 70b is detected by the end point detection process. Then, when the end point determination process of the pixel 70b with respect to the pixel 70a is OK, the number of processes is increased by 1. Then, when the number of processes has not reached a predetermined number, a new end point detection process is performed starting from the pixel 70b. As a result, the pixel 70c is detected as a new end point. Then, using the edge orientation 71c of the pixel 70c, the end point determination process of the pixel 70c with respect to the pixel 70b is performed again.

As described above, by repeatedly performing the end point detection process and the end point determination process, a plurality of pixels arranged in the candidate area 1 along the candidate area 1 at a predetermined interval 72 are extracted. At the same time, it is possible to determine whether or not the directions of the edges of the plurality of extracted pixels are oriented so as to converge with each other. Further, by setting the number of processes to a predetermined number, the number of pixels to be extracted in the candidate area 1 can be set.

If the end point determination process for the end point with respect to the start point is NG (step S405, No), it is determined that the candidate region 1 is not a water droplet region (step S409). Then, the determination unit 24 outputs the determination result to the water droplet removing device 30 (step S410), and ends the process.

In the modification 6, the number of pixels selected in the candidate region 1 can be appropriately set based on the desired detection accuracy and detection speed.

Further, the interval 72 between the pixels selected in the candidate area 1 can be appropriately set based on the number of selected pixels. For example, when the number of pixels to be selected is small, the interval is widened, and when the number of pixels to be selected is large, the interval is narrowed, so that the pixels can be selected from the entire candidate region 1.

Further, the angle difference between the direction of the edge of the start point and the direction of the edge of the end point can be appropriately set based on the distance between the pixels 72 and the like.

In the modification 6, the pixels distributed in a wide range in the candidate region 1 are extracted, and the adhesion of water droplets is determined based on the orientation of their edges. Therefore, it can be accurately determined that the candidate region 1 has a predetermined shape (arc shape).

Further, in the modification 6, the above-mentioned determination can be efficiently performed by repeatedly performing the end point detection process and the end point determination process. Therefore, it is possible to make a determination with excellent processing efficiency.

Therefore, according to the image processing device 20a according to the modification 6, the water droplets adhering to the camera 10 can be determined efficiently and with high accuracy.

(Modification 7)
In the image processing apparatus 20a according to the modification 6, different end point detection processes and end point determination processes can be performed to determine the adhesion of water droplets. The modified example 7 will be described with reference to FIG.

In the modified example 7, the calculation unit 26 performs an end point detection process for detecting two end points located at predetermined positions with respect to the start point. For example, as shown in FIG. 12, the pixel 70b, which is located at a predetermined interval 72 from the pixel 70a which is the start point and is perpendicular to the edge direction 71a of the pixel 70a (start point), is detected as the first end point. .. Then, the direction 71b of the edge of the pixel 70b, which is the first end point, is calculated.

Further, as an additional end point detection process, the calculation unit 26 is located at a predetermined interval 73 from the pixel 70b, which is the first end point, and is in the direction of the edge direction 71b of the pixel 70b, which is the first end point. 74b is detected as the second end point.

Next, the determination unit 24 performs end point determination processing for determining whether or not each of the first end point and the second end point is included in the candidate area 1. Specifically, as shown in FIG. 12, when the pixel 70b which is the first end point is included in the candidate area 1 and the pixel 74b which is the second end point is not included in the candidate area 1, the end point determination process is OK. Can be determined to be.

That is, the above-mentioned determination evaluates whether or not the first end point is located in the candidate area 1 and the second end point is located outside the candidate area 1.

Then, as in the modified example 6, when the end point determination process is OK, the determination unit 24 refers to the number of processes. Then, when the number of processes is less than or equal to a predetermined number of times, the determination unit 24 detects the end point again with the pixel 70b, which is the point processed as the first end point in the previous processing, as the start point for the calculation unit 26. Instruct to perform the process.

In the modified example 7, as shown in FIG. 12, for example, when the pixel 70a is used as the start point, the pixel 70b and the pixel 74b are detected by the end point detection process. Then, when the end point determination process of the pixel 70b and the pixel 74b is OK and the number of processes has not been reached, the end point detection process is newly performed with the pixel 70b as the start point. As a result, the pixels 70c and the pixels 74c are detected as new end points. Then, the end point determination processing of the pixels 70c and the pixels 74c is performed again.

When compared with the modified example 6, in the embodiment of the modified example 7, two end points are inspected in one end point detection process, and the end point determination process is performed based on them. Therefore, it can be accurately determined that the candidate region 1 has a predetermined shape (arc shape) even when the number of processing of the end point detection process and the end point determination process is smaller than that of the modification 6.

Therefore, according to the image processing device 20a according to the modification 7, the water droplets adhering to the camera 10 can be determined at high speed.

In the above-described embodiment and modification, when the captured image is dark or bright as a whole, the difference between the minimum brightness and the maximum brightness in the captured image may not be large. When the difference in brightness does not increase in this way, the intensity of the edges of the pixels in the captured image 100 is generally low. As a result, it is difficult to correctly detect the candidate region 1 based on the intensity of the edge of each pixel of the captured image 100 as it is.

In the above case, it is effective to perform an image process called a dynamic range enlargement process before the edge detection process of the captured image 100 by the detection unit 22. By the dynamic range enlargement processing, the difference between the minimum brightness and the maximum brightness in the captured image 100 can be enlarged. As a result, the detection unit 22 can correctly detect the candidate region 1.

Further, in the above-described embodiment and modification, after the edge detection process is performed by the detection unit 22, pixels (hereinafter, referred to as “edge points”) that should be included in the candidate region 1 due to various noises or the like are not detected. False positives may occur. As a result, the shape of the candidate region 1 which is an aggregate of edge points may be disturbed, and the candidate region 1 may not be detected accurately.

In the above case, it is effective to perform image processing called morphology processing after edge detection processing of the captured image 100 by the detection unit 22. Among the morphology processes, an image process called an expansion process can complement undetected edge points. Further, among the morphology processes, an image process called a shrinkage process can remove erroneously detected edge points. Then, by performing the expansion treatment and the contraction treatment in combination, the shape of the candidate region 1 is adjusted, and the candidate region 1 can be accurately detected.

In the above-described embodiment, the values of the detection condition and the determination condition are specifically shown, but these are merely examples and are not limited.

In addition, the image processing shown in the above-described embodiments and modifications may be combined as appropriate. For example, the determination method of the first embodiment and the determination method of the second embodiment may be combined. That is, only when the candidate region 1 can be determined to be a water droplet region by the determination method of the first embodiment and the candidate region 1 can be determined to be a water droplet region by the determination method of the second embodiment, the candidate region 1 May be finally determined to be the water droplet region. As a result, it is possible to further improve the accuracy of determining the adhesion of water droplets to the camera 10, and it is also possible to determine the adhesion of water droplets at a higher speed.

Further, in the above, an example in which the water droplet removal system 11 is applied to the camera 10 for a vehicle is shown, but the present invention is not limited to this, and other types such as a surveillance / security camera installed inside or outside a building or in an alley or the like are shown. It may be applied to the camera (optical sensor) of.

Further, in the above-described embodiment, the image processing devices 20 and 20a may be mounted inside the vehicle, or may be arranged outside the vehicle, and various information may be obtained by wirelessly using a camera (imaging unit), a water droplet removing device, or the like. You may send and receive.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments expressed and described as described above. Therefore, various modifications can be made without departing from the spirit or scope of the general concept of the invention as defined by the appended claims and their equivalents.

1 Candidate area 2 Center 2a Center area 3 Points of interest 10 Camera (imaging unit)
11 Water droplet removal system 20 Image processing device 21 Control unit 22 Detection unit 23 Estimating unit 24 Judgment unit 25 Storage unit 26 Calculation unit 30 Water droplet removal device 100 Captured image

Claims (4)

A detection unit that detects a candidate region that is a candidate for a water droplet region affected by water droplets adhering to the imaging unit based on the intensity of the edge of each pixel of the captured image acquired by the imaging unit.
A calculation unit that calculates the direction of the edge of each pixel in the attention point, which is the intersection of the rectangle inscribed in the candidate region and the candidate region, and the peripheral region of the attention point.
An image processing characterized by comprising a determination unit for determining whether or not the candidate region is the water droplet region based on the variation in the orientation of the edge of each pixel of the attention point and the region around the attention point. Device. The determination unit
When the variation in the orientation of the edge of each pixel in the candidate region that is before Kichu th point within a predetermined range is within a predetermined range, determining that the candidate region is the water droplet region The image processing apparatus according to claim 1. The determination unit
When the edges of a predetermined number of pixels arranged at a predetermined interval along the candidate region are oriented so as to converge with each other, the candidate region is defined as the water droplet region. The image processing apparatus according to claim 1 or 2, wherein the determination is made. The image processing apparatus according to any one of claims 1 to 3.
A water droplet removing system including a water droplet removing device that removes water droplets adhering to the imaging unit based on a determination result in the determination unit of the image processing device.

Images