JP7724300B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing device.

A technology has been known for some time that analyzes images captured by a stereo camera mounted on a vehicle (own vehicle) to detect the relative vertical misalignment between the left and right cameras and correct the misalignment accordingly.

Japanese Patent Application Publication No. 2018-105682

When an in-vehicle stereo camera captures an image of a road surface painted with repeated diagonal lines in the depth direction, such as a zebra strip, while there is a relative vertical misalignment between the optical axes of the left and right cameras, the camera may mistakenly detect the area with the road paint as a three-dimensional object. If the area with the road paint is mistakenly detected as a three-dimensional object, safety systems provided by the stereo camera, such as AEB and ACC, may be activated, causing unnecessary alarms or braking, potentially causing discomfort to occupants.

The conventional method described in Patent Document 1, mentioned above, is intended to correct the relative vertical shift between the left and right cameras, and corrects the relative vertical shift that occurs between the left and right cameras based on the parallax value calculated from the horizontal imaging position shift between the left and right cameras.

However, the method in Patent Document 1 uses the calculated disparity value as is to correct for vertical misalignment between the left and right cameras, which raises the risk of erroneous correction if an incorrect disparity value is calculated. For example, if an image of a road surface painted with repeated diagonal lines in the depth direction, such as a flow guidance strip, is captured while the vertical misalignment has been erroneously corrected, there is a risk that the area with the road paint will be erroneously detected as a three-dimensional object.

If the area containing road paint is mistakenly detected as a three-dimensional object, safety systems provided by the stereo camera, such as AEB and ACC, may be activated, causing unnecessary alarms or braking, which may cause discomfort to the occupants. Therefore, a mechanism is needed to determine whether the calculated disparity value is correct before using it.
An object of the present invention is to provide an image processing device that determines whether a disparity calculation result is correct or not based on an image captured by an on-board camera.

The image processing device of the present invention that solves the above problems comprises:
An image processing device that processes images captured by a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and that capture overlapping areas,
an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image;
a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount;
a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images;
a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of a vehicle in a predetermined attitude state and the flat road surface ahead is imaged;
a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid;
an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes;
The present invention is characterized by comprising:

According to the present invention, by determining whether the parallax calculated from images captured by a pair of imaging units is correct, it is possible to prevent erroneous correction of the optical axis of the imaging device.

Further features related to the present invention will become apparent from the description of this specification and the accompanying drawings. Furthermore, issues, configurations, and advantages other than those described above will become apparent from the description of the following embodiments.

1 is a functional block diagram illustrating an overall configuration of an image capturing apparatus according to an embodiment of the present invention to which an image processing apparatus of the present invention is applied; 5 is a flowchart for explaining a parallax accuracy determination of the imaging device of the present embodiment. 10A and 10B are diagrams illustrating simulated left and right captured images and a parallax image. 10A and 10B are diagrams showing the difference between distant parallax and near parallax in left and right captured images; FIG. 10 is a diagram illustrating a method for calculating parallax when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are relatively offset in the vertical direction. 10 is a graph showing the relationship between the vertical shift amount of the third image and the effective parallax number. 10A and 10B are diagrams illustrating the relationship between a parallax image of a road surface and an expected parallax value.

Next, one embodiment of the present invention will be described.
The imaging device of this embodiment to which the image processing device of the present invention is applied is an imaging device that is mounted on a vehicle and captures images of the outside world, and is configured by a stereo camera.

A stereo camera comprises a pair of cameras positioned to capture images of the road ahead, for example through the windshield, and an image processing device that processes the images captured by the pair of cameras. The pair of cameras are installed in a predetermined position on the vehicle in a predetermined orientation, positioned horizontally apart from each other on the left and right, and capture overlapping areas ahead.

The image processing device is composed of an ECU (Electronic Control Unit) that has a CPU (Central Processing Unit), memory, and software programs. The image processing device, for example, calculates parallax based on the left and right captured images, determines whether the parallax is correct, and performs an optical axis misalignment detection process using only the correct parallax. It then uses the amount of optical axis misalignment to perform a correction process that corrects the relative vertical misalignment between the optical axes of the pair of cameras.

First, we will explain how to calculate disparity in a stereo camera and the reasons why a stereo camera calculates incorrect disparity using Figures 3 to 5.

Figure 3 is a diagram showing simulated left and right captured images and parallax images. Figure 3(a) is a schematic diagram of a captured image 301 captured by the left camera looking ahead from the vehicle, and Figure 3(b) is a schematic diagram of a captured image 302 captured by the right camera looking ahead from the vehicle at the same time as the left camera. The left and right captured images 301 and 302 show a road surface 312 and a preceding vehicle 311 traveling on the road surface 312.

Figure 3(c) is a disparity image 303 created using the captured image 301 in Figure 3(a) and the captured image 302 in Figure 3(b), and shows an area 321 close to the vehicle, an area 323 far from the vehicle, and an intermediate area 322. From the relationship between distance and disparity shown in Figure 3(c), a stereo camera can calculate the distance to an object based on the disparity value.

A stereo camera detects a three-dimensional object, such as a preceding vehicle 311, from a disparity image 303 as shown in Figure 3(c). When there is no three-dimensional object, the disparity indicates the distance on the road surface, gradually decreasing as you move from near to far, and is also shown to gradually decrease vertically from bottom to top in the disparity image. On the other hand, when there is a three-dimensional object, the disparity is the same for near and far objects, and is shown to be the same value along the vertical direction in the disparity image. Therefore, it is determined that a three-dimensional object is present in an area in the disparity image where the disparity is the same along the vertical direction.

FIG. 4 is a diagram showing the difference between the distant parallax and the near parallax in the left and right captured images.
Two preceding vehicles 411 and 412 are captured in both an image 401 captured by the right camera and an image 402 captured by the left camera. Due to the characteristics of the stereo camera, the parallax for the preceding vehicle 411, which is close to the host vehicle, is large, and the parallax for the preceding vehicle 412, which is farther away from the host vehicle, is small, as shown in FIG.

Figure 5 shows the disparity when a three-dimensional object with a repeating pattern, such as a fence, is present in front of the vehicle. Figure 5(a) is a diagram showing a simulation of the method of calculating disparity when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are not shifted relative to each other in the vertical direction, and in which no vertical optical axis shift has occurred. Figure 5(b) is a diagram showing a simulation of the method of calculating disparity when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are shifted relative to each other in the vertical direction, and in which vertical optical axis shift has occurred.

For example, when calculating the parallax for a repeating pattern such as a fence with diagonally intersecting wire mesh as shown in Figure 5, the parallax of point 511 in left image 501 shown in Figure 5(a) will be matched with point 512 in right image 502. In other words, point 512 in right image 502 that corresponds to point 511 in left image 501 is searched for. If there is a relative vertical positional shift δ between the left and right cameras of the stereo camera, the result will be as shown in Figure 5(b). Figure 5(b) shows a state in which left image 503 is shifted upward by δ relative to right image 504.

In this case, point 513 on the left image 503 should match point 514 on the right image, but because the left and right images are misaligned in the vertical direction, matching is not possible and it is impossible to calculate the parallax. However, in the case of a repeating pattern such as a fence, a match can occur even when the pattern is slightly misaligned from its original position, as in the case of point 515 on the right image. In this case, the calculated parallax value includes an error equal to the difference in the horizontal positions of points 514 and 515.

In the case of the method shown in Patent Document 1, if optical axis correction is performed when there is a repeating pattern such as a fence, barrier, or zebra stripe marked on the road ahead of the vehicle, there is a risk that a parallax value containing the above-mentioned error will be calculated, and optical axis correction will be performed based on that parallax value containing the error.

To address this issue, the imaging device of this embodiment is characterized by the fact that, when correcting vertical misalignment that occurs between the left and right cameras, it performs a process to determine whether the parallax information used in the correction process is correct.With the imaging device of this embodiment, vertical misalignment can be corrected using only the correct parallax value from among multiple parallax values.

FIG. 1 is a functional block diagram showing the overall configuration of an image capturing apparatus according to an embodiment of the present invention to which an image processing apparatus of the present invention is applied.
1, the imaging device 100, which is a stereo camera, is generally composed of a pair of imaging units 11 and 12, an image storage unit 13, a movement image generation unit 14, a parallax calculation unit 15, a memory unit 16, a validity determination unit 17, and an optical axis deviation detection unit 18. The pair of imaging units 11 and 12 are composed of left and right cameras, and the image storage unit 13, the movement image generation unit 14, the parallax calculation unit 15, the memory unit 16, the validity determination unit 17, and the optical axis deviation detection unit 18 are configured as internal functions of the image processing device 101.

The pair of imaging units 11, 12 (first imaging unit 11, second imaging unit 12) are used to capture an image by imaging the direction of the imaging subject in the imaging device 100, and are arranged so that their optical axes and imaging directions face the same direction and are at the same height on the left and right sides of the imaging device 100. The first imaging unit 11 and second imaging unit 12 are, for example, cameras equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and a lens.

The first imaging unit 11 and the second imaging unit 12 capture images simultaneously and periodically to obtain first and second images, respectively. The image storage unit 13 cuts out and stores images of specific overlapping areas from the first and second images captured simultaneously by the first imaging unit 11 and the second imaging unit 12. The shifted image generation unit 14 generates multiple third images by shifting the first images stored in the image storage unit 13 by a fixed amount in the up-down direction (vertical direction).

The parallax calculation unit 15 calculates the parallax for each third image from the second image and the multiple third images generated by the movement image generation unit. A commonly known method can be used to calculate the parallax. For example, using the second image as a reference, a pixel row in one third image at the same vertical height as the second image is scanned horizontally (horizontally) to find a matching point with the second image, and the horizontal deviation between the second and third images is calculated as the parallax, a process known as stereo matching. This is performed for all vertical height positions in a specified area. The same process is then performed for all remaining multiple third images.

The memory unit 16 stores the expected parallax value of the road surface. The expected parallax value of the road surface is the parallax that should be obtained from the captured image of the flat road surface when the first imaging unit 11 and the second imaging unit 12 are installed at a predetermined position on the vehicle in a predetermined attitude and the flat road surface ahead is imaged, and can be calculated geometrically. The memory unit 16 stores the expected parallax value calculated for each horizontal line from the first image or the second image captured by the first imaging unit 11 or the second imaging unit 12. The expected parallax value of the road surface is acquired each time the parallax between the second image and the third image is calculated. The expected parallax value for each horizontal line can be calculated from the height at which the imaging device 100 is installed and the width (baseline length) between the first imaging unit and the second imaging unit.

FIG. 7 is a diagram illustrating the relationship between the parallax image of the road surface and the parallax expectation value.
7 shows a parallax image 701 of a flat road surface 702 extending forward from the own vehicle, and parallax expectation values d1-d3 corresponding to the parallax image 701. As shown in FIG. 7, the parallax expectation values d1-d3 of the road surface 702 exhibit values that gradually decrease as the road surface moves further forward from the own vehicle, that is, as the road surface moves upward along the Y-axis direction in the parallax image 701 (parallax expectation value d1 of a region 711 near the own vehicle > parallax expectation value d2 of a region 712 further away than the region 711 > parallax expectation value d3 of a region 713 further away than the region 712).

The validity determination unit 17 compares each parallax calculated by the parallax calculation unit 15 with the expected parallax value calculated by the memory unit 16 to determine whether each parallax calculated by the parallax calculation unit 15 is valid. Of the parallaxes calculated by the parallax calculation unit 15, parallaxes indicating distances farther than the expected parallax value calculated by the memory unit 16 (small parallax) are parallaxes that would not appear when capturing an image of the road surface, and are therefore determined to be incorrect parallax and invalid parallax. Furthermore, of the parallaxes calculated by the parallax calculation unit 15, parallaxes indicating nearby distances less than the expected parallax value calculated by the memory unit 16 (large parallax) are determined to be correct parallax and valid parallax.

The comparison between the parallax calculated by the parallax calculation unit 15 and the expected parallax value calculated by the memory unit 16 is performed for each horizontal line. The parallax value for each horizontal line calculated by the parallax calculation unit 15 may be the average value or the median value for each horizontal line.

The optical axis shift detection unit 18 calculates the number of parallaxes (effective number of parallaxes) for the parallax values (effective parallaxes) determined to be valid by the validity determination unit 17. Here, the number of parallaxes for the effective parallaxes calculated from the second image calculated by the parallax calculation unit 15 and each of the multiple third images is calculated. The optical axis shift amount δ is then detected based on the relationship between the vertical shift amount of each third image and the number of parallaxes for the effective parallax.

FIG. 6 is a graph showing the relationship between the vertical shift amount of the third image and the number of effective parallaxes.
In the graph of Figure 6, point 602 indicates the number of effective parallaxes between the first image and the second image, points 601a to 601d indicate the number of effective parallaxes between the second image and a third image obtained by shifting the first image upward by a predetermined number of pixels, and points 601e to 601i indicate the number of effective parallaxes between the second image and a third image obtained by shifting the first image downward by a predetermined number of pixels.

The technology of Patent Document 1 can be applied to a method for detecting the amount of optical axis shift from the shift amount and the number of parallaxes. Specifically, as shown in Figure 6, a distribution of the vertical shift amount (vertical shift) from the first image in each third image and the number of effective parallaxes is created, and an approximation curve Ca is calculated from this distribution. The approximation curve Ca for the number of effective parallaxes is calculated using a method such as the least squares method. The relative amount of optical axis shift δ in the vertical direction of the second imaging unit 12 with respect to the first imaging unit 11 is then calculated from this approximation curve Ca.

Here, the optical axis shift amount δ is calculated by taking advantage of the fact that the number of parallaxes in a parallax image obtained from two images captured by a pair of image capture units increases or decreases depending on the vertical shift of the optical axes of the pair of image capture units. In other words, the number of parallaxes in a parallax image is maximized when there is no vertical shift of the optical axes of the pair of image capture units (minimum case), and tends to decrease as the vertical shift of the optical axes increases. Since the number of parallaxes in a parallax image between the third image and the first image changes depending on the vertical (vertical) shift from the first image, and the number of parallaxes in a parallax image is maximized when there is no vertical shift of the optical axes (minimum case), the vertical shift amount δ of the optical axes of the pair of image capture units is calculated from the difference between the origin position 602 (i.e., when the third image is the same as the first image) in the horizontal direction (i.e., the vertical shift amount of the third image) and the shift amount at which the number of parallaxes is maximized (the peak value of the approximation curve Ca).

The image processing device 101 of the imaging device 100 uses the optical axis shift amount δ detected by the optical axis shift detection unit 18 to perform vertical shift correction to align the vertical positions of the optical axis of the first imaging unit 11 and the optical axis of the second imaging unit 12. Because the optical axis shift amount δ detected by the optical axis shift detection unit 18 is detected using only effective parallax, the image processing device 101 of the imaging device 100 can suppress erroneous correction when performing vertical shift correction.

FIG. 2 is a flowchart illustrating the contents of the determination of the parallax validity of the imaging device in this embodiment.
In step S201, image acquisition is performed. Here, a pair of captured images (first image and second image) captured at the same time by the first imaging unit 11 and the second imaging unit 12 is acquired. In step S202, an image obtained by vertically shifting the first image is acquired as a third image. For example, an image obtained by shifting the first image relative to the second image by each vertical coordinate or by each divided section is acquired as the third image. Then, in step S203, parallax is calculated from the second image and the third image by stereo matching. The area for which parallax is calculated may be the entire image, or only an arbitrary area.

In step S204, the expected parallax value of the road surface is calculated from one of the first, second, or third images. The expected parallax value of the road surface is calculated from an area similar to the area in which the parallax was calculated in step S203, and can be obtained for each horizontal line of the image. In this embodiment, the expected parallax value of the road surface is obtained each time the parallax between the second and third images is calculated, but this can also be achieved by having a parallax expected value table for each horizontal line in advance.

In step S205, the parallax calculated in step S203 is compared with the expected parallax calculated in step S204. The parallax calculated in step S203 can be compared with the expected parallax in units of one pixel in the horizontal direction for each horizontal line. Taking processing time into consideration, comparison may also be made in units of multiple pixels. In step S205, it is determined whether the parallax calculated in step S203 is greater than or equal to the expected parallax calculated in step S204.

Here, if the parallax calculated in step S203 is equal to or greater than the expected parallax calculated in step S204, more specifically, if the parallax calculated in step S203 is greater than the expected parallax calculated in step S204 by one or more predetermined number (parallax indicating the distance of the vicinity) (YES in step S205), it is determined to be a valid parallax in step S206.

On the other hand, if the parallax calculated in step S203 is smaller than the expected parallax calculated in step S204, more specifically, if the parallax calculated in step S203 is smaller than the expected parallax calculated in step S204 by one or more predetermined numbers (parallax indicating a distant distance) (NO in step S205), it is determined to be an invalid parallax in step S207, and this invalid parallax is not used in subsequent processing.

In step S208, the number of parallaxes determined to be valid in step S205 is counted.

In step S209, it is determined whether multiple third images have been generated. If the number of third images is less than a predetermined number of two or more, it is determined that multiple third images have not been generated (NO in step S209), and the process returns to step S202. Then, in steps S202 and S203, the vertical shift amount of the first image is changed to generate another third image with a different shift amount, and the parallax between the generated third image and the second image is calculated.

Then, in step S209, a process of calculating the parallax between the second image and the third image with the shift amount changed is performed until it is determined that there are a specified number of third images or more (YES in step S209). The shift amount can be, for example, a shift of one pixel or multiple pixels. Furthermore, the direction in which the first image is shifted can be at least one of upward and downward. In this embodiment, the first image is shifted both upward and downward relative to the second image to generate multiple third images, and the parallax for each is calculated.

In step S210, a distribution of vertical shift amounts and effective parallax numbers is created, as shown in Figure 6.

In step S211, an approximation curve Ca is calculated from the distribution calculated in step S210, and the relative vertical optical axis shift amount δ is calculated. The position on the approximation curve Ca where the number of effective parallaxes is the highest is the position where there is no vertical shift between the first image and the third image. The optical axis shift amount δ calculated in step S211 is calculated from only the effective parallaxes remaining after removing invalid parallaxes from the multiple parallaxes. Therefore, by using this optical axis shift amount δ to correct the optical axis shift between the first image capture unit 11 and the second image capture unit 12, the relative vertical shift that has occurred between the first image capture unit 11 and the second image capture unit 12 can be accurately corrected.

If the number of effective parallaxes required to calculate the amount of optical axis deviation cannot be obtained from the distribution in step S210, the optical axis deviation detection process in step S211 is not performed until an image that satisfies the specified conditions can be acquired. After the specified conditions are met, the optical axis deviation detection in step S211 is performed and the process ends.

The image processing device 101 included in the imaging device 100 of this embodiment stores a first image captured by the first imaging unit 11 and a second image captured by the second imaging unit 12, generates multiple third images by shifting the pixels of the first image vertically by a predetermined amount, and calculates each parallax from the second image and each of the multiple third images. It then compares each parallax with the expected parallax value of the road surface, which is the parallax that should be obtained from an image of a flat road surface ahead of the vehicle, to determine whether each parallax is valid, and uses only the parallaxes determined to be valid to detect the amount of misalignment of the optical axes in the vertical direction (up and down direction) between the first imaging unit 11 and the second imaging unit 12.

The image processing device 101 of this embodiment, when correcting the vertical misalignment of the optical axes that occurs between the left and right cameras, performs a correct/incorrect determination of the parallax information used in the correction process, thereby making it possible to prevent erroneous correction of the optical axis misalignment that uses an incorrect parallax. Therefore, for example, when capturing an image of a road surface painted with repeated diagonal lines in the depth direction, such as a zebra strip, it is possible to prevent the area with the road paint from being mistakenly detected as a three-dimensional object, thereby preventing safety systems provided by stereo cameras, such as AEB and ACC, from malfunctioning and causing unnecessary alarms or braking, causing discomfort to the occupants.

In the above embodiment, a stereo camera in which the image processing device 101 is integrated with the first imaging unit 11 and the second imaging unit 12 has been described as an example, but the configuration of the image processing device of the present invention is not limited to this embodiment. For example, the image processing device 101 may be provided separately from the pair of cameras.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various design modifications can be made without departing from the spirit of the present invention as set forth in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

100: Imaging device, 101: Image processing device, 11: First imaging unit, 12: Second imaging unit, 13: Image storage unit, 14: Movement image generation unit, 15: Parallax calculation unit, 16: Memory unit, 17: Validity determination unit, 18: Optical axis deviation detection unit

Claims (2)

An image processing device that processes images captured by a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and that capture overlapping areas,
an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image;
a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount;
a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images;
a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of a vehicle in a predetermined attitude state and the flat road surface ahead is imaged;
a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid;
an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes;
Equipped with
the moving image generating unit generates the third image by shifting a relative vertical positional relationship between the first image and the second image for each vertical coordinate or for each divided section;
the storage unit calculates and stores the expected parallax value for each vertical coordinate of the first image, the second image, or the third image;
the storage unit stores the expected parallax value of the road surface as a value for each horizontal line of the captured image of the flat road surface;
the validity determination unit compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface for each horizontal line;
the validity determination unit determines, among the parallaxes calculated by the parallax calculation unit, a parallax that indicates a distance farther than an expected parallax value of the road surface as an invalid parallax, and determines a parallax that indicates a nearby distance equal to or less than the expected parallax value of the road surface as a valid parallax;
the optical axis deviation detection unit calculates the number of parallaxes of the effective parallax determined to be valid by the validity determination unit, and detects the amount of optical axis deviation based on the relationship between the amount of vertical shift of each third image and the number of parallaxes of the effective parallax;
When the number of parallaxes determined to be valid or the distribution of the number of parallaxes determined to be valid does not satisfy a predetermined condition, the optical axis deviation detection unit does not detect the amount of optical axis deviation until the predetermined condition is satisfied . An imaging device mounted on a vehicle to capture an image of the outside world,
a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and capture images of overlapping areas;
an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image;
a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount;
a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images;
a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of the vehicle in a predetermined attitude state and the flat road surface ahead is imaged;
a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid;
an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes;
Equipped with
the moving image generating unit generates the third image by shifting a relative vertical positional relationship between the first image and the second image for each vertical coordinate or for each divided section;
the storage unit calculates and stores the expected parallax value for each vertical coordinate of the first image, the second image, or the third image;
the storage unit stores the expected parallax value of the road surface as a value for each horizontal line of the captured image of the flat road surface;
the validity determination unit compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface for each horizontal line;
the validity determination unit determines, among the parallaxes calculated by the parallax calculation unit, a parallax that indicates a distance farther than an expected parallax value of the road surface as an invalid parallax, and determines a parallax that indicates a nearby distance equal to or less than the expected parallax value of the road surface as a valid parallax;
the optical axis deviation detection unit calculates the number of parallaxes of the effective parallax determined to be valid by the validity determination unit, and detects the amount of optical axis deviation based on the relationship between the amount of vertical shift of each third image and the number of parallaxes of the effective parallax;
When the number of parallaxes determined to be valid or the distribution of the number of parallaxes determined to be valid does not satisfy a predetermined condition, the optical axis deviation detection unit does not detect the amount of optical axis deviation until the predetermined condition is satisfied.
An imaging device characterized by: