JP6846640B2 - On-board camera calibration device - Google Patents

Description

The present invention relates to an in-vehicle camera calibration device that calibrates a camera using time-series continuous images taken by the camera.

Conventionally, by displaying an image of the rear of the vehicle taken by the in-vehicle camera on the in-vehicle monitor, the situation near the rear of the vehicle, which is a blind spot from the driver, is visually recognized as an image displayed on the in-vehicle monitor, and is visually recognized when the vehicle is reversing. Improving sex is being done.

When displaying such an image of the in-vehicle camera on the in-vehicle monitor, a calibration target is installed at the rear of the vehicle in order to correct the mounting state of the in-vehicle camera on the vehicle, and the calibration target displayed on the in-vehicle monitor. While looking at the image of, the mounting state of the in-vehicle camera is adjusted so that the image of the calibration target is properly reflected.

Further, the image obtained by the in-vehicle camera is appropriately calibrated by performing a predetermined arithmetic process based on the image of the calibration target to properly calibrate the image displayed on the in-vehicle monitor.

In addition, the entire circumference of the vehicle is photographed by a plurality of in-vehicle cameras, and the plurality of images obtained by each in-vehicle camera are converted into an image (overhead image) as if looking down from directly above the vehicle, and between each image. By performing mapping that adjusts the position of, a single viewpoint conversion composite image is also obtained. In such a case, it is necessary to perform accurate alignment between two adjacent images, so that high-precision calibration is required.

Therefore, some techniques have been proposed conventionally. For example, in Patent Document 1, a plurality of calibration targets are imaged while the vehicle is moving straight ahead, and the amount of vehicle movement is estimated from the optical flow. A method of calibrating the camera from the amount of movement of the vehicle and the amount of movement of the feature points of the optical flow has been proposed. Further, Non-Patent Document 1 proposes a method of tracking feature points of continuous images and calculating an optical flow.

JP-A-2017-139612

Carlo Tomasi, Takeo Kanade, Detection and Tracking of Point Features, Technical Report CMU-CS-91-132, April 1991

However, in the calibration method described in Patent Document 1, the final camera calibration accuracy is low due to insufficient calculation accuracy of the F matrix (basic matrix), and there is room for further improvement. Further, in the calibration method described in Non-Patent Document 1, the camera can be calibrated only when the vehicle goes straight, and it does not correspond to the vehicle curve.

The present invention has been made in view of the above problems, and provides an in-vehicle camera calibration device capable of calculating an F matrix with high accuracy and capable of handling not only when the vehicle is traveling straight but also when the vehicle is curved. The purpose.

The in-vehicle camera calibration device of the present invention has an optical flow calculation unit that calculates an optical flow by extracting road surface feature points before and after movement from a road surface image before and after movement taken by a camera mounted on a vehicle, and from the optical flow. , The straight-ahead F matrix calculation unit that calculates the straight-ahead F matrix, which is the basic matrix when the vehicle is assumed to be straight, and before and after the movement when the vehicle is assumed to be straight-ahead from the straight-ahead F matrix. A straight-ahead RT matrix calculation unit that calculates a straight-ahead R matrix that is a rotation matrix between camera coordinates, a straight-ahead T matrix that is a parallel-traveling matrix between camera coordinates before and after movement when the vehicle is assumed to be straight, and the above. By optimizing the straight R matrix and the straight T matrix, the curve R matrix, which is a rotation matrix when the vehicle is assumed to be curved, and the parallel traveling matrix when the vehicle is assumed to be curved. The curve RT matrix calculation unit that calculates the curve T matrix, the triangular survey unit that calculates the camera coordinates of the road surface feature points from the curve R matrix and the curve T matrix, and the curve R matrix and the curve T matrix. A camera angle calculation unit that calculates the installation angle of the camera from the camera coordinates of the road surface feature point, and an angle recording unit that records the installation angle of the camera calculated while the vehicle is running in a database as angle information. It is provided with an optimum angle calculation unit for obtaining an optimum value of the installation angle of the camera from a plurality of the angle information recorded in the database.

According to this configuration, since the F matrix has three variables, the calculation accuracy of the F matrix is improved as compared with the conventional case (when the F matrix is eight). In addition, the camera can be calibrated not only when the vehicle goes straight, but also when the vehicle is curved.

Further, in the in-vehicle camera calibration device of the present invention, the angle recording unit records pitch, roll, and pan as the angle information in the database together with the recording time, and the optimum angle calculation unit records the latest pitch at the recording time. The roll having the latest recording time as the optimum value of the pitch may be the optimum value of the roll, and the bread having the latest recording time among the pans when the vehicle is traveling straight may be the optimum value of the pan.

According to this configuration, the optimum pitch, roll, and pan values can be appropriately calculated from the angle information recorded in the database.

Further, in the in-vehicle camera calibration device of the present invention, the angle recording unit records pitch, roll, and pan as the angle information in the database together with the recording time, and the optimum angle calculation unit records a plurality of pitches having a new recording time. The average value of the pitch is the optimum value of the pitch, the average value of a plurality of rolls having a new recording time is the optimum value of the roll, and the average value of a plurality of breads having a new recording time among the pans when the vehicle is traveling straight. May be the optimum value for the bread.

According to this configuration, the optimum pitch, roll, and pan values can be appropriately calculated from the angle information recorded in the database.

The in-vehicle camera calibration device of the present invention includes an optical flow calculation unit that calculates an optical flow by extracting road surface feature points before and after movement from road surface images before and after movement taken by a camera attached to the vehicle, and an optical flow calculation unit attached to the vehicle. From the steering angle information obtained from the steering angle sensor and the vehicle speed information obtained from the vehicle speed sensor attached to the vehicle, it is a rotation matrix between the camera coordinates before and after the movement when the vehicle is assumed to be traveling straight. A straight-ahead R matrix, a straight-ahead RT matrix calculation unit that calculates a straight-ahead T matrix that is a parallel-traveling sequence between camera coordinates before and after movement when the vehicle is assumed to be straight, a straight-ahead R matrix, and the straight-ahead T. By optimizing the matrix, a curve R matrix, which is a rotation matrix when the vehicle is assumed to be curved, and a curve T matrix, which is a parallel traveling matrix when the vehicle is assumed to be curved, are calculated. A curve RT matrix calculation unit, a triangular survey unit that calculates camera coordinates of the road surface feature point from the curve R matrix and the curve T matrix, and a camera of the curve R matrix, the curve T matrix, and the road surface feature point. A camera angle calculation unit that calculates the installation angle of the camera from the coordinates, an angle recording unit that records the installation angle of the camera calculated while the vehicle is running in a database as angle information, and an angle recording unit recorded in the database. It is provided with an optimum angle calculation unit for obtaining an optimum value of the installation angle of the camera from the plurality of the angle information.

According to this configuration, it is possible to correct the sensor error and the turning characteristic error of the vehicle. This makes it possible to provide a highly accurate camera calibration device.

Further, the in-vehicle camera calibration device of the present invention includes a straight-ahead determination unit that determines whether or not the vehicle is traveling straight based on the steering angle information obtained from the steering angle sensor attached to the vehicle, and the optical The flow calculation unit calculates the optical flow when the straight-ahead determination unit determines that the vehicle is traveling straight, and when the straight-ahead determination unit determines that the vehicle is not traveling straight, the flow calculation unit calculates the optical flow. It is not necessary to calculate the optical flow.

According to this configuration, the F matrix can be calculated with high accuracy by using only the data when the vehicle goes straight. This makes it possible to provide a camera calibration device having high camera calibration accuracy.

According to the present invention, it is possible to calculate the F matrix with high accuracy, and it is possible to handle not only when the vehicle is traveling straight but also when the vehicle is curved.

Explanatory drawing of in-vehicle camera calibration apparatus of 1st Embodiment of this invention A block diagram showing a configuration of an in-vehicle camera calibration device according to a first embodiment of the present invention. Explanatory drawing of optical flow in 1st Embodiment of this invention Explanatory drawing of camera coordinate system and world coordinate system before and after movement in 1st Embodiment of this invention Explanatory drawing of travel vector and world coordinate system in 1st Embodiment of this invention The flow diagram explaining the operation of the vehicle-mounted camera calibration apparatus of the 1st Embodiment of this invention. A block diagram showing a configuration of an in-vehicle camera calibration device according to a second embodiment of the present invention. The flow diagram explaining the operation of the vehicle-mounted camera calibration apparatus of the 2nd Embodiment of this invention. A block diagram showing a configuration of an in-vehicle camera calibration device according to a third embodiment of the present invention. The flow diagram explaining the operation of the vehicle-mounted camera calibration apparatus of the 3rd Embodiment of this invention.

Hereinafter, the in-vehicle camera calibration device according to the embodiment of the present invention will be described with reference to the drawings. In this embodiment, the case of an in-vehicle camera calibration device used for a vehicle navigation system or the like will be illustrated.

(First Embodiment)
The configuration of the vehicle-mounted camera calibration device 10 according to the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an explanatory diagram of the vehicle-mounted camera calibration device 10 of the present embodiment. As shown in FIG. 1, the vehicle 1 travels on the road surface 2 in the traveling direction. The camera 3 is installed behind the vehicle 1, and the optical axis of the camera 3 is set in the road surface direction. When the vehicle 1 travels on the road surface 2, the camera 3 captures two road surface images (pre-movement image and post-movement image, which will be described later) with a time lag. Although not shown in FIG. 1, the in-vehicle camera calibration device 10 is mounted on the vehicle 1.

FIG. 2 is a block diagram showing the configuration of the vehicle-mounted camera calibration device 10. As shown in FIG. 2, the in-vehicle camera calibration device 10 includes an optical flow calculation unit 11, a straight F matrix calculation unit 12, a straight RT matrix calculation unit 13, a curve RT matrix calculation unit 14, a triangulation unit 15, and an angle calculation unit 16. The angle recording unit 17 and the optimum angle calculation unit 18 are provided.

The optical flow calculation unit 11 calculates the optical flow by extracting the road surface feature points before and after the movement from the two road surface images (road surface images before and after the movement) taken by the camera 3. FIG. 3 shows two road surface images taken by the camera 3 with a time lag. The road surface image taken before in chronological order can be called a pre-movement image, and the road surface image taken after in chronological order can be called a post-movement image. Further, the image before movement and the image after movement can be collectively referred to as a time series 2 image.

An image before movement and an image after movement are input to the optical flow calculation unit 11. The optical flow calculation unit 11 extracts the feature points (road surface feature points) on the road surface of the pre-movement image, and calculates the feature points (road surface feature points) on the road surface of the corresponding post-movement image. A known technique can be used to calculate the feature points. The feature points on the road surface of the image before movement and the feature points on the road surface of the image after movement can be expressed by image coordinates. As shown in FIG. 3, the image coordinates have the origin at the upper left of the image, the X direction to the right from the origin, and the Y direction from the origin to the bottom. When there are N feature points on the road surface, the movement source image coordinates and the movement destination image coordinates can be expressed as follows.

The straight-ahead F matrix calculation unit 12 calculates a straight-ahead F matrix, which is a basic matrix when the vehicle 1 is assumed to be straight-ahead, from the optical flow. More specifically, the straight-ahead F matrix calculation unit 12 calculates the straight-ahead F-matrix by the following method on the assumption that the vehicle 1 goes straight and the height from the road surface 2 is constant before and after the movement.

First, the image coordinates of the feature points are converted into the sensor coordinates on the image sensor.

なお、レンズ歪係数は、必要に応じてより高次の係数を使用する。 Next, the sensor coordinates on the image sensor are converted into distortion-free sensor coordinates.

As the lens distortion coefficient, a higher-order coefficient is used as needed.

By performing the above conversion, the movement source image coordinates and the movement destination image coordinates can be expressed in sensor coordinates without distortion.

The relationship between the cameras before and after the movement can be expressed by the following equation A.

The elementary matrix (E matrix) can be expressed by the following equation B using a rotation matrix (R matrix) and a parallel traveling matrix (T matrix).

When the vehicle 1 goes straight, the R matrix becomes an identity matrix, so the basic matrix can be expressed by the following equation C.

The camera coordinate system and the image coordinate system without distortion can be expressed by the following equation D.

Therefore, the basic matrix (F matrix) when the vehicle 1 goes straight is represented by the following equation E.

Here, due to the nature of the basic matrix (F matrix), the following equation F holds for all the feature points.

Comparing the above equations E and F, the following relationship can be obtained.

By substituting the above relationship into equation F and expanding it, the following equation G is derived.

The straight-ahead F matrix calculation unit 12 obtains an eigenvector for the minimum eigenvalue satisfying the above equation G, and obtains a straight-ahead F matrix.

From the straight F matrix, the straight RT matrix calculation unit 13 assumes that the straight R matrix, which is a rotation matrix between the camera coordinates before and after the movement when the vehicle 1 is straight, and the vehicle 1 are straight. A straight-ahead T matrix, which is a parallel traveling matrix between the camera coordinates before and after the movement at the time of movement, is calculated. More specifically, the straight RT matrix calculation unit 13 calculates the straight R matrix and the straight T matrix by the following method.

First, the E matrix is calculated from the F matrix by the following equation H.

Next, the minimum eigenvector for the minimum eigenvalue of W is obtained. That is the translation vector T. Further, the direction of the translation vector is determined in consideration of the moving direction of the vehicle 1. The rotation vector R is a unit vector. Then, the straight T matrix is calculated from the translation vector T, and the straight R matrix is calculated from the rotation vector R.

The curve RT matrix calculation unit 14 optimizes the straight R matrix and the straight T matrix, so that the curve R matrix, which is a rotation matrix when the vehicle 1 is assumed to be curved, and the vehicle 1 are curved. The curve T matrix, which is a parallel traveling matrix when assuming that, is calculated.

The time difference between the images of the pre-movement camera and the post-movement camera is short, and even if the steering wheel of the vehicle 1 is operated, it can be calculated as a deviation between the T matrix and the R matrix when traveling straight. Hereinafter, a method of calculating the curve R matrix and the curve T matrix of the curve RT matrix calculation unit 14 will be described.

(Step 1)
The straight T matrix calculated by the straight F matrix calculation unit 12 is used as the initial value of the curve T matrix. The curve R matrix is represented by tilt (T), roll (R), and pan (P). The initial value of the curve R matrix is an identity matrix (T = 0, R = 0, P = 0).

(Step 2)
The R matrix is calculated from the tilt (T), roll (R), and pan (P).

(Step 3)
Equation B is used to calculate the E matrix from the R and T matrices.

(Step 4)
The formula E is used to calculate the F matrix from the E matrix.

(Step 5)
Cost is calculated using the following formula I.

(Step 6)
If Cost is equal to or less than the threshold value, the process ends, and the R matrix and T matrix at that time become the optimized values.

(Step 7)
For example, using a nonlinear optimization method such as Newton-Raphson's method, the T, R, P and T matrices are slightly deformed to obtain new values (the following equation J).

(Step 8)
If the amount of minute deformation is less than the threshold value, the process ends, and the R matrix and the T matrix at that time become the optimized values. If not, return to step 2.

By optimizing the R matrix and the T matrix as described above, the curve R matrix (optimized value of the R matrix) and the curve T matrix (optimal value of the T matrix) are calculated.

The triangulation unit 15 calculates the camera coordinates of the road surface feature points. More specifically, the camera coordinates of the road surface feature points are calculated from the curve R matrix and the curve T matrix as follows.

First, the relationship between the pre-movement camera coordinates and the pre-movement sensor coordinates can be obtained from the equation D (the following equation K).

Similarly, the relationship between the post-movement camera coordinates and the post-movement sensor coordinates can be obtained (the following equation L).

The relationship between the pre-movement camera coordinates and the post-movement camera coordinates is expressed by Equation A. By transforming the formula A, the following formula M is obtained.

Substituting Equation M into Equation L gives Equation N:

Further, by modifying the formula D, the following formula O is obtained.

Since there are cameras before and after movement, the following formula P is used.

By solving the equation P by the least squares method, the camera coordinates of the feature points of the pre-movement camera can be obtained.

The angle calculation unit 16 calculates the installation angle of the camera 3 from the camera coordinates of the curve R matrix, the curve T matrix, and the road surface feature points. Hereinafter, a method of calculating the installation angle of the camera 3 will be described with reference to FIG.

FIG. 4 is an explanatory diagram of the camera coordinate system and the world coordinate system before and after the movement. In FIG. 4, there is a coordinate system of the pre-movement camera (pre-movement camera coordinate system) on the road surface. When the vehicle 1 goes straight, it becomes the coordinate system of the camera after movement (camera coordinate system after movement). The world coordinate system is directly below the camera coordinate system before movement. The XYZ directions of each coordinate system are as shown in FIG. Further, as shown in FIG. 4, the vector in the normal direction of the road surface 2 is referred to as a “normal vector”. The vector from the origin of the camera coordinate system before movement to the origin of the camera coordinate system after movement is defined as the "advance vector".

The camera coordinates in the pre-movement camera coordinate system of the feature points on the road surface are calculated by the triangulation unit 15. From the camera coordinates of this feature point, the coefficient of the following equation Q is calculated by the least squares method.

Equation Q is normalized by multiplying it by the following ratio.

If the distance H from the road surface 2 to the camera 3 is known, the above ratio is multiplied by the parallel traveling matrix to obtain the actual size.

The upper right frame of FIG. 4 is a diagram in which the origin of the camera coordinate system before movement and the origin of the world coordinate system are matched by moving on the normal line.

Find the rotation matrix that rotates from the camera coordinate system to the world coordinate system.

The coefficients (a, b, c) in equation Q are normal vectors. The normal vector is equal to the Y-axis of world coordinates. Therefore, the following relationship is established.

When the vehicle 1 goes straight, the traveling vector is obtained by making each element of the parallel traveling matrix in the equation A negative. When the vehicle 1 goes straight, it becomes equal to the Z axis of the world coordinates in the opposite direction. Therefore, the following relationship is established.

The X-axis of the world coordinates can be obtained as the outer product of the Y-axis and the Z-axis of the world coordinates. Therefore, the following relationship is established.

Convert from the world coordinate system to the camera coordinate system.

The pitch, roll, and bread can be obtained from the following formula R.

Since two time-series images can be continuously acquired while the vehicle 1 is traveling, the angle calculation unit 16 can calculate a plurality of angle information.

The angle recording unit 17 records the installation angle of the camera 3 calculated while the vehicle 1 is traveling in the database 19 as angle information. More specifically, the angle recording unit 17 records pitch, roll, and pan angle information in the database 19. Hereinafter, a method of recording angle information will be described with reference to FIG.

FIG. 5 is an explanatory diagram of the traveling vector and the world coordinate system. FIG. 5A is a diagram when the vehicle 1 travels straight, and FIG. 5B is a diagram when the vehicle 1 curves. As shown in FIG. 5A, when the vehicle 1 travels straight, the pan can be calculated accurately because the Z axis of the world coordinate system and the traveling vector coincide with each other. On the other hand, as shown in FIG. 5B, when the vehicle 1 curves, the pan cannot be accurately obtained because the world coordinate system and the traveling vector do not match.

The curve R matrix calculated by the curve RT matrix calculation unit 14 is straight when it is an identity matrix, and is a curve when it is not an identity matrix. As shown in Table 1, the angle recording unit 17 records angle information (tilt, roll, pan) in the database 19 together with recording time and straight / curve information.

The optimum angle calculation unit 18 obtains the optimum value of the installation angle of the camera 3 from the plurality of angle information recorded in the database 19.

For example, the optimum angle calculation unit 18 sets the pitch with the latest recording time as the optimum pitch value, sets the roll with the latest recording time as the optimum value for the roll, and sets the recording time in the pan when the vehicle 1 is traveling straight. Set the latest bread as the optimum value for bread.

Alternatively, the optimum angle calculation unit 18 sets the average value of a plurality of pitches having a new recording time as the optimum value of the pitch, sets the average value of the plurality of rolls having a new recording time as the optimum value of the roll, and the vehicle 1 is traveling straight. The optimum value of pans is the average value of a plurality of pans having a new recording time among the pans of the time.

The operation of the in-vehicle camera calibration device 10 configured as described above will be described with reference to the flow chart of FIG.

As shown in FIG. 6, in the in-vehicle camera calibration device 10 of the first embodiment, first, the road surface images (time series 2 images) before and after the movement taken by the camera 3 attached to the vehicle 1 are acquired (2 images in time series). S1), the optical flow is calculated by extracting the road surface feature points before and after the movement from the road surface images before and after the movement (S2).

Next, from the optical flow, the basic matrix (straight F matrix) when the vehicle 1 is assumed to be traveling straight is calculated (S3), and from the calculated straight F matrix, it is assumed that the vehicle 1 is traveling straight. The rotation matrix (straight R matrix) and the parallel traveling matrix (straight T matrix) between the camera coordinates before and after the movement of time are calculated (S4).

Next, by optimizing the straight R matrix and the straight T matrix, the rotation matrix (curve R matrix) and the parallel traveling matrix (curve T matrix) when the vehicle 1 is assumed to be curved are calculated (S5). , The camera coordinates of the road surface feature points are calculated from the calculated curve R matrix and the curve T matrix (S6), and the installation angle of the camera 3 is calculated from the camera coordinates of the curve R matrix, the curve T matrix and the road surface feature points (S6). S7).

Then, the installation angle of the camera 3 calculated while the vehicle 1 is traveling is recorded in the database 19 as angle information (S8), and the optimum value of the installation angle of the camera 3 is obtained from the plurality of angle information recorded in the database 19. Find (S9).

According to the in-vehicle camera calibration device 10 of the first embodiment as described above, as shown in Equation D, since the F matrix has three variables, it is compared with the conventional case (when the F matrix is eight). The calculation accuracy of the F matrix is improved. Further, the camera can be calibrated not only when the vehicle 1 goes straight, but also when the vehicle 1 is curved. Further, according to the present embodiment, the camera can be calibrated without using an external sensor such as a steering angle sensor.

Further, in the present embodiment, the pitch with the latest recording time is set as the optimum pitch value, the roll with the latest recording time is set as the optimum value for the roll, and the recording time is the latest among the pans when the vehicle 1 is traveling straight. Bread is the optimum value for bread. As a result, the optimum pitch, roll, and pan values can be appropriately calculated from the angle information recorded in the database 19.

Alternatively, the average value of a plurality of pitches having a new recording time is set as the optimum value of the pitch, the average value of a plurality of rolls having a new recording time is set as the optimum value of the roll, and the recording time in the pan when the vehicle 1 is traveling straight. Lets the average value of multiple new breads be the optimum value for breads. As a result, the optimum pitch, roll, and pan values can be appropriately calculated from the angle information recorded in the database 19.

(Second Embodiment)
Next, the vehicle-mounted camera calibration device 10 according to the second embodiment of the present invention will be described. Here, the in-vehicle camera calibration device 10 of the second embodiment will be mainly described as being different from the first embodiment. Unless otherwise specified here, the configuration and operation of this embodiment are the same as those of the first embodiment.

FIG. 7 is a block diagram showing the configuration of the vehicle-mounted camera calibration device 10 of the second embodiment. As shown in FIG. 7, the vehicle-mounted camera calibration device 10 is input with steering angle information obtained from the steering angle sensor 4 attached to the vehicle 1 and vehicle speed information obtained from the vehicle speed sensor 5 attached to the vehicle 1. To.

Then, the straight-ahead RT matrix calculation unit 13 of the present embodiment assumes that the vehicle 1 is traveling straight from the steering angle information obtained from the steering angle sensor 4 and the vehicle speed information obtained from the vehicle speed sensor 5. The rotation matrix (straight R matrix) and the parallel traveling matrix (straight T matrix) between the camera coordinates before and after the movement are calculated.

More specifically, the straight-ahead RT matrix calculation unit 13 uses the steering angle information and the vehicle speed information (vehicle speed pulse), the turning characteristics of the vehicle 1 (for example, the Ackerman model), and the vehicle mounting position of the camera 3 to determine the camera before movement and the camera after movement. The R matrix (straight R matrix) and the T matrix (straight T matrix) of the formula A showing the positional relationship of the cameras are calculated.

The operation of the in-vehicle camera calibration device 10 configured as described above will be described with reference to the flow chart of FIG.

As shown in FIG. 8, in the in-vehicle camera calibration device 10 of the second embodiment, first, road surface images (time series 2 images) before and after movement taken by the camera 3 attached to the vehicle 1 are acquired (time series 2 images). S1), the optical flow is calculated by extracting the road surface feature points before and after the movement from the road surface images before and after the movement (S2).

Next, the steering angle information and the vehicle speed information are acquired from the steering angle sensor 4 steering angle information and the vehicle speed sensor 5 attached to the vehicle 1 (S10), and the vehicle 1 goes straight from the acquired steering angle information and the vehicle speed information. The basic matrix (straight F matrix) is calculated (S11), and the rotation matrix between the camera coordinates before and after the movement when the vehicle 1 is assumed to be straight is calculated from the calculated straight F matrix (S11). A straight R matrix) and a parallel traveling matrix (straight T matrix) are calculated (S4).

Next, by optimizing the straight R matrix and the straight T matrix, the rotation matrix (curve R matrix) and the parallel traveling matrix (curve T matrix) when the vehicle 1 is assumed to be curved are calculated (S5). , The camera coordinates of the road surface feature points are calculated from the calculated curve R matrix and the curve T matrix (S6), and the installation angle of the camera 3 is calculated from the camera coordinates of the curve R matrix, the curve T matrix and the road surface feature points (S6). S7).

Then, the installation angle of the camera 3 calculated while the vehicle 1 is traveling is recorded in the database 19 as angle information (S8), and the optimum value of the installation angle of the camera 3 is obtained from the plurality of angle information recorded in the database 19. Find (S9).

Even with the in-vehicle camera calibration device 10 of the second embodiment as described above, as shown in Equation D, since the F matrix has three variables, F is compared with the conventional case (when the F matrix is eight). Matrix calculation accuracy is improved. Further, the camera can be calibrated not only when the vehicle 1 goes straight, but also when the vehicle 1 is curved.

Moreover, according to the in-vehicle camera calibration device 10 of the present embodiment, it is possible to correct the sensor error and the turning characteristic error of the vehicle 1. This makes it possible to provide a highly accurate camera calibration device.

(Third Embodiment)
Next, the vehicle-mounted camera calibration device 10 according to the third embodiment of the present invention will be described. Here, the in-vehicle camera calibration device 10 of the third embodiment will be mainly described as being different from the first embodiment. Unless otherwise specified here, the configuration and operation of this embodiment are the same as those of the first embodiment.

FIG. 9 is a block diagram showing the configuration of the vehicle-mounted camera calibration device 10 according to the third embodiment. As shown in FIG. 9, the steering angle information obtained from the steering angle sensor 4 attached to the vehicle 1 is input to the vehicle-mounted camera calibration device 10. Further, the in-vehicle camera calibration device 10 of the present embodiment includes a straight-ahead determination unit 20 that determines whether or not the vehicle 1 is traveling straight based on the steering angle information obtained from the steering angle sensor 4. Then, the optical flow calculation unit 11 calculates the optical flow when the straight-ahead determination unit 20 determines that the vehicle 1 is traveling straight. On the other hand, the optical flow calculation unit 11 does not calculate the optical flow when the straight-ahead determination unit 20 determines that the vehicle 1 is not traveling straight.

The operation of the in-vehicle camera calibration device 10 configured as described above will be described with reference to the flow chart of FIG.

As shown in FIG. 10, the in-vehicle camera calibration device 10 of the third embodiment first acquires road surface images (time series 2 images) before and after movement taken by the camera 3 attached to the vehicle 1 (time series 2 images). S1). Then, based on the steering angle information obtained from the steering angle sensor 4, it is determined whether or not the vehicle 1 is traveling straight (S12). When it is determined that the vehicle 1 is traveling straight, the optical flow is calculated by extracting the road surface feature points before and after the movement from the road surface images before and after the movement (S2). If it is determined that the vehicle 1 is not traveling straight, the optical flow is not calculated.

Next, from the optical flow, the basic matrix (straight F matrix) when the vehicle 1 is assumed to be traveling straight is calculated (S3), and from the calculated straight F matrix, it is assumed that the vehicle 1 is traveling straight. The rotation matrix (straight R matrix) and the parallel traveling matrix (straight T matrix) between the camera coordinates before and after the movement of time are calculated (S4).

Next, by optimizing the straight R matrix and the straight T matrix, the rotation matrix (curve R matrix) and the parallel traveling matrix (curve T matrix) when the vehicle 1 is assumed to be curved are calculated (S5). , The camera coordinates of the road surface feature points are calculated from the calculated curve R matrix and the curve T matrix (S6), and the installation angle of the camera 3 is calculated from the camera coordinates of the curve R matrix, the curve T matrix and the road surface feature points (S6). S7).

Then, the installation angle of the camera 3 calculated while the vehicle 1 is traveling is recorded in the database 19 as angle information (S8), and the optimum value of the installation angle of the camera 3 is obtained from the plurality of angle information recorded in the database 19. Find (S9).

Even with the in-vehicle camera calibration device 10 of the third embodiment as described above, as shown in Equation D, since the F matrix has three variables, F is compared with the conventional case (when the F matrix is eight). Matrix calculation accuracy is improved. Further, the camera can be calibrated not only when the vehicle 1 goes straight, but also when the vehicle 1 is curved.

Moreover, according to the in-vehicle camera calibration device 10 of the present embodiment, the F matrix can be calculated with high accuracy by using only the data when the vehicle 1 travels straight. This makes it possible to provide a camera calibration device having high camera calibration accuracy.

Although the embodiments of the present invention have been described above by way of illustration, the scope of the present invention is not limited to these, and can be changed or modified according to an object within the scope described in the claims. is there.

As described above, the in-vehicle camera calibration device according to the present invention has the effect of being able to calculate the F matrix with high accuracy and being able to handle not only when the vehicle is going straight but also when the vehicle is curved, and is used for vehicles. It is applied to navigation systems and is useful.

1 Vehicle 2 Road surface 3 Camera 4 Steering angle sensor 5 Vehicle speed sensor 10 In-vehicle camera Calibration device 11 Optical flow calculation unit 12 Straight F matrix calculation unit 13 Straight RT matrix calculation unit 14 Curve RT matrix calculation unit 15 Triangulation unit 16 Angle calculation unit 17 Angle recording unit 18 Optimal angle calculation unit 19 Database 20 Straight-ahead judgment unit

Claims (5)

An optical flow calculation unit that calculates the optical flow by extracting the road surface feature points before and after the movement from the road surface images before and after the movement taken by the camera attached to the vehicle.
From the optical flow, a straight-ahead F matrix calculation unit that calculates a straight-ahead F matrix, which is a basic matrix when the vehicle is assumed to be straight-ahead,
From the straight F matrix, the straight R matrix, which is a rotation matrix between the camera coordinates before and after the movement when the vehicle is assumed to be straight, and the camera coordinates before and after the movement when the vehicle is assumed to be straight. A straight RT matrix calculation unit that calculates a straight T matrix that is a parallel traveling matrix between
By optimizing the straight R matrix and the straight T matrix, the curve R matrix, which is a rotation matrix when the vehicle is assumed to be curved, and the translation when the vehicle is assumed to be curved. The curve RT matrix calculation unit that calculates the curve T matrix, which is a matrix,
A triangulation unit that calculates camera coordinates of the road surface feature points from the curve R matrix and the curve T matrix, and
A camera angle calculation unit that calculates the installation angle of the camera from the camera coordinates of the curve R matrix, the curve T matrix, and the road surface feature point.
An angle recording unit that records the installation angle of the camera calculated while the vehicle is running in a database as angle information, and
An optimum angle calculation unit for obtaining an optimum value of the installation angle of the camera from a plurality of the angle information recorded in the database.
An in-vehicle camera calibration device characterized by being equipped with. The angle recording unit records pitch, roll, and pan as the angle information in the database together with the recording time.
The optimum angle calculation unit sets the pitch with the latest recording time as the optimum value for the pitch, sets the roll with the latest recording time as the optimum value for the roll, and sets the latest recording time in the pan when the vehicle is traveling straight. The vehicle-mounted camera calibration device according to claim 1, wherein the pan is set to the optimum value of the pan. The angle recording unit records pitch, roll, and pan as the angle information in the database together with the recording time.
When the vehicle is traveling straight, the optimum angle calculation unit sets the average value of a plurality of pitches having a new recording time as the optimum value of the pitch, and sets the average value of a plurality of rolls having a new recording time as the optimum value of the roll. The vehicle-mounted camera calibration device according to claim 1, wherein the average value of a plurality of pans having a new recording time is set as the optimum value of the pans. An optical flow calculation unit that calculates the optical flow by extracting the road surface feature points before and after the movement from the road surface images before and after the movement taken by the camera attached to the vehicle.
From the steering angle information obtained from the steering angle sensor attached to the vehicle and the vehicle speed information obtained from the vehicle speed sensor attached to the vehicle, the distance between the camera coordinates before and after the movement when the vehicle is assumed to be traveling straight. A straight R matrix that is a rotation matrix of the above, a straight RT matrix calculation unit that calculates a straight T matrix that is a parallel traveling matrix between camera coordinates before and after the movement when the vehicle is assumed to be straight, and a straight RT matrix calculation unit.
By optimizing the straight R matrix and the straight T matrix, the curve R matrix, which is a rotation matrix when the vehicle is assumed to be curved, and the translation when the vehicle is assumed to be curved. The curve RT matrix calculation unit that calculates the curve T matrix, which is a matrix,
A triangulation unit that calculates camera coordinates of the road surface feature points from the curve R matrix and the curve T matrix, and
A camera angle calculation unit that calculates the installation angle of the camera from the camera coordinates of the curve R matrix, the curve T matrix, and the road surface feature point.
An angle recording unit that records the installation angle of the camera calculated while the vehicle is running in a database as angle information, and
An optimum angle calculation unit for obtaining an optimum value of the installation angle of the camera from a plurality of the angle information recorded in the database.
An in-vehicle camera calibration device characterized by being equipped with. A straight-ahead determination unit for determining whether or not the vehicle is traveling straight based on the steering angle information obtained from the steering angle sensor attached to the vehicle is provided.
The optical flow calculation unit calculates the optical flow when the straight-ahead determination unit determines that the vehicle is going straight, and when the straight-ahead determination unit determines that the vehicle is not going straight. The vehicle-mounted camera calibration device according to any one of claims 1 to 3, which does not calculate the optical flow.

Images