JP6974192B2 - Camera misalignment detection device and camera misalignment detection method - Google Patents

Description

The present invention relates to a technique for detecting a position shift of a camera.

Conventionally, driving support such as vehicle parking support has been provided by using an in-vehicle camera. The dashcam is fixedly attached to the vehicle before it is shipped from the factory. However, the in-vehicle camera may be displaced from the factory-installed state due to, for example, unexpected contact or aging. If the position of the in-vehicle camera (the position of the in-vehicle camera relative to the body of the vehicle) deviates, an error occurs in the steering amount of the steering wheel determined by using the camera image. is important.

Patent Document 1 discloses a technique for detecting an optical axis deviation of an in-vehicle camera. The optical axis deviation detecting device of the vehicle-mounted camera in Patent Document 1 includes an image processing means and a determination means. The image processing means detects the position information of the marking from the image taken by the vehicle-mounted camera that captures the range including the marking on the vehicle body portion that supports driving. The determination means determines the shooting optical axis deviation by comparing the initially set marking position information with the newly detected position information of the marking.

Japanese Unexamined Patent Publication No. 2004-173037

Since the image taken by the vehicle-mounted camera reflects the scenery around the vehicle, for example, when the marking is a specific shape of a part of the bonnet, the specific shape may not be easily extracted. If the specific shape is erroneously detected, the optical axis deviation of the vehicle-mounted camera may not be accurately determined.

In view of the above problems, it is an object of the present invention to provide a technique capable of accurately detecting the direction of a position shift of a camera.

The camera misalignment detection device of the present invention includes an acquisition unit that acquires a captured image captured by a camera, a first detection unit that detects the movement of a feature point in a first detection region in the captured image, and the above-mentioned. The second detection unit that detects the movement of a feature point in a second detection area smaller than the first detection area included in the first detection area, and the first detection unit based on the detection results of the first detection unit. When the determination unit determines whether or not the camera is misaligned and the determination unit determines that the camera is misaligned, the misalignment direction of the camera is determined based on the detection result of the second detection unit. It is a configuration (first configuration) including a camera shift direction detection unit for detection.

In the camera deviation detection device having the first configuration, when the determination unit determines that there is no positional deviation of the camera, the second detection unit moves the feature points in the second detection region. It may be a configuration that is not detected (second configuration).

In the camera deviation detection device having the first or second configuration, the second detection unit sets a plurality of the second detection areas, and the camera deviation direction detection unit sets the deviation directions in the PAN direction and the ROLL direction. The detection result of the second detection unit used for detection and the detection result of the second detection unit used when the camera deviation direction detection unit detects the deviation direction in the TILT direction are different from each other. It may be the configuration (third configuration) obtained by the detection in the detection region of.

In the camera deviation detection device having any of the first to third configurations, the second detection unit sets the second detection region according to the distribution of feature points in the first detection region. (Fourth configuration) may be used.

The camera deviation detection method of the present invention includes an acquisition step of acquiring a captured image captured by a camera, a first detection step of detecting the movement of a feature point in a first detection region in the captured image, and the above-mentioned. The second detection step of detecting the movement of a feature point in a second detection region smaller than the first detection region included in the first detection region, and the detection result of the first detection step are used. When it is determined by the determination step of determining whether or not the camera is misaligned and the determination process determines that the camera is misaligned, the misalignment direction of the camera is determined based on the detection result of the second detection step. It is a configuration (fifth configuration) including a camera shift direction detection step for detecting.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a technique capable of accurately detecting the direction of a position shift of a camera.

Block diagram showing the configuration of the camera misalignment detection device The figure which illustrates the position where four in-vehicle cameras are arranged in a vehicle A flowchart showing an example of the camera deviation detection flow by the camera deviation detection device. Schematic diagram showing an example of an image taken with a front camera A flowchart showing an example of camera misalignment detection processing Schematic diagram showing the position of ROI in the captured image Schematic diagram showing the position of ROI in the captured image Schematic diagram showing the position of ROI in the captured image Schematic diagram showing the position of ROI in the captured image

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the direction in which the vehicle travels straight from the driver's seat to the steering wheel is referred to as "forward direction". Further, the direction in which the vehicle travels straight from the steering wheel to the driver's seat is defined as the "rear direction". Further, the direction from the right side to the left side of the driver facing forward, which is the direction perpendicular to the straight line and the vertical line of the vehicle, is defined as the "left direction". Further, the direction from the left side to the right side of the driver facing forward, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as "right direction". Further, in the present specification, the misalignment of the camera may be simply referred to as the misalignment of the camera.

<1. Outline of camera misalignment detection device>
FIG. 1 is a block diagram showing a configuration of a camera deviation detecting device 1 according to the present embodiment.

The camera misalignment detection device 1 is a device that detects camera misalignment based on a captured image taken by an in-vehicle camera. The camera deviation detecting device 1 is provided for each vehicle equipped with an in-vehicle camera. In the present embodiment, the camera deviation detection device 1 acquires a captured image from the photographing unit 2 and processes it to detect the camera deviation. Further, in the present embodiment, the camera deviation detecting device 1 acquires information from the sensor unit 3 and detects the camera deviation based on the acquired information.

The photographing unit 2 is provided for the purpose of monitoring the situation around the vehicle. The photographing unit 2 includes four cameras 21 to 24. The four cameras 21 to 24 are in-vehicle cameras. FIG. 2 is a diagram illustrating the positions where the four vehicle-mounted cameras 21 to 24 are arranged in the vehicle 4.

The vehicle-mounted camera 21 is provided at the front end of the vehicle 4. Therefore, the in-vehicle camera 21 is also referred to as a front camera 21. The optical axis 21a of the front camera 21 is along the front-rear direction of the vehicle 4 in a plan view from above. The front camera 21 captures the front direction of the vehicle 4. The vehicle-mounted camera 22 is provided at the rear end of the vehicle 4. Therefore, the in-vehicle camera 22 is also referred to as a back camera 22. The optical axis 22a of the back camera 22 is along the front-rear direction of the vehicle 4 in a plan view from above. The back camera 22 captures the rear direction of the vehicle 4. The front camera 21 and the back camera 22 are preferably mounted at the center of the left and right sides of the vehicle 4, but may be slightly displaced from the center of the left and right sides in the left-right direction.

The vehicle-mounted camera 23 is provided on the left side door mirror 41 of the vehicle 4. Therefore, the in-vehicle camera 23 is also referred to as a left side camera 23. The optical axis 23a of the left side camera 23 is along the left-right direction of the vehicle 4 in a plan view from above. The left side camera 23 photographs the left direction of the vehicle 4. The vehicle-mounted camera 24 is provided on the right door mirror 42 of the vehicle 4. Therefore, the in-vehicle camera 24 is also referred to as a right side camera 24. The optical axis 24a of the right side camera 24 is along the left-right direction of the vehicle 4 in a plan view from above. The right side camera 24 captures the right direction of the vehicle 4. When the vehicle 4 is a so-called door mirrorless vehicle, the left side camera 23 is attached around the rotation axis (hinge portion) of the left side door without passing through the door mirror, and the right side camera 24 is attached to the right side door. It can be attached around the rotating shaft (hinge part) without using a door mirror.

The horizontal angle of view θ of each of the vehicle-mounted cameras 21 to 24 is 180 degrees or more. Therefore, the vehicle-mounted cameras 21 to 24 can photograph the entire circumference of the vehicle 4 in the horizontal direction. Further, the body of the vehicle 4 on which the vehicle-mounted cameras 21 to 24 are mounted is reflected in the images taken by the vehicle-mounted cameras 21 to 24.

In the present embodiment, the number of in-vehicle cameras is four, but this number may be changed as appropriate, and may be plural or singular. For example, when an in-vehicle camera is mounted for the purpose of assisting the vehicle 4 to park in the back, the in-vehicle camera is composed of a back camera 22, a left side camera 23, and a right side camera 24. May be good.

Returning to FIG. 1, the sensor unit 3 has a plurality of sensors for detecting information about the vehicle 4 on which the vehicle-mounted cameras 21 to 24 are mounted. The information about the vehicle 4 may include information about the vehicle itself and information about the surroundings of the vehicle. In the present embodiment, the sensor unit 3 is, for example, a vehicle speed sensor that detects the speed of the vehicle, a steering angle sensor that detects the rotation angle of the steering wheel, and a shift sensor that detects the operation position of the shift lever of the transmission of the vehicle. Etc. are included. In the present embodiment, the information detected by the sensor unit 3 is directly input to the camera deviation detecting device 1, but this is only an example. For example, the information detected by the sensor unit 3 may be input to the camera deviation detecting device 1 via a vehicle control device or a driving support device (not shown).

FIG. 3 is a flowchart showing an example of a camera deviation detection flow by the camera deviation detection device 1. As shown in FIG. 3, for example, in a factory such as an automobile manufacturer, four in-vehicle cameras 21 to 24 are attached to each vehicle 4 (step S1). Each of the vehicle-mounted cameras 21 to 24 is fixed at a predetermined mounting position with the optical axes 21a to 24a aligned in a predetermined direction.

Further, in the factory, data regarding the initial position of the vehicle-mounted camera (the relative initial position of the vehicle-mounted camera with respect to the vehicle body) is acquired (step S2). The data regarding the initial position of the in-vehicle camera is used to determine whether or not the in-vehicle cameras 21 to 24 are misaligned after the vehicle 4 equipped with the in-vehicle cameras 21 to 24 is put on the market. It is the data to be done. In the present embodiment, the data regarding the initial position of the vehicle-mounted camera is acquired for each vehicle-mounted camera 21 to 24 by using the camera deviation detecting device 1. The acquired data regarding the initial position of the vehicle-mounted camera is stored in a storage unit 13 described later included in the camera deviation detecting device 1.

In the present embodiment, after the vehicle 4 equipped with the in-vehicle cameras 21 to 24 is put on the market, the camera deviation is detected by the camera deviation detecting device 1. The camera deviation detection process is performed on each vehicle-mounted camera 21 to 24. The camera deviation detection device 1 determines whether or not it is the timing to start the camera deviation detection process (step S3). If No in step S3, the camera misalignment detection device 1 repeats the determination of whether or not it is the timing to start the camera misalignment detection process.

The timing for starting the camera deviation detection process may be, for example, when the vehicle 4 is traveling straight ahead or backward at a predetermined speed (for example, 5 km / h) or less. As a result, the camera deviation detection process can be started under the condition that the movement amount of the vehicle 4 can be estimated accurately based on the output of the sensor 3.

When the camera deviation detection device 1 determines that it is time to start the camera deviation detection process (Yes in step S3), the camera deviation detection device 1 executes the detection process (step S4).

When it is determined that the position shift of the in-vehicle cameras 21 to 24 has not occurred as a result of executing the detection process (Yes in step S5), the camera shift detection device 1 returns to step S3 and detects the next camera shift. When the processing start timing is detected, the camera deviation detection processing is repeated. When it is determined that the camera misalignment has not occurred, the camera misalignment detecting device 1 may notify, for example, a vehicle control device (not shown), a driving support device, or the like that the in-vehicle camera is normal.

When it is determined that the in-vehicle cameras 21 to 24 are misaligned (No in step S5), the camera misalignment detection device 1 determines that the in-vehicle cameras 21 to 24 are misaligned as the direction of the misalignment. It also notifies (step S6). In the present embodiment, the camera misalignment detection device 1 notifies that the misalignment of the vehicle-mounted cameras 21 to 24 has occurred and the direction of the misalignment by a display device (not shown). Voice or the like may be used to notify the occurrence of camera misalignment. Further, the camera misalignment detection device 1 uses a communication device (not shown) mounted on the vehicle 4 to transmit a display signal or an audio signal for notifying an external device such as a smartphone possessed by the driver of the occurrence of the camera misalignment. You may send it. Further, the camera deviation detecting device 1 notifies the driving support device 6 of the occurrence of the positional deviation of the in-vehicle cameras 21 to 24, and stops the driving support by the driving support device 6. In the present embodiment, there are four in-vehicle cameras 21 to 24, but if any one of the four in-vehicle cameras 21 to 24 is displaced, the above notification process and stop process are performed. It is preferable to be

If it is determined that the in-vehicle cameras 21 to 24 are misaligned (No in step S5), the vehicle 4 is returned to the position at the time when the determination start timing is detected by the control of the automatic driving control unit. It is preferable to keep it. This is because there is a high possibility that the position at the time when the determination start timing is detected is a place where the safety of the vehicle 4 can be ensured (for example, in the parking frame).

Due to the occurrence of camera misalignment, the vehicle user requests, for example, a dealer to adjust the mounting of the camera. In response to this, the dealer adjusts the mounting of the vehicle-mounted camera (step S7). Further, the dealer reacquires the data regarding the initial position of the vehicle-mounted camera described above by using the camera deviation detecting device 1 (step S8). After that, returning to step S3, when the camera shift detection device 1 detects the determination start timing, the camera shift detection device 1 determines the position shift of the camera based on the re-acquired data regarding the initial position of the vehicle-mounted camera.

In the present embodiment, when it is determined that the camera is displaced, the mounting adjustment of the in-vehicle camera is performed, but this is only an example. When it is determined that the camera deviation has occurred, the camera deviation detecting device 1 may be configured to correct the positional deviation of the vehicle-mounted camera and use the captured image. For example, the notification processing of the occurrence of the deviation and the stop processing of the driving support on the premise of the mounting adjustment of the in-vehicle camera may be performed only when the deviation state of the in-vehicle camera is unknown. The state in which the camera is displaced is unknown, for example, refers to a state in which it is unknown how much the camera is displaced or angled. When the deviation state of the in-vehicle camera is clear and can be corrected and used, the camera deviation detection device 1 may be configured not to perform the deviation occurrence notification process or the driving support stop process.

<2. Details of camera misalignment detector>
Returning to FIG. 1, the camera deviation detecting device 1 includes an acquisition unit 11, a microcomputer 12, and a storage unit 13. The microcomputer 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is a non-volatile memory. The microcomputer 12 processes and sends / receives information based on the program stored in the storage unit 13. The microcomputer 12 is connected to the photographing unit 2 and the sensor unit 3 by wire or wirelessly.

The acquisition unit 11 continuously acquires analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, a 1/30 second cycle). That is, the aggregate of the captured images acquired by the acquisition unit 11 is a moving image captured by the vehicle-mounted cameras 21 to 24. When the acquired captured image is analog, the acquisition unit 11 converts the analog captured image into a digital captured image (A / D conversion). The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the microcomputer 12. One captured image output from the acquisition unit 11 becomes one frame image.

The microcomputer 12 includes a first detection unit 121, a second detection unit 122, a determination unit 123, and a camera deviation direction detection unit 124. The functions of each of these units 121 to 124 included in the microcomputer 12 are realized by the CPU performing arithmetic processing according to a program.

The first detection unit 121 detects the movement (optical flow) of the feature point in the first ROI (Region of Interest) in the captured image captured by the vehicle-mounted cameras 21 to 24. For example, the movements of all the feature points in the first ROI may be made into a histogram, and the average value in the histogram may be the movement of the feature points in the first ROI. Here, "ROI" means "a detection area which is a region for detecting the movement of feature points in a captured image captured by a camera". In the present embodiment, the number of vehicle-mounted cameras 21 to 24 is four, and the detection process by the first detection unit 121 is performed for each vehicle-mounted camera 21 to 24.

The feature points are points that can be detected conspicuously, and examples thereof include edges of white lines drawn on the road surface, cracks on the road surface, stains on the road surface, and gravel on the road surface. After detecting the feature points, the first detection unit 121 tracks the detected feature points until the detected feature points deviate from the first ROI so as not to lose sight of the detected feature points.

The second detection unit 122 detects the movement of the feature point in the second ROI, which is smaller than the first ROI contained in the first ROI. For example, the movements of all the feature points in the second ROI may be made into a histogram, and the average value in the histogram may be the movement of the feature points in the second ROI. In the present embodiment, the number of vehicle-mounted cameras 21 to 24 is four, and the detection process by the second detection unit 122 is also the same as the detection process by the first detection unit 121, for each vehicle-mounted camera 21 to 24. It is done in. In the detection process by the second detection unit 122, the movement of the feature points in the second ROI is not detected from scratch, but the detection result of the first detection unit 121 is reconstructed to obtain a second. It is desirable to detect the movement of feature points in the ROI.

The determination unit 123 determines whether or not the in-vehicle cameras 21 to 24 are misaligned based on the detection result of the first detection unit 121. For example, if the difference between the movement amount of the vehicle 4 estimated from the movement of the feature point in the first ROI and the movement amount of the vehicle 4 estimated from the output of the sensor unit 3 is a predetermined amount or more, the position shift It may be determined that there is. When estimating the movement amount of the vehicle 4 from the movement of the feature points in the first ROI, the movement amount of the vehicle 4 is calculated on the assumption that the feature points in the first ROI are the feature points on the road surface. You can estimate it.

In the present embodiment, the number of vehicle-mounted cameras 21 to 24 is four, and the determination process by the determination unit 123 is similar to the detection process by the first detection unit 121 and the detection process by the second detection unit 122. It is performed every 21 to 24 of each in-vehicle camera. Instead of the movement amount of the vehicle 4 estimated from the output of the sensor unit 3, the movement amount estimated from the movement of the feature points obtained from the image taken by the camera other than the in-vehicle camera to be determined may be used. good. This is because it is unlikely that multiple cameras will be misaligned at the same time.

When the determination unit 123 determines that the vehicle-mounted cameras 21 to 24 are misaligned, the camera misalignment direction detection unit 124 determines that the vehicle-mounted cameras 21 to 24 are misaligned based on the detection result of the second detection unit 122. Is detected. The detection process by the camera misalignment direction detection unit 124 is performed for each camera determined to have a misalignment.

For camera misalignment, PAN (pan) where the camera rotates around the vertical axis, TILT (tilt) where the camera rotates around the horizontal axis, and ROLL where the camera rotates around the optical axis or the axis parallel to the optical axis. There are three directional deviations consisting of (rolls).

FIG. 4 is a schematic diagram showing an example of a photographed image P1 photographed by the front camera 21. The front camera 21 has an angle of view θ of 180 degrees or more, and as shown in FIG. 4, the body 40 of the vehicle 4 is reflected in the photographed image P1. FIG. 4A is an image taken with the front camera 21 fixed in the initial position. That is, it is an image taken by the front camera 21 in a normal state. FIG. 4B is an image taken in a state where the front camera 21 is displaced from the initial position in the PAN direction and the optical axis is tilted to the right side of the vehicle. FIG. 4C is an image taken in a state where the front camera 21 is displaced from the initial position in the TILT direction and the optical axis is tilted upward. FIG. 4D is an image taken in a state where the front camera 21 is displaced from the initial position in the ROLL direction and the shooting area is rotated counterclockwise.

<3. Details of camera misalignment detection processing>
FIG. 5 is a flowchart showing the details of the camera deviation detection process, that is, the details of step S4 in FIG.

When the process of step S4 is started, the first detection unit 121 first detects the movement of the feature point in the first ROI in the captured image captured by the vehicle-mounted cameras 21 to 24 (step S41).

After that, the determination unit 123 determines whether or not the in-vehicle cameras 21 to 24 are misaligned based on the detection result of the first detection unit 121 (step S42).

If it is determined that there is no misalignment, the process of step S4 is terminated as it is. On the other hand, if it is determined that there is a misalignment, the second detection unit 122 detects the movement of the feature point in the second ROI in the captured image captured by the vehicle-mounted cameras 21 to 24 (step S43). ..

In step S44 following step S43, the camera shift direction detection unit 124 detects the position shift direction of the vehicle-mounted cameras 21 to 24 based on the detection result of the second detection unit 122, and then ends the process of step S4. ..

In the flowchart shown in FIG. 5 described above, when it is determined by the determination unit 123 that there is no positional deviation, in other words, it is not necessary for the second detection unit 122 to detect the movement of the feature point in the second ROI. Occasionally, the second detection unit 122 does not detect the movement of the feature point in the second ROI. As a result, the processing load of the camera misalignment detection device 1 can be reduced.

The misalignment of the vehicle-mounted cameras 21 to 24 occurs very rarely. Therefore, when it is not necessary for the second detection unit 122 to detect the movement of the feature point in the second ROI, the second detection unit 122 does not detect the movement of the feature point in the second ROI. The effect of reducing the processing load is very large.

Further, in the flowchart shown in FIG. 5 described above, when the determination unit 123 determines that there is a positional deviation, the second detection unit 122 is smaller than the first ROI included in the first ROI. Detects the movement of feature points in the ROI. It may be difficult to detect the direction of the displacement of the camera from the detection result of the movement of the feature point in the first ROI, but even in such a case, the camera is detected from the detection result of the movement of the feature point in the second ROI. The direction of the misalignment of the camera can be detected with high accuracy.

For example, when the first ROI 5 is set as shown in FIG. 6, if the feature points are distributed almost uniformly over the entire first ROI 5, the PAN direction and the ROLL direction are included in the direction of the camera misalignment. However, the movement of the feature point located in the left half of the first ROI5 and the movement of the feature point located in the right half of the first ROI5 cancel each other out, and the average value in the histogram becomes small.

Therefore, for example, when the second ROI 61 is set in the left region of the captured image and the second ROI 62 is set in the right region of the captured image as shown in FIG. 7, the average value of the movements of the feature points of the second ROI 61 is set. And the average of the movements of the feature points of the second ROI 62, it is possible to accurately detect the PAN direction and the ROLL direction that may be included in the direction of the position shift of the camera.

For example, when the first ROI 5 is set as shown in FIG. 6, if the feature points are densely distributed below the upper side of the first ROI 5, the TILT direction is in the direction of the camera misalignment. Even if it is included, the average value in the histogram becomes small.

Therefore, for example, when the second ROI63 is set in the central region of the captured image as shown in FIG. 8, the TILT direction that can be included in the direction of the position shift of the camera due to the average value of the movements of the feature points of the second ROI63. Can be detected with high accuracy.

That is, when the detection result of the second detection unit 122 used when the camera deviation direction detection unit 124 detects the deviation direction in the PAN direction and the ROLL direction and when the camera deviation direction detection unit 124 detects the deviation direction in the TILT direction. It is desirable to obtain the detection result of the second detection unit 122 used in the above by detection in a different second ROI. As shown in FIG. 8 described above, the second ROIs 61 and 62 and the second ROI 63 may partially overlap. As a matter of course, as shown in FIG. 9, the second ROI 61 and 62 and the second ROI 63 do not have to overlap at all.

Further, it is desirable that the second detection unit 122 sets the second ROI according to the distribution of the feature points in the first ROI, and more specifically, sets the size and position of the second ROI. This makes it possible to prevent problems such as the feature points not being included in the second ROI. The second ROI may be selected by the second detection unit 122 from a plurality of predetermined second ROI candidates according to the distribution of feature points in the first ROI, and the second detection unit 122 may be selected. The size and position of the unit 122 may be arbitrarily determined according to the distribution of the feature points in the first ROI. The camera does not necessarily have to be mounted on an automobile, and may be used on a moving body such as a two-wheeled vehicle, a railroad, or a ship, in addition to the automobile.

<4. Modification example>
In addition to the above embodiments, the various technical features disclosed herein can be modified in various ways without departing from the spirit of the technical creation. In addition, a plurality of embodiments and modifications shown in the present specification may be combined and implemented to the extent possible.

In the embodiment shown above, it has been explained that various functions of the microcomputer are realized by software by the arithmetic processing of the CPU according to the program, but at least a part of these functions is electric hardware. It may be realized by a circuit. On the contrary, at least a part of the functions realized by the hardware circuit may be realized by software.

1 Camera misalignment detector 21 Front camera (vehicle-mounted camera)
22 Back camera (vehicle-mounted camera)
23 Left side camera (vehicle-mounted camera)
24 Right side camera (vehicle-mounted camera)
121 First detection unit 122 Second detection unit 123 Judgment unit 124 Camera misalignment direction detection unit

Claims (5)

The acquisition unit that acquires the captured image taken by the camera,
A first detection unit that detects the movement of a feature point in the first detection area in the captured image, and a first detection unit.
A second detection unit that detects the movement of a feature point in a second detection area smaller than the first detection area included in the first detection area, and a second detection unit.
A determination unit that determines whether or not the camera is misaligned based on the detection result of the first detection unit.
When the determination unit determines that the camera is misaligned, the camera misalignment direction detection unit that detects the misalignment direction of the camera based on the detection result of the second detection unit, and the camera misalignment direction detection unit.
A camera misalignment detection device. The camera misalignment detection according to claim 1, wherein the second detection unit does not detect the movement of the feature point in the second detection region when the determination unit determines that there is no misalignment of the camera. Device. The second detection unit sets a plurality of the second detection areas, and sets the second detection area.
The detection result of the second detection unit used when the camera deviation direction detection unit detects the deviation direction in the PAN direction and the ROLL direction, and
Claim 1 or claim 2 that the detection result of the second detection unit used when the camera deviation direction detection unit detects the deviation direction in the TILT direction is obtained by detection in the second detection region different from that of the second detection unit. The camera misalignment detection device described in. The camera deviation detection device according to any one of claims 1 to 3, wherein the second detection unit sets the second detection area according to the distribution of feature points in the first detection area. .. The acquisition process to acquire the captured image taken by the camera,
A first detection step of detecting the movement of a feature point in the first detection region in the captured image, and a first detection step.
A second detection step of detecting the movement of a feature point in a second detection area smaller than the first detection area included in the first detection area, and a second detection step.
A determination step of determining whether or not the camera is displaced based on the detection result of the first detection step, and a determination step.
A camera misalignment direction detection step of detecting the misalignment direction of the camera based on the detection result of the second detection step when it is determined by the determination step that there is a misalignment of the camera.
A camera misalignment detection method.

Images