JP6767255B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method.

Conventionally, driving support such as vehicle parking support has been provided by using an in-vehicle camera. The on-board camera is fixedly attached to the vehicle before it is shipped from the factory. However, the on-board camera may cause an optical axis shift from the factory-installed state due to, for example, unexpected contact or aging. It is important to detect the deviation of the in-vehicle camera because if the optical axis of the in-vehicle camera shifts, an error occurs in the steering amount of the steering wheel determined by using the camera image.

Patent Document 1 discloses a technique for detecting an optical axis deviation of an in-vehicle camera. The optical axis deviation detecting device of the vehicle-mounted camera in Patent Document 1 includes an image processing means and a determining means. The image processing means detects the position information of the marking from the captured image of the vehicle-mounted camera that captures the range including the marking on the vehicle body portion that supports driving. The determination means determines the shooting optical axis deviation by comparing the initially set marking position information with the newly detected marking position information.

Japanese Unexamined Patent Publication No. 2004-173037

Since the image taken by the in-vehicle camera reflects the scenery around the vehicle, for example, when the marking is a specific shape of a part of the bonnet, the specific shape may not be easily extracted. If the specific shape is erroneously detected, the optical axis deviation of the in-vehicle camera may not be accurately determined.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of appropriately detecting an optical axis deviation of an in-vehicle camera.

In order to achieve the above object, the image processing device of the present invention is an image processing device that processes a camera image captured by an in-vehicle camera, and based on the camera image, a body of a vehicle equipped with the in-vehicle camera. An inspection image acquisition unit for acquiring an inspection image is provided, and the inspection image acquisition unit performs image masking processing excluding a search area for an edge of the body preset according to the body shape for each vehicle type, and performs the inspection. It has a configuration for acquiring an image (first configuration).

In the image processing apparatus having the first configuration, the inspection image acquisition unit removes an edge image obtained after the mask processing that is determined to be different from the edge assumed from the design value of the body. A configuration for acquiring the inspection image (second configuration) is preferable.

The image processing device having the first or second configuration includes a storage unit that stores a reference image of the body obtained by processing the camera image captured by the in-vehicle camera normally mounted on the vehicle. A configuration (third configuration) further comprising a deviation detection unit that detects the optical axis deviation of the vehicle-mounted camera by comparing the reference image and the inspection image is preferable.

In the image processing apparatus having the third configuration, the inspection image acquisition unit acquires the inspection image by performing a remasking process of masking a narrow search area narrower than the search area after the mask processing. (Fourth configuration) is preferable.

In the image processing apparatus having the third or fourth configuration, when the number of pixels detected as edges in the inspection image is less than a predetermined first threshold value, the deviation detection unit reacquires the inspection image. It may be a configuration (fifth configuration) that requires.

In the image processing apparatus having the fifth configuration, after the mask processing, the inspection image acquisition unit performs a remasking process of masking a narrow search area narrower than the search area, and reacquires the inspection image. In this case, the inspection image acquisition unit acquires the inspection image by expanding at least the narrow search area of the search area and the narrow search area as compared with the time of acquiring the inspection image. (Sixth configuration) may be used.

In the image processing apparatus having the fifth or sixth configuration, when the inspection image is reacquired, the inspection image acquisition unit acquires the camera image at the acquisition timing based on the information about the vehicle acquired from the outside. The inspection image may be acquired by determining the above (seventh configuration).

In the image processing apparatus having any of the fifth to seventh configurations, the deviation detection unit is used when the number of pixels is less than the predetermined first threshold value even if the inspection image is reacquired a predetermined number of times. May be a configuration (eighth configuration) in which it is determined that the in-vehicle camera is misaligned with the optical axis.

In the image processing apparatus having any of the third to eighth configurations, the deviation detection unit divides the inspection image into a plurality of regions and performs comparison processing with the reference image for each of the plurality of regions. , The configuration may be such that the optical axis deviation is determined based on the obtained plurality of comparison processing results (9th configuration).

In the image processing apparatus having any of the third to ninth configurations, the reference image may have a configuration having a smaller number of pixels than the camera image (tenth configuration).

In the image processing apparatus having any of the third to tenth configurations, the deviation detection unit acquires comparison results of the plurality of inspection images with the reference image, and responds to the comparison results of the plurality of times. It may be a configuration (11th configuration) for determining the optical axis deviation.

In the image processing apparatus having any one of the first to tenth configurations, the inspection image acquisition unit performs a plurality of processes including the mask processing on each of the plurality of camera images captured at different timings. A configuration (12th configuration) is preferable in which the pre-composite image is acquired and the plurality of pre-composite images are combined to obtain the inspection image.

In the image processing apparatus having the twelfth configuration, the inspection image acquisition unit alpha-blends the pre-combination image with the image stored in the compositing processing memory and updates the image stored in the compositing processing memory. A configuration (13th configuration) in which the processing is sequentially repeated to acquire the inspection image is preferable.

In the image processing apparatus having the thirteenth configuration, the initial image stored in the compositing processing memory may be an image obtained by adjusting the reference image to a predetermined density (fourth configuration).

In the image processing apparatus having the thirteenth configuration, whether or not the inspection image acquisition unit sets the pre-composite image as an initial image stored in the compositing processing memory based on the reliability of the pre-synthesis image. It may be a configuration (15th configuration) for determining.

In the image processing apparatus having any of the thirteenth to fifteenth configurations, the inspection image acquisition unit has at least one of the reliability of the pre-composite image and the information about the vehicle acquired from the outside. Based on the above, the configuration may be such that the blending ratio of the alpha blend is changed (the 16th configuration).

In the image processing device having any of the first to sixteenth configurations, the inspection image acquisition unit determines the timing of acquiring the camera image based on the information about the vehicle acquired from the outside (17th). The configuration of) is preferable.

In the image processing apparatus having the 17th configuration, the inspection image acquisition unit may have a configuration (18th configuration) of acquiring the camera image when the ACC of the vehicle is turned on and the engine stop is detected. ..

In the image processing device having any of the first to eighteenth configurations, the inspection image acquisition unit turns on the light of the vehicle when it is determined that the illuminance around the vehicle is lower than a predetermined second threshold value. The configuration may be such that the camera image is acquired after the instruction (19th configuration).

In the image processing device having any of the first to nineteenth configurations, when there are a plurality of on-board cameras, it is preferable that the image processing conditions are determined for each of the on-board cameras (the twentyth configuration).

In order to achieve the above object, the image processing method of the present invention is an image processing method for processing a camera image captured by an in-vehicle camera by an image processing device, and the in-vehicle camera is mounted based on the camera image. The inspection image acquisition step of acquiring the inspection image of the body of the vehicle is provided, and the inspection image acquisition step masks the image except for the search area of the edge of the vehicle which is preset according to the body shape of each vehicle type. Is performed, and the inspection image is acquired (21st configuration).

According to the present invention, it is possible to provide a technique capable of appropriately detecting an optical axis deviation of an in-vehicle camera.

Block diagram showing the configuration of the camera misalignment detection system The figure which illustrates the position where four cameras are arranged in a vehicle Flow chart showing an example of camera deviation detection flow by the camera deviation detection system Flow chart showing an example of the acquisition flow of the reference image Schematic diagram showing an example of a camera image captured by a front camera Image of mask data prepared for each model Schematic diagram illustrating an edge image after masking Schematic diagram showing an example of a reference image Schematic diagram for explaining an example of a reference image stored in a storage unit A flowchart showing an example of a camera deviation detection flow of the first embodiment Image diagram to explain the remasking process Flowchart showing an example of processing flow of pattern matching processing Schematic diagram showing a comparison image between the inspection image and the reference image Flow chart showing a modified example of the camera deviation detection flow of the first embodiment Schematic diagram for explaining a modified example of deviation determination by the deviation detection unit Flow chart showing an example of camera deviation detection flow of the second embodiment Schematic diagram for explaining the synthesis process Schematic diagram for explaining a modified example of the synthesis process

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the direction in which the vehicle travels straight from the driver's seat to the steering wheel is referred to as the "forward direction". Further, the direction in which the vehicle travels straight from the steering wheel toward the driver's seat is defined as the "rear direction". Further, the direction from the right side to the left side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "left direction". Further, the direction from the left side to the right side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "right direction". Further, in the present specification, the optical axis deviation of the camera may be simply expressed as the camera deviation.

<< 1. First Embodiment >>
<1-1. Outline of camera misalignment detection system>
FIG. 1 is a block diagram showing a configuration of the camera deviation detection system SYS1 of the first embodiment. The camera deviation detection system SYS1 of the present embodiment is a system that detects the optical axis deviation of the in-vehicle camera. The camera shift detection system SYS1 includes an image processing device 1, an image pickup unit 2, a sensor unit 3, and a vehicle control device 4.

The image processing device 1 is a device that processes a camera image captured by an in-vehicle camera. The image processing device 1 is provided for each vehicle equipped with an in-vehicle camera. In the present embodiment, the image processing device 1 acquires a camera image from the imaging unit 2 and processes it. Further, the image processing device 1 acquires information from the sensor unit 3 and makes a determination regarding image processing based on the acquired information. In some cases, the sensor unit 3 may not be included in the camera deviation detection system SYS1.

In the present embodiment, the image processing device 1 processes the camera image captured by the imaging unit 2 to detect the camera deviation. Further, in the present embodiment, the image processing device 1 also processes the camera image captured by the imaging unit 2 to acquire a reference image for detecting the camera deviation. The image processing device that acquires the reference image may be another image processing device provided separately from the image processing device 1. In this case, another image processing device is also an example of the image processing device of the present invention.

In the present embodiment, the image processing device 1 outputs processing information to the display device 5 and the driving support device 6. The image processing device 1 may receive information from these devices 5 and 6 and perform camera deviation detection processing based on the received information. In this case, these devices 5 and 6 may also be included in the camera shift detection system SYS1.

The display device 5 is a device that displays information output from the image processing device 1 on a screen, and may be, for example, a liquid crystal display device. The display device 5 may be provided with an input unit such as a touch panel system so that information can be input from the outside.

The driving support device 6 is a device that supports driving by using an image captured by the imaging unit 2, and may include, for example, an automatic driving support device, an automatic parking support device, an automatic emergency braking device, and the like. The driving support device 6 may be configured to automatically start driving support when the engine is started, for example. Further, the driving support device 6 may be configured to start driving support when, for example, a button or the like is pressed by the driver.

The imaging unit 2 is provided for the purpose of monitoring the situation around the vehicle. The imaging unit 2 includes four cameras 21 to 24. The four cameras 21 to 24 are in-vehicle cameras. FIG. 2 is a diagram illustrating positions where the four cameras 21 to 24 are arranged on the vehicle 7.

The camera 21 is provided at the front end of the vehicle 7. Therefore, the camera 21 is also referred to as a front camera 21. The optical axis 21a of the front camera 21 is along the front-rear direction of the vehicle 7 in a plan view from above. The front camera 21 photographs the front direction of the vehicle 7. The camera 22 is provided at the rear end of the vehicle 7. Therefore, the camera 22 is also referred to as a back camera 22. The optical axis 22a of the back camera 22 is along the front-rear direction of the vehicle 7 in a plan view from above. The back camera 22 photographs the rear direction of the vehicle 7. The front camera 21 and the back camera 22 are preferably mounted at the center of the left and right sides of the vehicle 7, but may be slightly deviated from the center of the left and right sides in the left-right direction.

The camera 23 is provided on the left side door mirror 71 of the vehicle 7. Therefore, the camera 23 is also referred to as a left side camera 23. The optical axis 23a of the left side camera 23 is along the left-right direction of the vehicle 7 in a plan view from above. The left side camera 23 photographs the left direction of the vehicle 7. The camera 24 is provided on the right door mirror 72 of the vehicle 7. Therefore, the camera 24 is also referred to as a right side camera 24. The optical axis 24a of the right side camera 24 is along the left-right direction of the vehicle 7 in a plan view from above. The right side camera 24 captures the right direction of the vehicle 7.

The horizontal angle of view θ of each camera 21 to 24 is 180 degrees or more. Therefore, the cameras 21 to 24 can take a picture of the entire circumference of the vehicle 7 in the horizontal direction. Further, the body of the vehicle equipped with the cameras 21 to 24 is reflected in the images captured by the cameras 21 to 24.

In the present embodiment, the number of in-vehicle cameras is four, but this number may be changed as appropriate, and may be plural or singular. For example, when an in-vehicle camera is mounted for the purpose of assisting the vehicle 7 to park in the back, the in-vehicle camera is composed of three, a back camera 22, a left side camera 23, and a right side camera 24. May be good.

Returning to FIG. 1, the sensor unit 3 has a plurality of sensors that detect information about the vehicle 7 on which the cameras 21 to 24 are mounted. The information about the vehicle 7 may include information about the vehicle itself and information around the vehicle. In the present embodiment, the sensor unit 3 is, for example, a vehicle speed sensor that detects the speed of the vehicle, a steering angle sensor that detects the rotation angle of the steering wheel, and a shift sensor that detects the operating position of the shift lever of the transmission of the vehicle. , An illuminance sensor that detects the illuminance around the vehicle, and the like are included. In the present embodiment, the information detected by the sensor unit 3 is directly input to the image processing device 1, but this is only an example. For example, the information detected by the sensor unit 3 may be input to the image processing device 1 via the vehicle control device 4.

The vehicle control device 4 controls the overall operation of the vehicle. The vehicle control device 4 includes, for example, an engine ECU that controls an engine, a steering ECU that controls steering, a brake ECU that controls brakes, a shift ECU that controls shifts, an ECU for power supply control, a light ECU that controls lights, and an electric mirror. Includes a mirror ECU to control. The ECU is an abbreviation for "Electronic Control Unit". In the present embodiment, the vehicle control device 4 exchanges information with the image processing device 1.

FIG. 3 is a flowchart showing an example of a camera deviation detection flow by the camera deviation detection system SYS1. As shown in FIG. 3, four cameras 21 to 24 are attached to each vehicle 7 in a factory such as an automobile manufacturer (step S1). Each camera 21 to 24 is fixed at a predetermined mounting position with the optical axes 21a to 24a aligned in a predetermined direction.

Further, in the factory, the reference image is acquired (step S2). The reference image is an image used as a reference when detecting whether or not the optical axis of the cameras 21 to 24 is deviated after the vehicle 7 equipped with the cameras 21 to 24 is put on the market. In the present embodiment, the reference image is an edge image of the body of the vehicle 7 obtained by processing the camera images captured by the in-vehicle cameras 21 to 24 normally attached to the vehicle 7. The reference image is acquired for each camera 21 to 24 by using the image processing device 1. The details of the reference image acquisition method will be described later.

In the present embodiment, after the vehicles equipped with the cameras 21 to 24 are put on the market, the image processing device 1 detects the camera deviation. The detection of camera deviation is performed for each camera 21 to 24. The image processing device 1 determines whether or not it is the timing to detect the camera deviation (step S3). If No in step S3, the image processing device 1 repeats the determination of whether or not the camera deviation is detected. In the present embodiment, the image processing device 1 automatically determines the detection timing of the camera deviation. The details of the automatic determination will be described later.

The image processing device 1 may recognize the camera deviation detection timing by, for example, a camera deviation detection instruction input by the driver from the display device 5 or the like. Further, the image processing device 1 may be configured to acquire a camera image when instructed by the driving support device 6.

When the camera deviation detection timing comes (Yes in step S3), the image processing device 1 acquires the camera image and processes it to acquire the inspection image (step S4). The image processing device 1 compares the acquired inspection image with the reference image acquired in advance at the factory (step S5). As a result of comparison, the image processing device 1 determines whether or not the inspection image and the reference image match (step S6). The details of the camera deviation detection method including steps S3 to S6 will be described later.

When it is determined that the inspection image and the reference image match (Yes in step S6), the image processing device 1 determines that the optical axis shift of the camera has not occurred (step S7). In this case, the image processing device 1 may notify, for example, the driving support device 6 or the like that the camera is normal. The image processing device 1 returns to step S3 and repeats the detection of camera misalignment by detecting the next detection timing.

When it is determined that the inspection image and the reference image do not match (No in step S6), the image processing device 1 determines that the optical axis of the camera is deviated (step S8). In the present embodiment, the image processing device 1 notifies by the display device 5 that a camera shift has occurred. Further, the image processing device 1 notifies the driving support device 6 of the occurrence of camera deviation, and stops the driving support by the driving support device 6. In the present embodiment, there are four cameras 21 to 24, but if any one of the four cameras 21 to 24 has an optical axis shift, the notification process and the stop process are performed. Is preferable.

When the camera shifts, the vehicle user requests, for example, the dealer to adjust the mounting of the camera. In response to this, the dealer adjusts the mounting of the camera (step S9). Further, the dealer performs the reacquisition process of the reference image described above (step S10). After that, returning to step S3, when the image processing apparatus 1 detects the detection timing of the camera deviation, the image processing device 1 detects the camera deviation based on the re-acquired reference image.

In the present embodiment, the camera mounting adjustment is performed when the camera deviation is detected, but this is merely an example. When the camera deviation is detected, the image processing device 1 may be configured to correct the optical axis deviation of the camera and use the camera image. For example, the notification process of the occurrence of the deviation and the stop processing of the driving support on the premise of adjusting the mounting of the camera may be performed only when the deviation state of the camera is unknown. The state in which the camera is displaced is unknown, for example, a state in which it is unknown how much the camera is displaced or angled. When the misaligned state of the camera is clear and can be corrected and used, the image processing device 1 may be configured not to perform the misalignment occurrence notification process or the driving support stop process.

<1-2. Details of image processing equipment>
Returning to FIG. 1, the image processing device 1 includes a microcomputer 11 and a storage unit 12. The microcomputer 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 12 is a non-volatile memory. The microcomputer 11 processes and transmits / receives information based on the program stored in the storage unit 12. The microcomputer 11 is connected to the image pickup unit 2, the sensor unit 3, the vehicle control device 4, the display device 5, and the driving support device 6 by wire or wirelessly.

The microcomputer 11 includes a first inspection image acquisition unit 111, a comparison processing unit 112, a second inspection image acquisition unit 113, and a deviation detection unit 114. The functions of the respective units 111 to 114 included in the microcomputer 11 are realized by the CPU performing arithmetic processing according to the program.

The first inspection image acquisition unit 111 acquires an inspection image of the body of the vehicle 7 on which the in-vehicle cameras 21 to 24 are mounted, based on the camera images captured by the in-vehicle cameras 21 to 24. In the present embodiment, the number of in-vehicle cameras 21 to 24 is four, and the first inspection image acquisition unit 111 acquires an inspection image for each camera image captured by each camera 21 to 24. The first inspection image acquisition unit 111 is an example of the inspection image acquisition unit of the present invention.

The comparison processing unit 112 performs comparison processing between the inspection image acquired by the first inspection image acquisition unit 111 and the design data of the body shape of the vehicle 7. When the comparison processing unit 112 determines that the inspection image is a normal image as a result of comparison, the comparison processing unit 112 stores the inspection image as a reference image in the storage unit 12. When the comparison processing unit 112 determines that the inspection image is not a normal image as a result of the comparison, the comparison processing unit 112 notifies that the acquisition of the reference image has failed. As the notifying means, for example, a display device 5 may be used.

The first inspection image acquisition unit 111 and the comparison processing unit 112 are functional units that exert a function of acquiring a reference image. When the reference image is acquired by an image processing device different from the image processing device 1, the image processing device 1 does not have to include the first inspection image acquisition unit 111 and the comparison processing unit 112.

Similar to the first inspection image acquisition unit 111, the second inspection image acquisition unit 113 is an inspection image of the body of the vehicle 7 on which the in-vehicle cameras 21 to 24 are mounted, based on the camera images captured by the in-vehicle cameras 21 to 24. To get. In the present embodiment, the number of in-vehicle cameras 21 to 24 is four, and the second inspection image acquisition unit 113 acquires an inspection image for each camera image captured by each camera 21 to 24. In the present embodiment, the second inspection image acquisition unit 113 acquires the inspection image by using a process slightly different from that of the first inspection image acquisition unit 111. The second inspection image acquisition unit 113 is an example of the inspection image acquisition unit of the present invention.

The deviation detection unit 114 detects the optical axis deviation of the in-vehicle cameras 21 to 24 by comparing the reference image and the inspection image. The reference image is stored in the storage unit 12. In the present embodiment, the inspection image to be compared with the reference image is the inspection image acquired by the second inspection image acquisition 113. Further, in the present embodiment, the number of in-vehicle cameras 21 to 24 is four, and the deviation detection unit 114 detects the camera deviation for each of the cameras 21 to 24.

The second inspection image acquisition unit 113 and the deviation detection unit 114 are functional units that exert a function of detecting camera deviation in the market.

(1-2-1. Reference image acquisition method)
The reference image stored in the storage unit 12 is acquired at the factory as described above. In the present embodiment, the image processing device 1 performs the acquisition process of the reference image. The standard image acquisition method implemented in the factory will be described. FIG. 4 is a flowchart showing an example of the acquisition flow of the reference image. As described above, the reference image is acquired for each of the four cameras 21 to 24, but the reference image acquisition method is the same for all the cameras 21 to 24. For this purpose, here, the case where the reference image for the front camera 21 is acquired will be described as a typical example, and the description for the acquisition of the reference image for the other cameras will be omitted.

As shown in FIG. 4, first, the first inspection image acquisition unit 111 acquires a camera image from the front camera 21 normally attached to the vehicle 7 (step S11). When acquiring the camera image, it is preferable to appropriately set conditions such as lighting conditions and shading of the road surface so as not to cause erroneous detection of the edge of the body of the vehicle 7.

FIG. 5 is a schematic view showing an example of a camera image captured by the front camera 21. The front camera 21 has an angle of view θ of 180 degrees or more, and as shown in FIG. 5, the body 70 of the vehicle 7 is reflected in the camera image. In addition, in FIG. 5, the hatched portion corresponds to the body 70. The boundary portion of the body 70 with the road surface 8 is an arcuate curve 70a.

Returning to FIG. 4, the first inspection image acquisition unit 111 performs preprocessing (blurring) on the acquired camera image (step S12). The pre-processing is a process for smoothing a camera image and is performed for the purpose of reducing noise. The pretreatment may be, for example, Gaussian filter processing or median filter processing. Further, the preprocessing may be performed by changing the processing conditions for each part of the camera image. For example, when the center side of the camera image is sharp and the edge side is blurred, the processing conditions are set for each part of the camera image so that the blurring on the center side of the image is emphasized from the edge side. May be changed.

The first inspection image acquisition unit 111 performs edge detection on the preprocessed camera image (step S13). An edge is a portion where the pixel value of a pixel in an image changes rapidly, and is a line representing the outline of an object in the image. The first inspection image acquisition unit 111 detects an edge in the camera image by differentiating the pixel values of the pixels in the camera image. When the pixel values of the pixels in the camera image are differentiated, the differential value of the portion where the pixel values of the pixels in the camera image are rapidly changing becomes large. The first inspection image acquisition unit 111 detects a portion having a large differential value as an edge based on a predetermined reference. As the edge detection method, for example, the Canny method or the like may be used.

The first inspection image acquisition unit 111 performs mask processing on the edge image obtained by performing edge detection (step S14). In the present embodiment, the first inspection image acquisition unit 111 masks the image excluding the search area for the edge of the vehicle 7 which is preset according to the body shape of each vehicle type. Therefore, unnecessary edges such as landscapes can be appropriately removed, and the possibility of erroneously detecting the edges of the body 70 can be reduced. This makes it possible to quickly acquire the reference image.

FIG. 6 is an image diagram of mask data 9 prepared for each vehicle type. The mask data 9 has the same number of pixels as the camera image. As shown in FIG. 6, the mask data 9 has a mask region 9a and a search region 9b. In FIG. 6, the search region 9b is a bow-shaped portion (hatched portion).

In the present embodiment, the portion of the edge image that overlaps with the mask region 9a is blackened by mask processing. Specifically, each pixel value of the portion overlapping the mask area 9a is set to 0. Since the portion of the edge image that overlaps with the search area 9b is not masked, the pixel value is not changed before and after the masking process.

The range in which the edge of the body 70 of the vehicle 7 is reflected in the camera image can be specified in advance by the design value for each vehicle type. The edge of the body 70 of the vehicle 7 obtained from the camera image may be slightly displaced from the edge assumed by the design value due to a camera mounting error or the like. For this reason, the search region 9b is defined as a region having a width surrounding the edge assumed by the design value. In the present embodiment, the mask processing using the mask data 9 applies the mask except for the region where the edge of the body 70 of the vehicle 7 may be reflected.

Returning to FIG. 4, the first inspection image acquisition unit 111 performs a process of removing unnecessary edges from the edge image obtained after the mask process (step S15). In the present embodiment, the first inspection image acquisition unit 111 removes an edge image obtained after the mask processing that is determined to be different from the edge assumed from the design value of the body 70 of the vehicle 7. Do. Therefore, unnecessary edges can be appropriately removed, and the possibility of erroneously detecting the edges of the body 70 can be reduced.

FIG. 7 is a schematic view illustrating an edge image after mask processing. As shown in FIG. 7, even after the mask processing, unnecessary edges 10 other than the edges of the body 70 are present in the edge image. In the present embodiment, the first inspection image acquisition unit 111 detects the angle and the length of each of the divided edges in the edge image. The first inspection image acquisition unit 111 removes, for example, an edge having a large gap from the angle and length of the edge assumed from the design value as an unnecessary edge.

By the above preprocessing, edge detection, mask processing, and unnecessary edge removal, an inspection image that is adopted as a reference image and inspected for any problem is acquired.

Returning to FIG. 4, when the comparison processing unit 112 acquires the inspection image, the matching processing is performed (step S16). In the present embodiment, the comparison processing unit 112 performs template matching processing using the edge design line determined by the design value of the shape for each vehicle type and the inspection image. At this time, the design line is appropriately displaced or rotated in consideration of the camera mounting error range and the like. In other cases, the design line may be expanded or contracted.

Next, the template matching determines whether or not the inspection image is a normal image (step S17). If it is determined that the edge of the inspection image and the design line match due to movement within the assumed range, the inspection image is determined to be a normal image (Yes in step S17), and comparison processing is performed. The unit 112 stores the acquired inspection image as a reference image in the storage unit 12 (step S18).

FIG. 8 is a schematic view showing an example of a reference image. As shown in FIG. 8, in the reference image, there is a line L indicating the edge of the body 70 of the vehicle 7. In the template matching process, an unnecessary edge different from the edge of the body 70 of the vehicle 7 may be detected. In such a case, for example, an unnecessary edge may be removed after the template matching process, and an image from which the unnecessary edge has been removed may be stored in the storage unit 12 as a reference image.

The reference image stored in the storage unit 12 may have the same number of pixels as the camera image, but may have a smaller number of pixels than the camera image. FIG. 9 is a schematic diagram for explaining an example of a reference image stored in the storage unit 12.

In the example shown in FIG. 9, after the template matching process, only a part range r of the image R determined as the reference image is extracted and stored in the storage unit 12. A part of the range r is a range in which the edge L of the body 70 reflected in the reference image is left without being scraped and other unnecessary parts are scraped. At this time, in order to specify the extracted range, it is necessary to store some coordinate points X and Y together with the image. In the example shown in FIG. 9, the coordinates X and Y of one corner of the square extraction frame are stored. By reducing the number of pixels of the reference image stored in the storage unit 12 in this way, the capacity of the reference image occupied in the storage unit 12 can be reduced. When storing the reference image in the storage unit 12, image compression processing may be used. Examples of the compression process include run-length compression and the like.

Returning to FIG. 4, if the inspection image does not match the design line, it is determined that the inspection image is not a normal image (No in step S17), and the comparison processing unit 112 fails to acquire the reference image. Is notified (step S18). Notification of the error is performed, for example, by displaying an image. The error notification causes the reference image to be acquired again. When reacquiring the reference image, the environmental conditions such as the vehicle position and the lighting position are changed, and the parameters in the image processing are changed.

If the cause of the failure to acquire the reference image is clear, the failure factor may be notified by a screen display or the like. As a result, it is possible to take measures according to the cause of failure, and it is possible to quickly acquire a reference image. For example, in the template matching process, if a tendency is obtained that only the central portion and the edge portion of the image match, it can be inferred that the vehicle position and the lighting conditions are the cause of the failure. For this reason, the configuration may be such that the processing status in such template matching is notified.

In the reference image acquisition method shown above, the mask processing is performed after the edge detection processing, but this is an example. For example, the mask processing may be performed before the edge detection processing. Further, the pre-processing and the unnecessary edge removal processing may be omitted in some cases.

(1-2-2. Camera misalignment detection method)
Next, a camera deviation detection method using the image processing device 1 of the present embodiment will be described. FIG. 10 is a flowchart showing an example of a camera deviation detection flow of the first embodiment. As described above, the camera deviation is detected for each of the four cameras 21 to 24, but the camera deviation detection method is the same for all the cameras 21 to 24. For this purpose, here, the case where the detection of the camera deviation of the front camera 21 is acquired will be described as a typical example, and the description of the detection of the camera deviation with respect to other cameras will be omitted.

As shown in FIG. 10, first, the second inspection image acquisition unit 113 determines whether or not it is the acquisition timing of the camera image (camera image for inspection) for acquiring the inspection image (step S21). The second inspection image acquisition unit 113 repeats the confirmation until it is time to acquire the camera image for inspection. The timing of acquiring the camera image for inspection may be while the vehicle is running or stopped. The timing of acquiring the camera image for inspection is preferably the timing at which the gradation difference between the body 70 of the vehicle 7 and the road surface becomes large. Further, the timing of acquiring the camera image for inspection is preferably a timing in which there is no erroneous edge detection factor such as a white line or a shadow of the vehicle at the boundary between the body 70 of the vehicle 7 and the road surface.

The second inspection image acquisition unit 113 may determine the timing of acquiring the camera image based on the information about the vehicle 7 acquired from the outside. Examples of the information regarding the vehicle 7 include vehicle speed information, mileage information, steering angle information, illuminance information around the vehicle, and the like. This information can be obtained from, for example, the sensor unit 3 or the like. By considering the information about the vehicle 7 with respect to the timing for acquiring the camera image for inspection, for example, the situation where the shadow is always present at the same position can be avoided, or the body 70 of the vehicle 7 is clearly reflected in the image. You can avoid difficult situations. Therefore, the possibility of erroneous detection of the edge of the body 70 can be reduced.

Further, the second inspection image acquisition unit 113 may be configured to acquire a camera image when the ACC of the vehicle 7 is turned on and the engine stop is detected. The on / off state of the ACC and the operating state of the engine can be obtained from, for example, the vehicle control device 5. By acquiring the camera image for inspection while the engine is stopped, it is possible to acquire an image in which the influence of engine vibration is removed, so that the edge of the body 70 can be accurately detected. ..

Further, when it is determined that the illuminance around the vehicle 7 is lower than a predetermined threshold value (an example of the second threshold value of the present invention), the second inspection image acquisition unit 113 instructs the camera to turn on the light of the vehicle 7. It may be configured to acquire an image. According to this configuration, even when the periphery of the vehicle 7 is dark, for example, in the evening or at night, the body 70 of the vehicle 7 can be clearly reflected in the camera image for inspection. Therefore, it is possible to reduce the possibility that the edge of the body 70 cannot be detected or the edge is erroneously detected when the inspection image is acquired. The illuminance around the vehicle 7 can be obtained from, for example, an illuminance sensor included in the sensor unit 3. As another example, before acquiring the camera image for inspection, the camera image may be acquired and the illuminance around the vehicle 7 may be determined from the camera image.

The light to be turned on may be appropriately selected from a plurality of lights included in the vehicle 7, and may be one or a plurality of lights. The light to be turned on may include, for example, at least one of a headlight, a small light, a backlight, a number lamp, and a welcome lamp. Regarding the extinguishing of the light after acquiring the target camera image, for example, the configuration may follow the auto light function when the engine is operating, and the configuration may be such that the light is immediately extinguished after acquiring the target camera image when the engine is stopped.

Further, there is a vehicle 7 in which the positions of the cameras 21 to 24 are moved at a predetermined timing such as when the engine of the vehicle 7 is stopped. For example, in a configuration in which the electric side mirrors are automatically folded, the positions of the cameras 23 and 24 move. Further, there is a vehicle 7 in which the cameras 21 to 24 are automatically stored in the housing for the purpose of protecting the cameras 21 to 24. With respect to these vehicles 7, the second inspection image acquisition unit 113 determines whether or not the positions of the cameras 21 to 24 are present at positions suitable for inspection before acquiring the camera image for inspection. May be.

When the cameras 21 to 24 are not present at positions suitable for inspection, the second inspection image acquisition unit 113 instructs the vehicle control unit 4 to move the position of the camera to a position suitable for inspection. It may be configured to acquire a camera image for inspection after the instruction. Further, as another example, when the cameras 21 to 24 do not exist at positions suitable for inspection, the second inspection image acquisition unit 113 may be configured not to acquire the camera image for inspection. In this case, the driving support by the driving support device 6 may be prohibited.

In addition, the second inspection image acquisition unit 113 may be configured to detect camera misalignment when it detects impact information on the vehicle 7. When the impact information is detected, the driving support by the driving support device 6 may be prohibited until it is confirmed that the camera is not displaced.

When the second inspection image acquisition unit 113 determines that it is time to acquire the camera image for inspection (Yes in step S21), the second inspection image acquisition unit 113 acquires the camera image for inspection (step S22). After that, preprocessing (step S23), edge detection (step S24), mask processing (step S25), and unnecessary edge removal (step S26) are performed. Since these processes are the same as the processes described in the reference image acquisition method (see FIG. 4), detailed description thereof will be omitted.

The second inspection image acquisition unit 113 performs image masking processing except for the search area 9b (see FIG. 6) of the edge of the body 70, which is preset according to the body shape for each vehicle type. For this reason, the second inspection image acquisition unit 113 can appropriately remove unnecessary edges such as landscapes, and can reduce the possibility of erroneously detecting the edges of the body 70. In addition, the second inspection image acquisition unit 113 can quickly acquire the inspection image. In addition, the second inspection image acquisition unit 113 performs a process of removing an edge image obtained after the mask process, which is determined to be different from the edge assumed from the design value of the body 70 of the vehicle 7. Therefore, the second inspection image acquisition unit 113 can reduce the possibility of erroneously detecting the edge of the body 70. The mask processing may be performed before the edge detection, for example.

As shown in FIG. 10, after the masking process, the second inspection image acquisition unit 113 performs a remasking process of masking the narrow search area narrower than the search area 9a (step S27). In the present embodiment, unnecessary edge removal processing is performed between the mask processing and the remask processing. However, the unnecessary edge removal process after the mask process may be omitted in some cases. Further, for example, the unnecessary edge removal process may be performed after the remask process.

FIG. 11 is an image diagram for explaining the remasking process. As shown in FIG. 11, in the present embodiment, the narrow search area 9c is surrounded by the search area 9b as a whole. The second inspection image acquisition unit 113 uses the edge (line L) of the body 70 in the reference image as a reference, and sets a region expanded by n pixels in the periphery (up / down / left / right direction in FIG. 11) as a narrow search region 9c. The n pixels are preferably several pixels, and are set within a range in which the narrow search region 9c does not overlap with or protrude from the search region 9b.

In the remasking process, the second inspection image acquisition unit 113 masks the edge image except for the narrow search region 9c. Specifically, the pixel value of the masked portion is set to zero. In the narrow search region 9c where masking is not applied, the pixel value is not changed before and after the remasking process. In the present embodiment, the second inspection image acquisition unit 113 acquires the inspection image by remasking the edge image.

In the present embodiment, in order to apply a mask to a wider range than the initial masking process, unnecessary edges existing in the vicinity of the edges of the body 70 can be removed more appropriately. For example, according to the present embodiment, unnecessary edges such as white lines existing near the edge of the body 70 and shadows existing near the edge of the body 70 or reflected on the body 70 are appropriately removed. Becomes possible. Therefore, the possibility of erroneous edge detection can be reduced. Further, since the cameras 21 to 24 are firmly fixed to the vehicle 7, the camera deviation does not easily occur. Therefore, the edge of the body 70 of the vehicle 7 in the inspection image is likely to occur in the vicinity of the edge line of the reference image. For this reason, if the remasking process is performed as in the present embodiment, the camera misalignment confirmation process can be performed accurately and quickly in most cases.

Returning to FIG. 10, when the inspection image is acquired, the deviation detection unit 114 performs pattern matching processing between the inspection image and the reference image (step S28). The reference image is read from the storage unit 12. Here, the pattern matching process will be described in detail with reference to FIGS. 12 and 13. FIG. 12 is a flowchart showing an example of a processing flow of the pattern matching process. FIG. 13 is a schematic diagram showing a comparative image of the inspection image and the reference image.

As shown in FIG. 12, in the deviation detection unit 114, first, the number of pixels detected as edges in the inspection image (hereinafter, may be referred to as edge pixels) is a predetermined threshold value (first threshold value of the present invention). (Example)) It is confirmed whether or not it is the above (step S280). In the present embodiment, the inspection image is binarized. The pixel value of the portion without an edge is 0, and the pixel value of the portion with an edge is 1. Therefore, in the inspection image, the portion where the pixel value is 1 becomes the edge pixel. It is not essential that the inspection image is binarized. If it is not binarized, for example, it may be determined whether or not the pixel value is equal to or greater than a predetermined value, and a pixel having a pixel value equal to or greater than a predetermined value may be determined to be an edge pixel.

If the number of edge pixels is too small, reliable detection cannot be performed eventually. Therefore, in the present embodiment, the deviation detecting unit 114 is when the number of pixels (edge pixels) detected as edges in the inspection image is less than a predetermined threshold value (first threshold value) (No in step S280). , Request reacquisition of the inspection image (step S281). In the present embodiment, the deviation detection unit 114 requests the second inspection image acquisition unit 113 to reacquire the inspection image. As a result, the camera deviation can be determined using a highly reliable inspection image, so that the detection accuracy of the camera deviation can be improved.

When the deviation detection unit 114 requests the reacquisition of the inspection image, the pattern matching process is temporarily terminated. The second inspection image acquisition unit 113, which is requested to reacquire the inspection image, may acquire the inspection image under the same conditions as the above conditions, but it is preferable to reacquire the inspection image under different conditions. .. This makes it possible to reduce the possibility that the request for reacquiring the inspection image is repeated.

For example, when the inspection image is reacquired, the second inspection image acquisition unit 113 is configured to determine the acquisition timing of the camera image based on the information about the vehicle 7 acquired from the outside and acquire the inspection image. Is preferable. If the timing for acquiring the camera image for inspection is determined based on the information about the vehicle 7, it is possible to acquire the camera image for inspection under a different environment from that at the time of acquiring the previous inspection image. That is, the re-acquired inspection image can be expected to be an edge image different from the previous inspection image. Therefore, it is possible to reduce the possibility that the request for reacquiring the inspection image is repeated.

Examples of the information regarding the vehicle 7 include information from the steering angle sensor included in the sensor unit 3. The second inspection image acquisition unit 113 determines that it is the timing to acquire the camera image for inspection when it is determined that the steering wheel has been operated by the steering angle sensor. With this configuration, the inspection image can be acquired in a state where the direction of the vehicle 7 is changed as compared with the case where the inspection image is acquired first. When the direction of the vehicle 7 is changed, for example, the shadow generation situation is likely to change, and the possibility that the inspection image reacquisition request is repeated can be reduced.

Further, the information about the vehicle 7 may be a mileage or the like. If the position of the vehicle 7 is separated from the timing when the camera image for inspection is acquired earlier by a certain amount or more, for example, the white line condition on the road surface is likely to change, and the acquisition result of the inspection image can be expected to change. Is. Further, the information about the vehicle 7 may be the illuminance around the vehicle. This is because when the illuminance changes, for example, there is a high possibility that the vehicle will be clearly photographed or the shadow generation situation will change, and it can be expected that the acquisition result of the inspection image will change.

Further, for example, when reacquiring the inspection image, the second inspection image acquisition unit 113 at least the narrow search area 9c of the search area 9b and the narrow search area 9c as compared with the time of acquiring the previous inspection image. May be configured to acquire an inspection image by enlarging. When the reacquisition request is repeated, for example, the second inspection image acquisition unit 113 may be configured to gradually expand the range of the narrow search region 9c for each inspection image reacquisition process. If the reacquisition request is repeated even if the narrow search area 9c is expanded, the range of the search area 9b may be expanded. With this configuration, for example, even if the camera 21 has a large optical axis deviation, it is possible to perform pattern matching processing and check how much the optical axis deviation has occurred. ..

Further, the deviation detection unit 114 determines that the in-vehicle camera 21 is displaced by the optical axis when the number of edge pixels in the inspection image is less than a predetermined threshold value even if the inspection image is reacquired a predetermined number of times. It is preferable to do so. With such a configuration, it is possible to prevent the time required for detecting the camera deviation from becoming longer than necessary. If it is determined that the camera 21 has an optical axis shift, the display device 5 may indicate that fact, and the driving support device 6 may prohibit the driving support.

Further, when the re-acquisition is repeated a predetermined number of times, instead of determining that the optical axis is displaced, it may be determined that the determination is impossible due to the poor acquisition environment of the inspection image. In this case, for example, the camera deviation may be detected after a predetermined time or more has elapsed. Further, when it is determined that the determination cannot be made, the display device 5 may display the fact and the driving support by the driving support device 6 may be prohibited.

Further, in the above, the deviation detection unit 114 is configured to request reacquisition of the inspection image when the number of edge pixels in the inspection image is small, but this is an example. The deviation detection unit 114 may be configured to simply determine that the deviation cannot be determined when the number of edge pixels is small. If it is determined that the deviation cannot be determined, the display device 5 may display the fact and the driving support by the driving support device 6 may be prohibited.

When the deviation detection unit 114 determines that the number of edge pixels in the inspection image is equal to or greater than a predetermined threshold value (Yes in step S280), the deviation detection unit 114 selects the reference image from the plurality of edge pixels in the inspection image. The edge pixels to be compared are extracted one by one in order (step S282). Explaining with reference to the example of FIG. 13, the deviation detection unit 114 first extracts the edge pixel E1 from the eight edge pixels E1 to E8 in the inspection image. In FIG. 13, the plurality of small square areas refer to pixels. Further, the pixel in which the character is not described is not an edge pixel.

Next, the shift detection unit 114 determines whether or not the edge pixel exists at the same position as the edge pixel extracted from the inspection image in the reference image (step S283). Explaining with reference to the example of FIG. 13, in the reference image, the edge pixel e1 exists at the same position as the edge pixel E1. Therefore, the deviation detection unit 114 determines that the edge pixel E1 of the inspection image has the edge pixel at the same position of the reference image. In the present embodiment, the reference image is also a binarized edge image, the pixel value of the edge pixel is 1, and the other pixel values are 0. However, this is an example and may not be binarized like the reference image.

When the edge pixel exists at the same position as the extracted edge pixel (Yes in step S283), the deviation detection unit 114 counts up the exact match score (step S284). The initial value of the exact match score is 0. Explaining with reference to the example of FIG. 13, for the edge pixel E1, the score of the exact match is counted up, and the score becomes 0 to 1.

When the deviation detection unit 114 counts up the exact match score, it confirms whether or not all the edge pixels in the inspection image have been determined by comparison with the reference image (step S285). When the determination is made for all the edge pixels (Yes in step S285), the shift detection unit 114 ends the pattern matching process. If the determination is not made for all the edge pixels (No in step 285), the process returns to step S282 and the process is repeated.

In the example of FIG. 13, after the determination of the edge pixel E1 is performed, the determination of the edge pixel E2 is performed. Since the edge pixel e2 exists at the same position as the edge pixel E2 in the reference image, the same processing as that of the edge pixel E1 is repeated, and the exact match score is 2. Regarding the edge pixel E3 performed next, the edge pixel does not exist at the same position in the reference image. Therefore, the shift detection unit 114 determines that the edge pixel does not exist at the same position of the reference image (No in step S283), and expands the search range to one peripheral pixel (step S286). As shown in the reference image of FIG. 13, the shift detection unit 114 extends the search range to eight pixels around the pixel at the same position as the edge pixel E3.

The deviation detection unit 114 confirms whether or not edge pixels are present in the search range of the enlarged reference image (step S287). When the edge pixel exists in the search range (Yes in step S287), the deviation detection unit 114 counts up the score of the deviation of one pixel (step S288). The initial value of the score of 1 pixel deviation is also 0. In addition, there may be a plurality of edge pixels instead of one in the expanded search range, but in any case, the number of count-ups is one. After the count-up, the process proceeds to step S285 described above, and it is confirmed whether or not the determination of all the edge pixels is completed. On the other hand, when the edge pixel does not exist in the search range (No in step S287), the deviation detection unit 114 proceeds to step S285 described above and confirms whether or not the determination of all the edge pixels is completed.

In the example of FIG. 13, for the edge pixel E3 in the inspection image, since it is determined in step S287 that the edge pixel does not exist within the search range of the reference image, the score of the exact match and the score of one pixel deviation are obtained. Neither is counted up. For the edge pixels E4 and E5 in the inspection image, the exact match score is counted up. Regarding the edge pixel E6 in the inspection image, the exact match score is not counted up because the edge pixel does not exist at the same position in the reference image, but the edge pixels e5 and e6 exist in the search range expanded to one peripheral pixel. Therefore, the score of one pixel shift is counted up. For the edge pixels E7 and E8 in the inspection image, the exact match score is counted up. When the determination of the edge pixel E8 is completed, it is determined in step S285 that the determination of all the edge pixels is completed, and the pattern matching process is completed. In the example shown in FIG. 13, at the end, the score of exact match is 6, and the score of 1 pixel deviation is 1.

In the above example, the search range was expanded to only one pixel around it, but this is only an example. For example, if no edge pixel is found even if the search range is expanded to 1 peripheral pixel, the search range is expanded to 2 peripheral pixels, and the search range is gradually expanded to n peripheral pixels. Good. By increasing the n value, it becomes possible to grasp the details of the deviation between the inspection image and the reference image.

Returning to FIG. 10, when the pattern matching process is completed, the shift detection unit 114 determines the camera shift based on the result of the pattern matching process (step S29). In the present embodiment, the deviation detection unit 114 calculates the matching rate of the inspection image with respect to the reference image by the following formula (1).
Match rate (%) = (exact match score + peripheral n pixel shift score) / (total number of edge pixels in the inspection image) x 100 (1)
In this embodiment, n in the formula (1) is 1. The number of n may be changed as appropriate. Further, in the example of FIG. 13, the match rate (%) = 87.5 (= (6 + 1) / 8 × 100).

In the present embodiment, the deviation detection unit 114 determines that no camera deviation has occurred when the matching rate is equal to or greater than a predetermined threshold value. The deviation detection unit 114 determines that the camera deviation has occurred when the matching rate is smaller than a predetermined threshold value. In the present embodiment, the deviation detection unit 114 can quickly and accurately determine the camera deviation because the inspection image can be obtained while reducing the false detection of the edge by the mask processing and the remask processing.

The determination standard when the deviation detection unit 114 determines the deviation may be constant, but may be changed depending on the acquisition status of the inspection image. For example, for the inspection image acquired immediately after the engine is started, the predetermined threshold value for determining the magnitude of the above-mentioned matching rate is smaller than the inspection image acquired at other timings, or the range to be compared with the reference image. May be narrowed to a specific range.

FIG. 14 is a flowchart showing a modified example of the camera deviation detection flow of the first embodiment. In the modified example shown in FIG. 14, the inspection image acquisition step (steps S21 to S27) by the second inspection image acquisition unit 113 and the pattern matching process (step S28) by the deviation detection unit 114 are the same as those described above. Is. In the modified example shown in FIG. 14, the deviation detection unit 114 acquires comparison results with the reference image for a plurality of inspection images, and determines the optical axis deviation of the camera according to the comparison results a plurality of times. This point is different from the above-described configuration.

When the pattern matching process is completed, the shift detection unit 114 determines whether or not the processing result of the specified number of times has been obtained (step S28'). When the processing result of the specified number of times is obtained (Yes in step S28'), the deviation detection unit 114 determines the camera deviation based on the processing results of a plurality of times (step S29). For example, the average value of the results of a plurality of times may be used for determining the camera deviation. On the other hand, when the processing result of the specified number of times has not been obtained (No in step S28'), the deviation detection unit 114 instructs the second inspection image acquisition unit 113 to further perform the inspection image acquisition process. As a result, returning to step S21, the second inspection image acquisition unit 113 further acquires the inspection image.

In this modification, the shift detection unit 114 performs a process of storing the result of each pattern matching process in, for example, a RAM (not shown) included in the microcomputer 11. As a result, the deviation detection unit 114 can compare and process the processing results a plurality of times. In the present embodiment, since the camera deviation is determined using a plurality of inspection images, the camera deviation determination accuracy can be improved.

FIG. 15 is a schematic diagram for explaining a modified example of the deviation determination by the deviation detecting unit 114. In the above, the deviation detection unit 114 is configured to calculate the matching rate for the entire area of the inspection image, which is an example. The deviation detection unit 114 divides the inspection image into a plurality of regions, performs comparison processing with the reference image for each of the plurality of regions, and based on the obtained plurality of comparison processing results, shifts the optical axis of the camera. It may be configured to make a determination.

For example, as shown in FIG. 15, the deviation detection unit 114 divides the detection image into three regions R1 to R3 divided by broken lines, and compares each of the three regions R1 to R3 with the reference image to determine the matching rate. It may be calculated. In this example, three matching rates are obtained for one inspection image. The deviation detection unit 114 can comprehensively determine the camera deviation based on the coincidence rates of the three regions R1 to R3. Therefore, it becomes possible to detect the camera deviation more appropriately. The number of divided regions is not limited to three, and the number may be changed as appropriate. Further, the method of dividing the region in the detected image is not limited to the configuration in which the region is divided at the position in the horizontal direction of the image. For example, the plurality of regions may be designated by the start point coordinates and the number of vertical and horizontal pixels with respect to the start point coordinates, respectively. The number of vertical and horizontal pixels in each region may be the same or different.

The deviation detection unit 114 may determine that no camera deviation has occurred, for example, only when the matching rate is equal to or greater than the threshold value in all three regions R1 to R3. According to the determination method of this configuration, when the match rate is determined as a whole, even in a situation where it is determined that there is no camera deviation, if a low match rate is found in any of the divided areas, the camera Detect deviation. When the camera shift occurs, for example, the shift may occur unevenly in one of the three regions. Therefore, according to the determination method of this configuration, the possibility of overlooking the occurrence of camera misalignment can be reduced.

As another example, the deviation detection unit 114 may determine the camera deviation by changing the weighting for the determination of the camera deviation according to the degree of influence on the three regions R1 to R3 when the camera deviation occurs. .. For example, when a camera shift occurs, it may have a large effect on the matching rate of the central region R2 and a small effect on the matching rate of the end regions R1 and R3. In such a case, the criterion for the matching rate may be changed between the central region R2 and the end regions R1 and R3. As a result, the camera deviation can be determined accurately and quickly.

Further, when there is a region having a high matching rate even in one of the three regions R1 to R3, the configuration may be used while correcting the camera image without notifying the outside of the camera deviation.

Further, as shown in FIG. 14, in the case of the configuration in which the camera deviation is determined using the comparison results of a plurality of times, the comparison area between the inspection image and the reference image may be changed each time. For example, the comparison region may be sequentially changed by a method in which the leftmost region R1 is compared at the first time, the central region R2 is compared at the second time, and the rightmost region is compared at the third time. With this configuration, the load of comparison processing at each time can be reduced.

<< 2. Second Embodiment >>
<2-1. Camera misalignment detection system>
The camera deviation detection system of the second embodiment is the same as the configuration of the first embodiment. For this reason, the description of the camera misalignment detection system will be omitted.

<2-2. Details of image processing equipment>
The rough configuration of the image processing apparatus of the second embodiment is the same as that of the first embodiment. Further, the reference image acquisition method executed by the image processing apparatus of the second embodiment is also the same as that of the first embodiment. For this reason, the description thereof will be omitted. Further, the configuration overlapping with the first embodiment will be described with the same reference numerals. The image processing device 1 of the second embodiment has a portion different from that of the first embodiment in the camera deviation detection method. In the following, the explanation will focus on the different parts.

(2-2-1. Camera misalignment detection method)
The image processing device 1 of the second embodiment also includes a second inspection image acquisition unit 113 and a deviation detection unit 114. Then, as in the first embodiment, these exert a function of detecting camera deviation.

However, in the second embodiment, the second inspection image acquisition unit 113 acquires a plurality of pre-composite images by performing processing including the above-mentioned mask processing on each of the plurality of camera images captured at different timings. The difference is that the configuration is such that a plurality of uncombined images are combined to obtain an inspection image. With such a configuration, it is possible to improve the reliability of the camera deviation determination result as compared with the case where the camera deviation is determined by comparing the inspection image acquired from one camera image with the reference image. ..

FIG. 16 is a flowchart showing an example of a camera deviation detection flow of the second embodiment. In the second embodiment as well, as in the first embodiment, the second inspection image acquisition unit 113 determines the timing of acquiring the camera image for inspection (step S31), and acquires the camera image at the acquisition timing (step S31). Step S32). The second inspection image acquisition unit 113 performs preprocessing (step S33), edge detection (step S34), mask processing (step S35), unnecessary edge removal (step S36), and remasking of the acquired camera image. The process (step S37) is sequentially performed. Since the details of steps S31 to S37 are the same as those of the first embodiment, the description thereof will be omitted.

The second inspection image acquisition unit 113 performs a composition process using the pre-composition image acquired in step S37 (step S38). In the present embodiment, the second inspection image acquisition unit 113 alpha-blends the pre-combination image acquired in step S37 with the image stored in the compositing processing memory to update the image stored in the compositing processing memory. Perform the process of The memory for synthesis processing may be, for example, a RAM (not shown) included in the microcomputer 11.

When the image stored in the compositing processing memory is updated, the second inspection image acquisition unit 113 confirms whether or not the compositing process may be terminated (step S39). The end of the synthesis process may be determined, for example, by whether or not the number of times the synthesis process has been performed exceeds the specified number of times. When the compositing process is completed (Yes in step S39), the image stored in the compositing processing memory at this point is taken as the inspection image. On the other hand, when the compositing process is not completed (No in step S39), the second inspection image acquisition unit 113 returns to step S31 and performs the next pre-composite image acquisition process.

FIG. 17 is a schematic diagram for explaining the synthesis process in detail. The synthesis process of the present embodiment will be described in more detail with reference to FIG. As shown in FIG. 17, the second inspection image acquisition unit 113 acquires the camera image A at the time when the acquisition timing of the camera image for inspection comes, and performs the above-mentioned processing to acquire the pre-composite image A. In the present embodiment, the pre-combination image is a grayscale image with 256 gradations, and the pixel value of each pixel is represented by any numerical value from 0 to 255.

The second inspection image acquisition unit 113 reads out the initial image stored in the compositing processing memory. The second inspection image acquisition unit 113 acquires the updated image A by alpha-blending the pre-composite image A with the read initial image. The blend ratio of the alpha blend may be appropriately determined, and is, for example, 10%. That is, the blending process is performed at a ratio of 90% for the initial image and 10% for the pre-composite image A.

In the present embodiment, the initial image stored in the compositing processing memory is an image obtained by adjusting the reference image to a predetermined density. Specifically, the initial image is an image in which the pixel values of the edge pixels of the reference image are reduced in gradation. For example, it is assumed that the pixel value of the edge pixel of the reference image is 255. In this case, assuming that the gradation of the edge pixels is reduced to about 20% of the reference image, the pixel value of the edge pixels is set to 50.

When the second inspection image acquisition unit 113 acquires the updated image A by the blending process, the second inspection image acquisition unit 113 updates the initial image stored in the compositing processing memory to the updated image A. When the second inspection image acquisition unit 113 acquires the updated image A, the second inspection image acquisition unit 113 acquires the next pre-combination image if it is not at the stage of ending the composition process.

The end of the synthesis process may be determined by the number of times of the synthesis process as described above, but may be determined by another method. For example, the end of the compositing process may be determined by whether or not the pixel value of the edge pixel of the image obtained by the alpha blend exceeds a predetermined threshold value. It may end when the pixel values of all the edge pixels exceed a predetermined threshold value, but it may end when the proportion of edge pixels exceeding the predetermined threshold value exceeds a predetermined ratio. Further, as another example, the end determination of the compositing process may be performed based on both the number of compositing processes and the pixel value of the edge pixel. If the number of compositing processes is extremely small, the reliability of the edges in the compositing image may be low. Therefore, when the number of times of synthesis processing does not exceed a predetermined number of times, the composition processing may not be completed even if the pixel value of the edge pixel exceeds a predetermined threshold value.

In the example shown in FIG. 17, the second inspection image acquisition unit 113 updates the initial image stored in the compositing processing memory to the updated image A, and then when the acquisition timing of the inspection camera image comes. Get B. The acquisition timing of the camera image B may be, for example, a time when a predetermined time has elapsed with reference to the timing of acquiring the camera image for the previous inspection or the timing of updating the image of the compositing processing memory. Apart from this, for example, when there is a change in the vehicle information such as a change in the steering angle of the steering wheel, or when there is a change in the vehicle surrounding information such as a change in the illuminance around the vehicle, the camera image B may be acquired.

When the second inspection image acquisition unit 113 acquires the camera image B, the second inspection image acquisition unit 113 acquires the pre-composite image B in the same manner as in the case of the camera image A. The second inspection image acquisition unit 113 reads the updated image A stored in the compositing processing memory, alpha-blends the pre-compositing image B with the updated image A, and acquires the updated image B. The second inspection image acquisition unit 113 updates the updated image A stored in the compositing processing memory to the updated image B. The blend ratio of the alpha blend is the same as when the updated image A is obtained.

After that, the second inspection image acquisition unit 113 performs the update image C, the update image D, and so on in the same procedure as when the update image B is stored in the memory for the composition process until it is determined that the composition process is completed. The inspection image is acquired by sequentially repeating the process of updating the image stored in the compositing processing memory. For example, in the driving scene of the vehicle 7, there are many noise sources such as shadows, white lines, obstacles, etc., and it is difficult to extract the edge of the body 70 of the vehicle 7 only by the inspection image obtained by processing only one camera image. In some cases. In this respect, in this configuration, a plurality of camera images acquired at different timings are sequentially blended. Noise such as shadows does not continue to appear at the same location due to, for example, the movement of the vehicle 7, while the body 70 of the vehicle 7 continues to appear at the same location. Therefore, when the blending process is performed as in the present configuration, it is possible to acquire an inspection image in which the edge strength of the body 70 is strengthened and the strength of unnecessary edges such as shadows and white lines is weakened.

Further, in the present embodiment, an image obtained by lowering the gradation of the reference image is used as the base image (initial image) of the alpha blend. Since camera deviation usually does not easily occur, the edge position of the inspection image usually does not deviate from the edge position of the reference image. For this reason, by configuring as in the present embodiment, in many cases, it is possible to reduce the number of times of image composition processing, and it is possible to shorten the time required for camera deviation detection processing. is there.

Returning to FIG. 16, when an inspection image using the alpha blend is acquired, the shift detection unit 114 performs pattern matching processing (step S40). In the second embodiment, among the inspection images, pixels having a pixel value exceeding a predetermined threshold value are regarded as edge pixels. This point is different from the first embodiment, but other than that, the pattern matching process performed in the second embodiment is substantially the same as that in the first embodiment. In the second embodiment, the matching rate with the reference image is calculated using a plurality of pixels whose pixel values exceed a predetermined threshold value. Even for pixels that do not exceed a predetermined threshold value, if the pixel value is not 0 or the pixel value is not extremely small, pattern matching processing may be performed to calculate a provisional matching rate.

The deviation detection unit 114 determines the camera deviation based on the matching rate obtained by the pattern matching process (step S41). Since the camera deviation determination method is the same as that of the first embodiment, the description thereof will be omitted. In the present embodiment, since the reliability of the inspection image used for the pattern matching process can be improved, the reliability of the camera deviation determination can also be improved.

In the above, the initial image is stored in the compositing processing memory when the first pre-compositing image is acquired. This is just an example. FIG. 18 is a schematic diagram for explaining a modification of the synthesis process. As shown in FIG. 18, the first acquired pre-composite image A may be used as the initial image stored in the compositing processing memory. In the example shown in FIG. 18, the processing after the initial image is set is the same as the example shown in FIG.

If the reliability of the first image before compositing is low, if such an image is used as the initial image, it may take a long time to complete the compositing process, or the inspection image obtained by the compositing process may have an erroneous edge. It may be included in large quantities. For this purpose, the second inspection image acquisition unit 113 is configured to determine whether or not the pre-combination image is to be the initial image stored in the compositing processing memory based on the reliability of the pre-combination image. Is preferable.

The reliability of the pre-composite image can be determined, for example, by the number of pixels determined to be edges in the composite image. For example, when the number of pixels judged to be edges is extremely low, it is judged that the reliability is low. When the second inspection image acquisition unit 113 determines that the reliability of the first acquired pre-composite image is high, the second inspection image acquisition unit 113 sets the pre-combination image as the initial image stored in the compositing processing memory. On the other hand, when it is determined that the reliability of the first acquired pre-composite image is low, the second inspection image acquisition unit 113 repeats the acquisition of the pre-combination image until a highly reliable pre-combination image is obtained. .. The second inspection image acquisition unit 113 sets the pre-composite image as the initial image when a highly reliable pre-composite image is obtained. Whether or not the reliability is high may be determined by, for example, whether or not the number of pixels determined to be edges has reached a predetermined threshold value. With such a configuration, it is possible to reduce the possibility that the time until the end of the compositing process becomes extremely long and that the inspection image obtained by the compositing process contains many erroneous edges.

Further, in the above, the blending ratio in each alpha blend is set to be constant. However, this is an example. For example, the second inspection image acquisition unit 113 is configured to change the blend ratio of the alpha blend based on at least one of the reliability of the pre-composite image and the information about the vehicle 7 acquired from the outside. May be good.

For example, the blending ratio of the uncombined image having high reliability may be increased, and the blending ratio of the uncombined image having low reliability may be decreased. In some cases, the pre-composite image having low reliability may have a blend ratio of 0 and may not be blended. By doing so, it is possible to efficiently acquire an inspection image in which the edge strength of the body 70 is strengthened and the strength of unnecessary edges such as shadows and white lines is weakened.

The information regarding the vehicle 7 for determining whether or not to change the blend ratio may include, for example, at least one of vehicle speed information, mileage information, steering angle information, and illuminance information around the vehicle. .. These information can be obtained from, for example, the sensor unit 3. The illuminance may be acquired from the camera image.

For example, the blend ratio may be increased when the vehicle speed at the time of acquiring the camera image is high, and the blend ratio may be decreased when the vehicle speed is slow. This takes into consideration that the magnitude of the change in the driving scene when compared with the previously acquired camera image changes depending on the vehicle speed. When the change in the driving scene is large, the elements that cause the extraction of unnecessary edges such as shadows and white lines change, and the strength of the unnecessary edges can be weakened by the blending process.

Based on the same idea, for example, the blend ratio may be increased when the mileage from the acquisition of the previous camera image to the acquisition of the current camera image is long, and the blend ratio may be decreased when the mileage is short. .. It is taken into consideration that the change situation of the driving scene differs depending on the difference in the mileage. Further, for example, the blend ratio is increased when there is a large change in the steering angle between the acquisition of the previous camera image and the acquisition of the current camera image, and the blend ratio is increased when there is no significant change in the steering angle. May be made smaller. It is taken into consideration that the change situation of the driving scene differs depending on the difference in the traveling direction.

Further, for example, the blend ratio may be increased when the illuminance change of the vehicle 7 is large from the acquisition of the previous camera image to the acquisition of the current camera image, and the blend ratio may be decreased when the illuminance change is small. .. Further, when there is no constant illuminance at the time of acquiring the camera image, the blending ratio may be reduced or the blending ratio may be set to 0. It is taken into consideration that the way edge detection is performed changes greatly depending on the difference in illuminance around the vehicle. For example, there is a possibility that edge detection cannot be performed around the left and right side cameras 23 and 24 because there is no light source at night, and it may be preferable not to perform blending processing in order to prevent false detection of camera misalignment. ..

In addition, once a false detection regarding camera misalignment is performed, it is preferable to change the subsequent blending ratio.

<< 3. Points to keep in mind >>
The configurations of the embodiments and modifications in the present specification are merely examples of the present invention. The configurations of the embodiments and modifications may be appropriately changed without exceeding the technical idea of the present invention. In addition, a plurality of embodiments and modifications may be combined and implemented to the extent possible.

When there are a plurality of in-vehicle cameras, image processing conditions may be determined for each in-vehicle camera. The image processing conditions may include, for example, edge detection parameters, matching processing parameters, alpha blend blending ratio, and the like. By changing the parameters and the like according to the mounting position of the camera and the like, accurate camera deviation can be detected.

When a camera deviation is detected for debugging purposes, it is preferable to save various data used for detecting the deviation. For example, it is preferable that the inspection image used for the deviation determination is stored together with the deviation determination data (number of edge pixels, matching rate, deviation rate, etc.). The deviation determination data may be superimposed on the image. It is also preferable that the camera image that is the basis of the inspection image is also saved. Further, when an inspection image is obtained by synthesizing a plurality of camera images, it is preferable to save the parameters of the synthesizing process (blending ratio and the like), the image before the synthesizing process, and the like.

Further, in the above-described embodiment, it has been described that various functions are realized by software by the arithmetic processing of the CPU according to the program, but at least a part of these functions is electrical hardware. It may be realized by a circuit. On the contrary, at least a part of the functions realized by the hardware circuit may be realized by software.

1 ... Image processing device 7 ... Vehicle 9b ... Search area 9c ... Narrow search area 12 ... Storage unit 21 ... Front camera 22 ... Back camera 23 ... Left side camera 24 ... Right side camera 70 ... Body 111 ... First inspection image acquisition unit 113 ... Second inspection image acquisition unit 114 ... Misalignment detection unit

Claims (20)

An image processing device that processes camera images captured by an in-vehicle camera.
An inspection image acquisition unit that acquires an inspection image of the body of the vehicle on which the in-vehicle camera is mounted based on the camera image is provided.
The inspection image acquisition unit performs mask processing of the image excluding the search area for the edge of the body, which is preset according to the body shape for each vehicle type , and among the edge images obtained after the mask processing, the body An image processing device that acquires the inspection image by removing an edge that is determined to be different from the edge assumed from the design value . A storage unit that stores a reference image of the body obtained by processing the camera image captured by the in-vehicle camera normally attached to the vehicle, and a storage unit.
A deviation detection unit that detects the optical axis deviation of the in-vehicle camera by comparing the reference image and the inspection image.
Further comprising, an image processing apparatus according to claim 1. The image processing apparatus according to claim 2 , wherein the inspection image acquisition unit acquires the inspection image by performing a remasking process of masking a narrow search area narrower than the search area after the mask processing. The image processing according to claim 2 or 3 , wherein when the number of pixels detected as edges in the inspection image is less than a predetermined first threshold value, the deviation detection unit requests reacquisition of the inspection image. apparatus. After the masking process, the inspection image acquisition unit performs a remasking process of masking a narrow search area narrower than the search area.
When reacquiring the inspection image, the inspection image acquisition unit expands at least the narrow search area of the search area and the narrow search area as compared with the time of acquiring the inspection image. The image processing apparatus according to claim 4 , which acquires the inspection image. The claim that when the inspection image is reacquired, the inspection image acquisition unit determines the acquisition timing of the camera image based on the information about the vehicle acquired from the outside and acquires the inspection image. The image processing apparatus according to 4 or 5 . If the number of pixels is less than the predetermined first threshold value even after repeating the reacquisition of the inspection image a predetermined number of times, the deviation detection unit determines that the in-vehicle camera is displaced by the optical axis. The image processing apparatus according to any one of claims 4 to 6 . The deviation detection unit divides the inspection image into a plurality of regions, performs comparison processing with the reference image for each of the plurality of regions, and based on the obtained plurality of comparison processing results, the optical axis deviation. The image processing apparatus according to any one of claims 2 to 7 , wherein the image processing apparatus according to claim 2 to 7 . The image processing apparatus according to any one of claims 2 to 8 , wherein the reference image has a smaller number of pixels than the camera image. Any of claims 2 to 9 , wherein the deviation detection unit acquires comparison results of the plurality of inspection images with the reference image, and determines the optical axis deviation according to the comparison results of a plurality of times. The image processing apparatus according to item 1. The inspection image acquisition unit performs a process including the mask processing on each of the plurality of camera images captured at different timings to acquire a plurality of pre-combination images, and synthesizes the plurality of pre-combination images. The image processing apparatus according to any one of claims 2 to 9 , wherein the inspection image is acquired. The inspection image acquisition unit acquires the inspection image by sequentially repeating a process of alpha-blending the pre-synthesis image with the image stored in the compositing processing memory and updating the image stored in the compositing processing memory. The image processing apparatus according to claim 11 . The image processing apparatus according to claim 12 , wherein the initial image stored in the compositing processing memory is an image obtained by adjusting the reference image to a predetermined density. The image according to claim 12 , wherein the inspection image acquisition unit determines whether or not the pre-combination image is an initial image stored in the compositing processing memory based on the reliability of the pre-combination image. Processing equipment. The inspection image acquiring unit, a reliability of the composite image before, on the basis of at least one of information about the vehicle acquired from the outside, to vary the blend ratio of the alpha blend, claim 12 The image processing apparatus according to any one of 14 . The image processing apparatus according to any one of claims 1 to 15 , wherein the inspection image acquisition unit determines a timing for acquiring the camera image based on information about the vehicle acquired from the outside. The image processing device according to claim 16 , wherein the inspection image acquisition unit acquires the camera image when the ACC of the vehicle is turned on and the engine stop is detected. Any of claims 1 to 17 , wherein the inspection image acquisition unit acquires the camera image after instructing the light of the vehicle to turn on when it is determined that the illuminance around the vehicle is lower than a predetermined second threshold value. The image processing apparatus according to item 1. The image processing apparatus according to any one of claims 1 to 18 , wherein when there are a plurality of in-vehicle cameras, image processing conditions are determined for each of the in-vehicle cameras. An image processing method for processing a camera image captured by an in-vehicle camera by an image processing device.
The inspection image acquisition step of acquiring the inspection image of the body of the vehicle equipped with the in-vehicle camera based on the camera image is provided.
In the inspection image acquisition step , mask processing is performed on the image excluding the search area for the edge of the vehicle, which is preset according to the body shape for each vehicle type , and among the edge images obtained after the mask processing, the body An image processing method for acquiring the inspection image by removing an edge determined to be different from the expected edge from the design value of .

Images