JP6713938B2 - Surrounding situation grasp support system - Google Patents

Description

The present invention relates to a technique for assisting in grasping the surroundings of an automobile.

As a technology to assist in grasping the situation around the vehicle, a camera stereo camera consisting of two cameras, each of which is located at the left and right of the vehicle, is provided to capture the rear of the vehicle, and the rear of the vehicle is based on the image captured by each camera. There is known a system that detects the presence of the object and the distance to the object, and outputs a warning according to the detected distance to the object behind the automobile (for example, Patent Documents 1 and 2).

In addition, as a technology to assist in grasping the situation around the automobile, there is also a technique in which the display is placed above the front windshield instead of the rearview mirror, and the image captured by the camera capturing the rear of the automobile is displayed on the display. It is known (for example, Patent Documents 3 and 4).

JP, 2008-123462, A JP, 2013-174510, A JP, 2016-60411, A JP 2007-214749 A

When using a camera to capture the surroundings of an automobile, if the subject that has a higher brightness than the surroundings such as the headlights of other vehicles is included in the capture range at night, the gradation of the bright part of the captured image will change. Whiteout may occur that is lost due to saturation and becomes pure white.

Then, in a system that measures the presence of an object behind the vehicle and the distance to the object using the stereo camera described above, if whiteout occurs in the captured image, the presence of the object behind the vehicle and the distance to the object are detected. It becomes impossible to detect, and the user cannot correctly grasp the situation behind the vehicle. Even when whiteout does not occur, it is difficult to measure the distance to an object behind the automobile in the dark by using a stereo camera at night.

Further, in a system in which an image of the rear of the vehicle captured by the camera described above is displayed on a display provided in place of the rearview mirror, if whiteout occurs in the captured image, the user can see the rear of the vehicle from the captured image. Sometimes it becomes impossible to understand the situation correctly.

Therefore, the present invention uses an image captured by a camera, in a surrounding grasp support system that assists a user in grasping the situation around a vehicle, under each situation including a situation in which a shot image may cause overexposure. It is an object to support the user so that they can grasp the surroundings of the vehicle as well as possible.

In order to achieve the above-mentioned object, the present invention provides a first situation camera and a second situation camera, which are arranged in a laterally spaced manner, for photographing a state around a vehicle in a predetermined direction, in a surrounding condition grasping support system mounted in a vehicle. An object detection unit that positions an object existing in the predetermined direction using a stereo image including a first image that is an image captured by the first camera and a second image that is an image captured by the second camera; Night detecting means for detecting whether or not is the night, exposure amount changing means for changing the light amount of the light imaged as the second image by the second camera, a display, and whiteout in the first image. An image portion in the area of the second image corresponding to the area is generated on the area where is generated as a nighttime display image, in which the image portion is viewed in an opaque or semi-transparent manner. The night display image generation means for displaying the generated night display image on the display and the control means are provided. However, the control unit causes the object detection unit to perform positioning of an object existing in the predetermined direction when the nighttime detection unit detects that it is not currently nighttime, and the nighttime detection unit causes the nighttime detection unit to perform positioning. Is detected, the exposure amount changing means reduces the light quantity of the light photographed as the second image by the second camera, and the night display image generating means displays the night display. An image is generated and displayed on the display.

Here, in such a peripheral situation grasping support system, the exposure amount changing means changes the light amount of light in a specific wavelength band of the light taken as the second image by the second camera, and In the control means, when the nighttime detection means detects that it is currently nighttime, the exposure amount changing means is caused to set the light amount of the light of the specific wavelength band photographed as the second image by the second camera. You may make it reduce.

Further, in the above-mentioned surrounding situation grasping support system, in the control means, when the nighttime detection means detects that it is not currently nighttime, one of the first image and the second image is displayed on the display. It may be configured to display one of them.

In addition, the above-mentioned peripheral situation grasping support is performed by the nighttime display image generation means in a predetermined manner with respect to the image portion in the area of the second image corresponding to the area in the first image in which the whiteout occurs. After performing the image processing, the image portion subjected to the image processing is overlaid on the area in the first image where whiteout occurs in an opaque or semi-transparent visual form so that the nighttime display image is displayed. It may be configured to generate.

Further, in the above-mentioned surrounding situation grasping support system, the predetermined direction in which the first camera and the second camera photograph the surroundings of the automobile may be behind the automobile.
According to the surrounding situation grasp support system as described above, while the object around the automobile is positioned by the stereo image using the first camera and the second camera during the daytime, the object around the automobile can be positioned by the stereo image. At night when it becomes difficult, the portion of the first image captured by the first camera in which whiteout has occurred is reduced in the second image captured so as not to cause whiteout by reducing the exposure amount. An image in which corresponding portions are superimposed, that is, an image in which no whiteout has occurred is generated and displayed as a nighttime display image.

Therefore, in the daytime, it is possible to provide an advanced service for grasping the surroundings of the vehicle according to the positioning result of the object around the vehicle, and at night, the user can see the surroundings of the vehicle by displaying the nighttime display image. It is possible to grasp relatively well.

As described above, according to the present invention, in the surrounding situation grasping support system that assists the user in grasping the situation around the automobile by using the image photographed by the camera, the situation in which the blown-out highlight may occur in the photographed image In each of the situations including the above, the user can be assisted so that the user can grasp the surroundings of the vehicle as well as possible.

It is a block diagram showing composition of a peripheral situation grasp support system concerning an embodiment of the present invention. It is a figure showing arrangement of a peripheral situation grasp support system concerning the present invention. It is a figure which shows the daytime operation|movement of the surrounding condition grasping support system which concerns on embodiment of this invention. It is a figure which shows the operation|movement at night of the surrounding condition grasp support system which concerns on embodiment of this invention. It is a figure which shows the operation|movement at night of the surrounding condition grasp support system which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described.
FIG. 1a shows the configuration of the surrounding situation grasp support system according to this embodiment.
The surrounding condition grasping support system is a system installed in an automobile, and as shown in the figure, a night detection sensor 1, a right back camera 2, a left back camera 3, a variable optical filter 4, an alarm output unit 5, a display 6, The situation grasp support device 7 is provided.

Further, the situation grasp support device 7 includes an obstacle detection unit 71, a right image viewpoint conversion unit 72, a left image viewpoint conversion unit 73, an image processing unit 74, a combining unit 75, and a control unit 76.
Here, the nighttime detection sensor 1 detects whether or not it is currently nighttime, depending on the surrounding illuminance, the on/off state of the headlights of the automobile, and the like.
The variable optical filter 4 is a filter that reduces the light amount of light of each color gamut or the specific wavelength band that is incident on the left back camera 3 at the attenuation rate set by the control unit 76 of the control device. When the variable optical filter 4 is a filter that reduces the light amount of light in a specific wavelength band, the specific wavelength is set to a typical wavelength band of light from an automobile headlight.

Next, the right back camera 2 and the left back camera 3 are cameras having the same specifications, and the optical axes thereof are parallel to each other. Take a picture of the rear of the car.

Further, as shown in FIG. 2B, the display 6 is arranged above the front windshield, that is, at the position where the room mirror has traditionally been arranged, with the display surface facing rearward.

Returning to FIG. 1, the obstacle detection unit 71 detects a rear object by using a stereo image including a right image that is an image captured by the right back camera 2 and a left image that is an image captured by the left back camera 3. Also, the position of the rear object is measured by the stereo method.

The right image viewpoint conversion unit 72 performs a viewpoint conversion that converts the right image captured by the right-back camera 2 into a right image that represents a state in which the center of the vehicle in the left-right direction is viewed as a viewpoint, and the left image viewpoint conversion unit 73. Is a viewpoint conversion for converting the left image captured by the left back camera 3 into a left image that represents the same viewpoint as the viewpoint of the image converted by the right image viewpoint conversion unit 72 in the center of the vehicle in the left-right direction. To do.

Further, the image processing unit 74 outputs the left image, which has been subjected to the viewpoint conversion by the left image viewpoint conversion unit 73, as it is or after performing predetermined image processing.
Then, the combining unit 75 sets the right image that has been viewpoint-converted by the right image viewpoint converter 72, the left image that the image processor 74 outputs, and the image that the controller 76 outputs, set by the controller 76. Are combined and displayed on the display 6.

Then, the alarm output unit 5 outputs a voice, a warning sound, or an alarm by turning on a warning light under the control of the control unit 76.
Now, in such a configuration, the control unit 76 performs the following control depending on whether or not the nighttime detection sensor 1 detects whether it is currently nighttime.
That is, when the nighttime detection sensor 1 detects that it is not currently nighttime, that is, when the nighttime detection sensor 1 detects that it is currently daytime, the amount of light incident on the left back camera 3 by the variable optical filter 4. The tunable optical filter 4 is controlled so that the attenuation amount of 1 is minimized.

Further, when the night-time detection sensor 1 detects that it is currently in the daytime, the control unit 76 causes the synthesizing unit 75 to display and output the right image, which has been viewpoint-converted by the right-image viewpoint converting unit 72, to the display 6.

As a result, on the display 6, for example, as shown in FIG. 3a, an image showing the rear of the vehicle is displayed.
Further, when the nighttime detection sensor 1 detects that it is currently in the daytime, the control unit 76 causes the obstacle detection unit 71 to perform detection and positioning of a rear object.
Here, the obstacle detection unit 71 performs detection of a rear object by image recognition processing and positioning of the rear object by a stereo method.
That is, the obstacle detection unit 71 performs image recognition processing on the right image captured by the right back camera 2 and the left image captured by the left back camera 3 to detect an object in each image. Then, for example, as shown in FIG. 3b, the parallax dx between the position xR of the object in the right image captured by the right-back camera 2 and the position xL of the object in the left image captured by the left-back camera 3 and the write-back Using the focal length f of the camera 2 and the left back camera 3 and the distance b (distance between cameras) between the optical axis Rax of the right back camera 2 and the optical axis Lax of the left back camera 2, the distance z to the object is z= The position of the object is calculated from (f×b)/dx, and from the distance z and the direction of the object obtained from the position in the right image or the left image in the image of the object.

Then, the control unit 76 detects the presence of a rear object within a predetermined distance or a rear object approaching with a predetermined speed abnormality within a predetermined distance from the detection result of the obstacle detection unit 71, and outputs an alarm. 5 outputs a sound or outputs an alarm by turning on a warning light, outputs a warning image to the synthesizing unit 75, and causes the synthesizing unit 75 to view the warning image by the right image viewpoint converting unit 72. Warning output processing is performed to display the converted right image combined image on the display 6.

On the other hand, when the nighttime detection sensor 1 detects that it is currently nighttime, the control unit 76 stops the execution of the above-mentioned warning output process.
Further, when the nighttime detection sensor 1 detects that it is currently nighttime, the variable optical filter 4 attenuates the light amount of each color gamut or specific wavelength band incident on the left back camera 3 at a predetermined attenuation rate, The variable optical filter 4 is controlled.

Further, the control unit 76 causes the image processing unit 74 to perform the following processing when the nighttime detection sensor 1 detects that it is currently nighttime.
That is, the image processing unit 74 calculates an area in which whiteout occurs in the viewpoint-converted right image as a complementary area, and notifies the controller 76 of the calculated area. The region where whiteout occurs in the image can be detected as, for example, the region of the pixel having the maximum brightness (the pixel value is saturated).

On the other hand, the image processing unit 74 outputs the viewpoint-converted left image to the combining unit 75.
Further, when the night-time detection sensor 1 detects that it is currently nighttime, the control unit 76 causes the combining unit 75 to display the complementary region of the right image, which has been converted in the viewpoint, of the left image output from the combining unit 75. An image replaced with the image of the area corresponding to the complementary area is generated and displayed on the display 6.

As a result, at night, for example, when a viewpoint-converted right image in which overexposure occurs as in FIG. 4a and a viewpoint-converted left image as in FIG. 4b occur, first, image processing is performed. The area in the right image in which the whiteout has occurred is calculated as the complementary area 401 by the unit 74. Then, the synthesizing unit 75 replaces the complementary region 401 of the right image with the image of the region 402 corresponding to the complementary region 401 in the left image output from the image processing unit 74 shown in FIG. Is generated and displayed on the display 6.

Here, since the viewpoint-converted left image of FIG. 4b is a viewpoint-converted image of the left image captured by the left back camera 3 in which the amount of light incident on the variable optical filter 4 is attenuated, the viewpoint shown in FIG. The exposure amount (light amount of light converted into an image by the camera) is smaller than that of the converted right image, and a dark image is generated in the viewpoint shown in FIG. 4a. It can be expected that the converted left image does not cause overexposure.

Therefore, according to the image obtained by replacing the complementary region 401 of the right image shown in FIG. 4c with the image of the region 402 corresponding to the complementary region 401 in the image output from the image processing unit 74 shown in FIG. It becomes possible to understand the appearance of the portion where whiteout occurs in the left image from the image of the portion corresponding to the portion of the right image displayed by replacing the portion. In addition, since the left image is displayed as it is in the left image instead of the right image which is dark because the exposure amount is small, the user can see It is possible to satisfactorily understand the part other than the part where the occurrence.

The embodiments of the present invention have been described above.
By the way, in the above-mentioned embodiment, the left-viewpoint-converted left image is output as it is to the synthesizing unit 75 in the image processing unit 74 at night. The image portion of the area 402 corresponding to the complementary area 401 may be subjected to predetermined image processing and output to the combining unit 75.

That is, for example, the image processing unit 74 performs an edge detection process in the area 402 of the left image corresponding to the complementary area 401 in the right image in which the whiteout has occurred, and an image representing the detected edge in the area 402. May be output to the combining unit 75.

By doing so, for example, when a viewpoint-converted right image in which whiteout occurs as shown in FIG. 5a and a viewpoint-converted left image as shown in FIG. In the processing unit 74, an image as shown in FIG. 5C representing an edge in the region 402 of the left image corresponding to the complementary region 401 in which the whiteout occurs in the right image is output to the synthesizing unit 75, and the synthesizing unit 75 causes the image. As shown in FIG. 5d, an image in which the complementary region 401 of the right image is replaced with the image of FIG. 5c representing the edge in the region 402 corresponding to the complementary region 401 in the left image output from the image processing unit 74 is generated. And is displayed on the display 6.

The embodiments of the present invention have been described above. In the above embodiments, in the combining unit 75, the complementary region 401 of the right image is converted into the region 402 corresponding to the complementary region 401 of the left image output from the image processing unit 74. The above image or an image obtained by replacing the image with the image subjected to the image processing is generated and displayed on the display 6, but this is described in the combining unit 75 on the complementary region 401 of the right image. Alternatively, the image of the area 402 corresponding to the complementary area 401 of the left image output from the image processing unit 74 or the image obtained by performing the image processing on the image may be displayed in a semi-transparent manner.

Further, in the above-described embodiment, in the control unit 76, at night, the exposure time for photographing each image frame of the left back camera 3 is set shorter than that in the daytime (by increasing the so-called shutter speed), so that the left-hand side can be moved at night. The exposure amount of the left image captured by the back camera 3 may be reduced so that whiteout does not occur in the left image. Further, in this case, the variable optical filter 4 may not be provided.

Further, in the above-described embodiment, in the image processing unit 74, the complementary region 401 of the right image is set as the region in which whiteout or color skipping of the right image occurs, with or without the variable optical filter 4. In the image processing, processing of lowering the saturated color component of the area 402 corresponding to the complementary area 401 of the right image may be performed. Even in this case, it is possible to facilitate the user's grasp of the area of the right image where the color skip has occurred.

Further, in the above embodiment, the left back camera 3 and the right back camera 2 may be provided so as to photograph the front of the vehicle or the side of the vehicle.
Further, the left back camera 3 and the right back camera 2, and the left image and the right image in the above description of the embodiment may be replaced with each other.

DESCRIPTION OF SYMBOLS 1... Night detection sensor, 2... Right back camera, 3... Left back camera, 4... Variable optical filter, 5... Alarm output part, 6... Display, 7... Situation grasp support device, 71... Obstacle detection part, 72... Right image viewpoint conversion unit 73... Left image viewpoint conversion unit 74... Image processing unit 75... Compositing unit 76... Control unit.

Claims (5)

A surrounding condition grasp support system installed in a car,
A first camera and a second camera, which are arranged apart from each other on the left and right, and which photograph the surroundings of the vehicle in a predetermined direction;
An object detection unit that positions an object existing in the predetermined direction using a stereo image including a first image that is an image captured by the first camera and a second image that is an image captured by the second camera,
Night detection means to detect whether it is currently night,
Exposure amount changing means for changing the light amount of the light imaged as the second image by the second camera,
Display,
An image portion in the region of the second image corresponding to the region is overlaid on the region in the first image where the whiteout has occurred, in a form in which the image portion is visually recognized as opaque or semi-transparent. An image is generated as a nighttime display image, and the nighttime display image generating means and the control means for displaying the generated nighttime display image on the display,
The control means is
The nighttime detection means, when detecting that it is not currently nighttime, causes the object detection unit to perform positioning of the object existing in the predetermined direction,
When the nighttime detection means detects that it is currently nighttime, the exposure amount changing means decreases the light quantity of the light imaged as the second image by the second camera, and the nighttime display is performed. A surrounding situation grasping support system, characterized by causing an image generating means to generate the nighttime display image and display it on the display. The surrounding situation grasp support system according to claim 1,
The exposure amount changing unit changes a light amount of light in a specific wavelength band of light taken by the second camera as the second image,
When the night detecting means detects that it is currently night, the control means causes the exposure amount changing means to emit a light amount of light in a specific wavelength band captured by the second camera as the second image. A surrounding situation grasp support system characterized by reducing The surrounding situation grasp support system according to claim 1 or 2,
A surrounding situation, wherein the control means causes the display to display one of the first image and the second image when the nighttime detection means detects that it is not currently nighttime. Grasping support system. The surrounding situation grasping support system according to claim 1, 2 or 3,
The nighttime display image generation means performs predetermined image processing on an image portion in the region of the second image corresponding to the region in the first image in which whiteout has occurred, and then performs the image processing. Grasping the surrounding situation, characterized in that the night-time display image is generated by superimposing the processed image portion on the region of the first image in which the whiteout has occurred in an opaque or semitransparent visual form. Support system. The surrounding situation grasp support system according to claim 1, 2, 3 or 4,
The surrounding situation grasping support system, wherein the predetermined direction in which the first camera and the second camera take a picture of the surroundings of the vehicle is behind the vehicle.