JP7244221B2 - Object recognition device - Google Patents

Description

The present invention relates to an object recognition device.

Conventionally, there is a pedestrian recognition device that recognizes a pedestrian in front of a vehicle using the imaging results of a visible light camera and a far-infrared camera mounted on the vehicle (see, for example, Patent Document 1). This pedestrian recognition device is used particularly in dark road environments such as nighttime and tunnels, and has a high pedestrian identification recognition rate by the imaging result of the far-infrared camera and a low pedestrian identification recognition rate by the visible light camera. to inform the driver of the existence of

JP 2006-338594 A

However, the technique disclosed in Patent Document 1 requires the use of two types of cameras, a visible light camera and a far-infrared camera. That is, it was necessary to calculate the recognition rate for each of the two types of camera images. Therefore, there is a problem that complicated processing such as recognizing two types of camera images at the same time is required, and the processing load required of the system is increased.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an object recognition apparatus capable of reducing the processing load by using a camera having sensitivity in both the visible light region and the near-infrared region. With the goal.

The object recognition device according to the present invention has sensitivity in both the visible light region and the near-infrared region, and converts the images of the surroundings of the vehicle captured with visible light and near-infrared light into one image. and an object shape determination unit that determines the presence or absence of an object existing around the vehicle based on one image obtained by the imaging unit imaging both visible light and near-infrared light, a position calculation unit for calculating a position of the object based on the determination result of the object shape determination unit; and a driver visually recognizing the object in the one image based on the calculation result of the position calculation unit. and an output determination unit that determines an output pattern based on the determination result of the visual visibility determination unit , wherein the visual visibility determination unit performs the calculation of the position calculation unit. Based on the result, the saturation of the object in the one image is calculated, and based on the calculated saturation, whether or not the object in the one image can be visually recognized by the driver. It is characterized by determining whether

In the present invention, with the above configuration, it has sensitivity in both the visible light region and the near-infrared region, and converts the image captured in the visible light and the near-infrared light that captures the surroundings of the vehicle into one image. Processing load can be reduced by using a camera.
By providing the visual visibility determination unit, the visual visibility determination unit calculates the saturation of an object in one image based on the calculation result of the position calculation unit, and based on the calculated saturation, It can be determined whether an object in one image is visually recognizable by the driver.

1 is a hardware block diagram showing how the object recognition device according to the first embodiment is mounted on a vehicle; FIG. 2 is a functional block diagram showing the functional configuration of the object recognition device according to Example 1; FIG. It is a figure which shows an example of the object in the image imaged in dark environments, such as nighttime. It is a figure which shows an example of pattern image information. It is a figure which shows an example of the icon displayed on an image. 4 is a flow chart showing the overall flow of processing performed in Example 1. FIG. 7 is a flowchart showing a detailed flow of object shape determination processing shown in FIG. 6; 7 is a flowchart showing a detailed flow of object position determination processing shown in FIG. 6; FIG. 7 is a flowchart showing a detailed flow of visual visibility determination processing shown in FIG. 6. FIG. FIG. 7 is a flowchart showing a detailed flow of a variable light distribution headlight irradiation process shown in FIG. 6; FIG. FIG. 4 is an explanatory diagram showing a scene in which the object recognition device according to the first embodiment is applied to a pedestrian (No. 1); FIG. 2 is an explanatory diagram showing a scene where the object recognition device according to the first embodiment is applied to a pedestrian (No. 2); It is explanatory drawing which shows the irradiation image with the headlight provided with the function to switch a high beam/low beam automatically (part 1). It is explanatory drawing which shows the irradiation image with the headlight provided with the function to switch a high beam/low beam automatically (part 2). It is a figure which shows an example of the bird's-eye view image from which the upper direction of the own vehicle is taken as a viewpoint. FIG. 10 is an explanatory diagram showing a modification of the object recognition device according to the first embodiment;

Hereinafter, specific embodiments of an object recognition device according to the present invention will be described with reference to the drawings.

(Explanation of hardware configuration of object recognition device)
FIG. 1 is a hardware block diagram showing how an object recognition device 100 according to the first embodiment is mounted on a vehicle.

The object recognition device 100 includes an imaging unit 10, an ECU (Electronic Control Unit) 20, an AR (Augmented Reality) head-up display 31, a liquid crystal display 32, a near-infrared light 40, and a variable light distribution headlight 50. and

The imaging unit 10 has sensitivity in the near-infrared region (for example, around 800 nm to 950 nm) and visible light region (around 380 nm to 800 nm), and images the surroundings of the vehicle. The imaging unit 10 has a lens 11, an imaging device 12, and an IC 13 for signal processing adjustment.

The lens 11 refracts near-infrared light and visible light with a predetermined refractive power to form an image on the imaging device 12 .

The imaging element 12 is composed of a photoelectric conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and converts the formed light into an image I. An image signal converted into an electrical signal is output from the imaging device 12 .

The signal processing adjustment IC 13 adjusts the gain of the image signal and converts it into a predetermined image signal. The converted image signal is output as an adjusted image signal.

The signal processing adjustment IC 13 outputs an imaging parameter adjustment signal to the lens 11 and the image sensor 12 to adjust the aperture and shutter speed and set the exposure amount.

Note that the imaging unit 10 is installed on the back side of the windshield of the vehicle, for example, near the back side of the rearview mirror, facing the front of the vehicle.

The ECU 20 is configured by a microcomputer, for example, and outputs an imaging parameter adjustment instruction signal to the imaging section 10 to set imaging parameters of the imaging section 10 . The ECU 20 processes the adjusted image signal imaged with the imaging parameters to recognize objects existing around the vehicle. Here, the objects include, for example, animals (deer, raccoon dogs, etc.), motorcycles (bicycles, motorcycles, etc.), pedestrians (adults, children, wheelchairs, strollers, etc.), other vehicles (small vehicles, ordinary vehicles, large vehicles, etc.). etc.) is applicable.

When recognizing an object, the ECU 20 outputs an object recognition signal to that effect to the AR head-up display 31 or the liquid crystal display 32 .

The ECU 20 outputs a near-infrared light intensity setting signal for setting the irradiation light intensity of the near-infrared light to the near-infrared light 40 .

The ECU 20 outputs an irradiation position setting signal for setting the irradiation position of visible light to the variable light distribution headlight 50 .

The AR head-up display 31 is, for example, a combiner-type display using a transparent panel, and is arranged at a position where the driver can visually recognize it.

The liquid crystal display 32 is, for example, a rectangular flat panel display, and is arranged at a position where the driver can visually recognize it.

The near-infrared light 40 is, for example, a near-infrared light device installed near the variable light distribution headlight 50 . The near-infrared light 40 may also be used as the variable light distribution type headlight 50 of the vehicle.

The near-infrared light 40 illuminates the traveling direction of the vehicle, and irradiates the inside of the imaging range of the imaging unit 10 with the amount of near-infrared light instructed by the near-infrared light amount setting signal output from the ECU 20. do. The near-infrared light 40 outputs to the ECU 20 an illumination light off signal indicating whether or not the near-infrared light 40 is in a lighting state. This illumination light turn-off signal is used to confirm the lighting state of the near-infrared light 40 .

The variable light distribution headlight 50 illuminates the traveling direction of the vehicle, has a plurality of LED light sources, and irradiates visible light toward a position indicated by a visible light irradiation position setting signal output from the ECU 20. . The variable light distribution headlight 50 outputs to the ECU 20 an illumination light off signal indicating whether or not the variable light distribution headlight 50 is in a lighting state. This illumination light turn-off signal is used to confirm the lighting state of the variable light distribution headlight 50 .

(Description of the functional configuration of the object recognition device)
FIG. 2 is a functional block diagram showing the functional configuration of the object recognition device 100 according to the first embodiment.

The object recognition device 100 includes an imaging unit 10, an ECU 20, a display unit 30, and a variable light distribution headlight 50.

The ECU 20 includes an object shape determination section 21 , an object position determination section 22 (position calculation section), a visual visibility determination section 23 , an output determination section 24 , a light distribution control section 25 and a display control section 26 .

The object shape determination unit 21 determines whether or not there is an object around the vehicle based on the image I obtained by the imaging unit 10 capturing the surroundings of the vehicle.

The object shape determination unit 21 includes a matching determination unit 21A and a database 21B. The matching determination unit 21A performs matching processing between the image I obtained by imaging the surroundings of the vehicle by the imaging unit 10 and the content stored in the database 21B. Specifically, the template stored in the database 21B is superimposed on the image I and scanned. Then, the degree of similarity is obtained at each position, a region similar to the template is searched from the image I, and the type of object (for example, raccoon dog, bicycle, etc.) is recognized.

The object position determination unit 22 includes an object position measurement unit 22A, an object width/height measurement unit 22B, an object distance measurement unit 22C, and an object orientation measurement unit 22D.

The object position measurement unit 22A obtains the position coordinates (X, Y) on the image I of the object extracted from the image I based on the recognition result of the object shape determination unit 21 . The position coordinates (X, Y) are represented by two-dimensional coordinates with the position of the own vehicle as the origin.

The object width/height measurement unit 22B uses a well-known technique to determine the area (W×H) on the image I of the object based on the position coordinates (X, Y) measured by the object position measurement unit 22A. . Here, W indicates width and H indicates height.

The object distance measuring unit 22C, for example, converts the position coordinates (X, Y) into world coordinates, and obtains the distance from the origin of the world coordinates to the object.

The object orientation measurement unit 22D uses a well-known technique to measure which direction an object faces relative to the traveling direction of the vehicle. Specifically, the object orientation measurement unit 22D measures whether the object is facing the same direction as the traveling direction of the vehicle, is facing rightward, leftward, or faces the traveling direction. For example, the orientation of a pedestrian's body is measured by applying technology that detects the orientation of a person's face.

The visual visibility determination unit 23 includes a pixel number measurement unit 23A, a saturation detection unit 23B, an achromatic chromaticity calculation unit 23C, and a visual visibility determination unit 23D.

Based on the position coordinates of the object determined by the object position determination unit 22, the pixel number measurement unit 23A uses a well-known method to calculate the number of pixels proportional to the area (W×H) of the region on the image I of the object. demand. Here, W indicates width and H indicates height.

Next, regarding the details of the saturation detection unit 23B, the method of expressing colors numerically will be described separately for the RGB format and the YUV format.

In the case of the RGB format, the more equal the amount of each primary color, the higher the degree of achromaticity, which will be described later. In the case of complete achromaticity, R=G=B, and in the case of high achromaticity, R≈G≈B. Using this characteristic, the saturation detection unit 23B calculates each value (0 to 255) of RGB (red, green, blue) included in the object area.

In the case of the YUV format, the closer the color information (U, V) other than the luminance V to 0, the higher the achromaticity. In this case, U≈0 and V≈0. Using this characteristic, the saturation detection unit 23B calculates Y (luminance), U (blue color difference component), and V (red color difference component) of the image data included in the object region.

Note that the luminance V indicates the brightness of the color and is represented by a value of 0 to 100 (%), and the higher the value, the brighter the color. 0% corresponds to black and 100% corresponds to white. The color difference component U of blue and the color difference component V of red are represented by 256 gradations from -128 to 128.

23 C of achromaticity calculation parts calculate achromaticity (%) using the following formulas (1).

Here, the low chromaticity pixels include not only completely achromatic pixels but also pixels with achromaticity higher than a predetermined value.

FIG. 3 is a diagram showing an example of an object in an image captured in a dark environment such as nighttime. The visual visibility determination unit 23D determines whether or not the achromaticity (%) exceeds a predetermined threshold value (eg, 50%). If the achromaticity exceeds the threshold (achromaticity>50%), the object in image I (the pedestrian shown on the right side of FIG. 3) is determined to be difficult for the driver to see.

On the other hand, when the achromaticity is equal to or less than the threshold value (for example, achromaticity = 20%), it is determined that the object in the image I (the pedestrian shown on the left side of Fig. 3) can be visually recognized by the driver. be.

The output determination unit 24 determines the output pattern based on the determination result of the visual visibility determination unit 23D. An example of the output pattern is a light distribution pattern or notification pattern.

That is, when the visual availability determination unit 23D determines that the achromatic chromaticity exceeds the threshold value, the output determination unit 24 outputs the light distribution pattern to the light distribution control unit 25 or outputs the notification pattern to the display control unit 26. or

The light distribution pattern is determined by various conditions such as the type and specifications of the variable light distribution headlight 50, for example. Similarly, the notification pattern is determined by various conditions such as the type and specifications of the display unit 30, for example.

The light distribution control section 25 controls the light distribution of the variable light distribution type headlight 50 based on the determination result of the output determination section 24 . That is, the light distribution control unit 25 adjusts parameters such as the range, direction, and amount of light to be illuminated by the headlights based on the light distribution pattern output by the output determination unit 24, and maintains the adjusted values. do. The light distribution control unit 25 changes the irradiation direction of the variable light distribution headlight 50 to include the direction in which the object exists so that the driver can visually recognize the object.

The display control unit 26 displays pattern image information indicating the existence of the object on the display unit 30 based on the determination result of the output determination unit 24 . That is, the display control section 26 displays pattern image information as shown in FIG.

The pattern image information includes an annular shape with a notch (hereinafter simply referred to as "ring"). This notch is provided so as not to overlap the pedestrian on the image I (FIG. 4). A ring RI shown in FIG. 4 indicates a ring as an object detection frame displayed near the foot of the pedestrian.

The pattern image information includes an icon IC displayed on the image (FIG. 4). The icon IC shown in FIG. 4 is an icon as a highlight displayed above the head of the pedestrian. Icon types representing types of icon ICs include, for example, automobiles, pedestrians, two-wheeled vehicles, and animals (see the lower left side of FIG. 5). The right side of FIG. 5 shows a state in which the ring RI and the icon IC are superimposed on objects such as pedestrians, two-wheeled vehicles, and automobiles on the actual camera image.

The variable light distribution headlight 50 has an irradiation section 51 (FIG. 2). The irradiation unit 51 controls the lighting state of the variable light distribution headlight 50 based on the adjusted parameter value adjusted by the light distribution control unit 25 .

Next, the overall processing flow of the embodiment described above will be described with reference to the flowchart of FIG.

(Description of the overall flow of processing in the embodiment)
(Step S1) It is determined whether or not the light switch is turned on and the variable light distribution headlight 50 is on. When the light switch is on, the process proceeds to step S2, and when the light switch is off, the step S1 is repeated.

(Step S2) Near-infrared light is emitted from the near-infrared light 40 .

(Step S3) In accordance with the irradiation of the near-infrared light 40, the imaging unit 10 simultaneously captures a near-infrared image and a color image.

(Step S4) The object shape determination unit 21 determines the presence or absence of an object existing around the vehicle based on the image I obtained by the imaging unit 10, and performs object shape determination processing for recognizing the shape. A detailed flow of this process will be described later.

(Step S5) Based on the recognition result of the object shape determination process, it is determined whether or not there is an object around the vehicle. When it is determined that the object exists, the process proceeds to step S6, and when it is determined that the object does not exist, the process returns to step S3.

(Step S6) When it is determined that an object exists, the object position determination section 22 performs object position determination processing. A detailed flow of this processing will be described later.

(Step S<b>7 ) Visual visibility determining section 23 performs visual visibility determination processing based on the position coordinates determined by object position determining section 22 . A detailed flow of this processing will be described later.

(Step S8) Based on the determination result of the visual visibility determination process, the output determination unit 24 determines the output pattern according to whether or not there is an object that is difficult for the driver to visually inspect. If there is an object that is difficult to see, the process proceeds to step S9; otherwise, the process returns to step S3.

(Step S9) When the output determination unit 24 determines that there is an object that is difficult to see, a variable light distribution headlight irradiation process is performed. A detailed flow of this processing will be described later.

(Step S10) It is determined whether or not the ignition of the vehicle is off. When the ignition is off, the process of FIG. 6 is terminated, and when the ignition is on, the process returns to step S1. Here, the vehicle speed of the vehicle may be detected, and when the vehicle speed is zero, the process of FIG. 6 may be terminated, and when the vehicle speed is not zero, the process may return to step S1.

Next, the detailed flow of the object shape determination processing shown in FIG. 6 will be described using the flowchart of FIG.

(Description of the flow of object shape determination processing)
(Step S11) Template matching is performed between the image I captured in step S3 and the templates stored in the database 21B to recognize the type of object.

(Step S12) It is determined whether or not all the objects in the image I have been recognized. If all objects have been recognized, the process returns to the main routine (FIG. 6). Otherwise, the process returns to step S11 to recognize another object.

Next, the detailed flow of the object position determination processing shown in FIG. 6 will be described using the flowchart of FIG.

(Description of the flow of object position determination processing)
(Step S21) Calculate the position of the object on the image I captured in step S3.

(Step S22) The area of the object on the image I is calculated.

(Step S23) Calculate the distance from the object on the image I to the own vehicle.

(Step S24) The orientation of the object on the image I is calculated.

(Step S25) Based on the results calculated in steps S21 to S24, the object recognition result is output. After that, the process returns to the main routine (FIG. 6).

Next, the detailed flow of the visual visibility determination process shown in FIG. 6 will be described using the flowchart of FIG.

(Description of the flow of visual visibility determination processing)
(Step S31) Obtain the number of pixels proportional to the area of the object region.

(Step S32) Analyze the color information (RGB value or YUV value) of each pixel.

(Step S33) Calculate the saturation of the color information.

(Step S34) It is determined whether or not the saturation has been calculated for all the pixels in the object region. If the saturation has been calculated for all pixels, the process proceeds to step S35. Otherwise, the process returns to step S32 to analyze the color information of another pixel.

(Step S35) Achromaticity (%) is calculated. After that, the process returns to the main routine (FIG. 6).

Next, the detailed flow of the variable light distribution headlight irradiation process shown in FIG. 6 will be described using the flowchart of FIG. 11 and 12, the variable light distribution headlight irradiation process will be described. 11 and 12 show a scene in which the object recognition device 100 is applied to a pedestrian walking on a road at night.

(Description of the flow of variable light distribution headlight irradiation processing)
(Step S41) The position (x, y) of a pedestrian (FIG. 11) is calculated as a detected object that is difficult for the driver to see.

(Step S42) Calculate the area of the rectangular area RA (FIG. 12) including the pedestrian.

(Step S43) A range to be illuminated by the variable light distribution headlight 50 is calculated.

(Step S44) The direction in which the variable light distribution headlight 50 should illuminate is calculated.

(Step S45) The amount of light to be emitted by the variable light distribution headlight 50 is calculated.

(Step S46) Based on the results calculated in steps S43 to S45, the lighting state of the variable light distribution headlight 50 is controlled. After that, the process returns to the main routine (FIG. 6).

In the process shown in FIG. 6, when the output determination unit 24 determines that there is an object that is difficult to see (YES in step S8), the variable light distribution headlight irradiation process (step S9) is performed. rice field. The present invention is not limited to this aspect, and as shown in FIG. 5 and 15 is displayed (step S59).

FIG. 13 and FIG. 14 are explanatory diagrams showing irradiation images of general headlights having a function of automatically switching between high beam and low beam.

In the scene shown in FIG. 13, the left headlight with no pedestrians does not turn on the high beams.

FIG. 14 shows a situation in which a headlight having a function capable of finely variable light distribution control is applied. In the scene shown in FIG. 14, the screen is divided vertically, and in step S8 shown in FIG. 6, it is determined where the pedestrian is in the divided area. Then, when it is determined that there is an object that is difficult for the driver to visually see, only the area where the pedestrian exists is illuminated by the headlights. In the scene shown in FIG. 14, consideration is given to illuminating areas other than areas where pedestrians are present as much as possible so that the driver of an oncoming vehicle facing the host vehicle is not dazzled.

In the object recognition apparatus 100 of the first embodiment described above, the achromaticity is calculated using a camera having sensitivity in both the visible light region and the near-infrared region.

That is, it is not necessary to determine the achromaticity for each of the two types of camera images as in the conventional case. Therefore, complicated processing such as recognizing two types of camera images at the same time is not required. Therefore, the processing load required for the system can be reduced.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the embodiments are merely illustrative of the present invention, and the present invention is not limited only to the configurations of the embodiments. It goes without saying that even if there is a change in design within the scope of the gist of the invention, it is included in the present invention.

In the first embodiment described above, an example in which the icon IC is displayed on the display unit 30 has been described (FIG. 5). The present invention is not limited to this aspect, and an object that can be seen with the naked eye of the driver and an object that is difficult to see with the naked eye may be displayed in different colors. Objects visible to the naked eye may be displayed in cool colors such as bluish-green, blue, and violet, or in neutral colors such as yellow-green, green, purple, and achromatic colors. Objects that are difficult to see with the naked eye may be displayed in a conspicuous color such as pink, red, yellow, or orange.

In the first embodiment described above, an example has been described in which an object such as a pedestrian is displayed with the ring RI and the icon IC superimposed on the camera image (FIG. 5). The present invention is not limited to this aspect, and the map may be displayed by superimposing the ring RI and the icon IC on the bird's-eye view image with the viewpoint above the own vehicle (FIG. 15). In the scene shown in FIG. 15, when the distance from the own vehicle to the object exceeds a certain distance (for example, about 50 m), the driver sees the object differently.

In the first embodiment described above, an example in which a combiner type display is used as the AR head-up display 31 has been described. The present invention is not limited to this aspect, and the AR head-up display 31 may be, for example, a method of directly displaying an image on the front window.

10... Imaging unit 21... Object shape determination unit 22... Object position determination unit (position calculation unit)
23D Visual visibility determination unit 24 Output determination unit 25 Light distribution control unit 26 Display control unit 30 Display unit 31 AR head-up display 32 Liquid crystal display 50... Variable light distribution type headlight 100... Object recognition device IC... Icon (pattern image information)
RI... Ring (pattern image information)

Claims (5)

an imaging unit that has sensitivity in both the visible light region and the near-infrared region and converts an image captured with visible light and near-infrared light that captures the surroundings of the vehicle into one image;
an object shape determination unit that determines the presence or absence of an object existing around the vehicle based on one image obtained by the imaging unit capturing images with both visible light and near infrared light;
a position calculation unit that calculates the position of the object based on the determination result of the object shape determination unit;
a visual visibility determination unit that determines whether or not the object in the one image can be visually recognized by a driver based on the calculation result of the position calculation unit;
an output determination unit that determines an output pattern based on the determination result of the visual visibility determination unit ;
The visual visibility determination unit calculates the saturation of the object in the one image based on the calculation result of the position calculation unit, and calculates the saturation of the object in the one image based on the calculated saturation. 3. An object recognition device, characterized in that it determines whether or not the object of claim 1 can be visually recognized by a driver . In the object recognition device according to claim 1 ,
The visual visibility determination unit includes an achromatic chromaticity calculation unit that calculates an achromatic chromaticity indicating a ratio of the number of pixels with low chromaticity to all pixels included in the region of the object based on the calculated chroma. ,
The visual availability determination unit determines that visual recognition by the driver is difficult when the achromaticity exceeds a threshold value,
An object recognition device, wherein when the achromaticity is equal to or less than the threshold value, it is determined that the driver can visually recognize the object. The object recognition device according to claim 1,
An object recognition device, comprising: a light distribution control section that controls light distribution of a headlight based on a determination result of the output determination section. In the object recognition device according to claim 3 ,
The light distribution control unit
An object recognition device, wherein the irradiation direction of the headlight is changed so as to include the direction in which the object exists, based on the determination result of the output determination unit. In the object recognition device according to claim 1,
An object recognition apparatus, comprising: a display control unit for displaying pattern image information indicating existence of the object on a display unit based on the determination result of the output determination unit.

Images