JP4924041B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that determines a light environment for an object from a captured image.

  Various devices have been developed in order to prevent vehicle accidents, such as an aside look determination device and a collision prevention device. In the case of the aside look determination device, the face area is detected from the captured image around the face of the driver imaged by the camera, the face direction is detected from the face area image, and it is determined whether the face is looking aside. .

When sunlight hits a part of the driver's face from the side, bright and dark parts appear on the face. In this case, since the camera performs shutter control in accordance with the brightness, there is a possibility that a portion of the driver's face in the captured image that is exposed to light is crushed in white and a portion that is not exposed to light is crushed in black. From such a captured image, each part such as the eyes and nose of the driver's face cannot be detected with high accuracy, and the face area and the face direction are detected incorrectly. Therefore, in the apparatus described in Patent Document 1, each part of the face is detected, and light is emitted when the difference in luminance value of a certain line at a predetermined position on the left and right sides of the face (lower portions of the left and right eyes) exceeds a threshold value. Judge that the environment is deteriorating (Judge that sunlight is shining from the side of the driver).
JP 2002-15322 A

  In the device described in Patent Document 1, since it is determined whether or not sunlight is hit from the difference between the left and right luminance values at a predetermined position in the face (particularly near the center), an incorrect determination is made. There is a case. For example, when a part of the driver's face is exposed to sunlight, there is a possibility that each part or face area of the driver's face is erroneously detected from the captured image as described above. Therefore, the left and right predetermined positions in the face may be wrong positions, and correct determination cannot be made. Further, depending on the incident angle of sunlight with respect to the driver's face, both the left and right lower eyes may receive light or both may not receive light. In this case, since the difference between the left and right luminance values does not increase, it is determined that the driver's face is not exposed to sunlight from the side.

  Therefore, an object of the present invention is to provide an image processing apparatus that determines with high accuracy whether or not a part of an object is exposed to light.

An image processing apparatus according to the present invention is an image processing apparatus that determines a light environment for a face from a captured image obtained by capturing the face, and an edge detection unit that detects an edge of a boundary between the face and the background based on the captured image; , next to a portion of the face when the difference in the number of pixels that is the edge between the left and right near the boundary between the boundary of the edge based face and the background of the detected face and the background in the edge detecting means is equal to or larger than the threshold direction characterized in that it comprises a determining means for determining that light is hitting a.

  In this image processing apparatus, the edge detection means detects the edge of the object based on the captured image. Since there is a luminance difference between the object in the captured image and the background, the boundary portion becomes an edge (the luminance gradient increases). If a part of the object is exposed to light, the object in the captured image can have a higher brightness side and a lower brightness side. Contrast with the side where the luminance is low appears, and a clear luminance difference occurs between the positions where the edges are symmetrical. Therefore, in the image processing apparatus, the determination unit sets a comparison position (for example, a symmetrical position in the left-right direction and a symmetrical position in the vertical direction) that is symmetric based on the detected edge of the object, and brightness between the comparison positions. It is determined whether or not the difference is equal to or greater than a threshold value. If the luminance difference is equal to or greater than the threshold value, it is determined that light is applied to a part of the object. As described above, in the image processing apparatus, it is possible to determine with high accuracy whether or not light is applied to a part of the object by determining based on the edge of the object. Note that the luminance used for determination includes various information related to luminance in addition to the luminance value itself, for example, edge information indicating a boundary of luminance change (where the luminance gradient is large).

In the image processing apparatus of the present invention, the determination unit is configured when the difference in the number of pixels serving as edges between the edge portions that are symmetrical with respect to the left and right sides of the face area based on the edge detected by the edge detection unit is equal to or greater than a threshold value. It is preferable to determine that a part of the face is exposed to light from the lateral direction .

  In this image processing apparatus, an object region is acquired. The target area is an area where the target object exists in the captured image, and any acquisition method may be used. For example, the target area may be set in the past process or set in advance by an experiment or the like. The boundary portion between the object and the background in the captured image exists near the end of the object region. Therefore, in the image processing apparatus, when the judgment unit sets the vicinity of the end of the target area based on the detected edge of the target as a comparison position, and the brightness difference between the ends is greater than or equal to the threshold value, It is determined that a part of the object is exposed to light. Thus, in the image processing apparatus, by setting the comparison position with reference to the symmetrical end of the object region, the comparison position can be obtained easily and with high accuracy, and light is emitted to a part of the object. It can be determined with higher accuracy whether or not it is hit.

  In this image processing apparatus, the determination means sets left and right comparison positions that are symmetric based on the detected edge of the object, and determines whether or not the luminance difference between the left and right comparison positions is greater than or equal to a threshold value. When the luminance difference is equal to or greater than the threshold value, it is determined that a part of the object is irradiated with light from the lateral direction. When light strikes from the lateral direction of the object, the luminance on one side of the object is high and the other side is low, so that a clear luminance difference occurs between the left and right comparison positions.

  When a part of the object is exposed to light, the contrast between the high brightness side and the low brightness side appears in the edge part as described above. A difference in luminance gradient (difference in edge appearance) occurs between symmetrical positions. Therefore, in the image processing apparatus, the number of pixels that are edges at the comparison position that is symmetric based on the detected edge of the object is counted by the determination unit, and the difference in the number of edge pixels between the comparison positions is counted. If it is equal to or greater than the threshold, it is determined that a part of the object is exposed to light. As described above, in the image processing apparatus, it is possible to determine with high accuracy whether or not light is applied to a part of the object by using the edge information in which the luminance change is likely to appear.

  According to the present invention, it is possible to determine with high accuracy whether or not light is applied to a part of the object by determining based on the edge of the object in the captured image.

  Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the image processing device according to the present invention is applied to a face orientation detection device mounted on a vehicle. The face direction detection device according to the present embodiment detects the face direction of a driver and provides the face direction information to various driving support devices such as an aside look determination device and a collision prevention device.

  With reference to FIG.1 and FIG.2, the face direction detection apparatus 1 is demonstrated. FIG. 1 is a configuration diagram of a face orientation detection apparatus according to the present embodiment. FIG. 2 is an example of a captured image to which a face area and a vertical edge histogram are added.

  The face orientation detection device 1 captures the periphery of the driver's face, detects the driver's face area from the captured image, and further detects the face orientation. In particular, the face direction detection apparatus 1 determines whether or not the light environment for the driver's face is suitable for processing in order to reduce erroneous detection of the face area and the face direction (specifically, the driver's face). To determine whether light (sunlight) is shining on one side from the side). The face orientation detection device 1 includes a camera 2 and an image processing device 3.

  The camera 2 is a near-infrared camera that acquires a captured image around the driver's face, captures near-infrared light, and generates a near-infrared image corresponding to the brightness of the near-infrared light. The camera 2 is attached to the instrument panel or steering column in the direction of the driver's face. The camera 2 captures images at regular time intervals (for example, 1/30 seconds), and transmits near-infrared image information of each frame to the image processing device 3 as an image signal. In order to obtain a clear near-infrared image, it is preferable that a near-infrared projector is provided, and the near-infrared projector emits near-infrared light around the driver's face.

  The image processing apparatus 3 includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like for image processing, and performs various processes by executing software held in the ROM by the CPU. Do. The image processing device 3 receives an image signal from the camera 2 at regular time intervals, and sequentially takes near-infrared captured images of each frame. Then, the image processing device 3 performs the light environment determination process for each frame. When it is determined that the driver's face is not exposed to light from the lateral direction or when the light environment determination cannot be performed, the face area detection process is performed. And face orientation detection processing. In the present embodiment, the processing in the image processing apparatus 3 corresponds to edge detection means and determination means described in the claims.

  When the image processing apparatus 3 obtains a captured image of a new frame, it performs a light environment determination process. First, the image processing apparatus 3 determines whether or not the face area value has been updated in the process for the previous frame. The face area is an area where the driver's face exists in the captured image, and the frame of this area corresponds to the boundary between the face and the background. Since the frame interval is extremely short, it is predicted that the driver's face position has hardly moved between the previous frame and the current frame. Therefore, the vicinity of the boundary between the driver's face and the background is searched using the left and right frames of the face area value of the previous frame. Conversely, if there is no face area value in the previous frame, the face area value cannot be used because the driver's face position in the current frame may have moved from the face area value.

  When the face area value is updated in the process for the previous frame, the image processing apparatus 3 obtains a vertical edge using a vertical edge filter for all pixels of the captured image (luminance image), and generates a vertical edge image. . The vertical edge image is a binary image composed of pixels that are edges and pixels that are not edges. Then, the image processing device 3 generates a vertical edge histogram from the vertical edge image. The vertical edge histogram is a columnar graph in which the number of pixels that are edges is counted for each line in the vertical direction, and the count value is frequency-distributed. The edge is a boundary between a bright part and a dark part in the image, and indicates a boundary of luminance change (where the luminance gradient is large).

  Since the interior of the vehicle is relatively dark, when light hits the driver's face, the boundary between the face and the background becomes an edge. Therefore, the number of pixels that become vertical edges is significantly greater at the boundary between the face and the background than at other portions. In particular, when a part of the driver's face is exposed to light from the lateral direction, one side of the face is a bright part and the other side is a dark part. Therefore, the number of pixels that become vertical edges is very large at the boundary between the bright face and the background. On the other hand, in the boundary portion between the dark side face and the background, the number of pixels serving as vertical edges is larger than that in the center portion of the face, but clearly smaller than that in the bright side boundary portion. As described above, when light is applied to a part of the driver's face from the lateral direction, a difference occurs in the vertical edge at the left and right boundary portions of the face.

  Subsequently, the image processing apparatus 3 searches the vertical edge histogram for the number of search pixels on both the left and right sides centering on the left frame (horizontal pixel position) of the previous face region value, and finds the left edge peak within the range. Explore. Further, the image processing device 3 searches the vertical edge histogram for the number of search pixels on both the left and right sides with the right frame (horizontal pixel position) of the previous face area value as the center, and searches for the right edge peak within that range. To do. The number of search pixels is set in advance by an experiment or the like in consideration of the relationship between the frame interval time and the amount of movement of the face. The left edge peak and right edge peak are vertical edges in the vertical line in the range adjacent to or close to the horizontal frame of the previous face area value (that is, near the boundary between the left and right sides of the face and the background). This is the maximum value of the pixel.

  Then, the image processing apparatus 3 calculates the absolute value of the difference between the left edge peak and the right edge peak, and determines whether or not the absolute value of the difference is greater than or equal to a threshold value. The threshold is a threshold for discriminating that there is a clear difference between the left and right edge peaks due to light hitting a part of the driver's face, and is set in advance by experiments or the like. When the absolute value of the difference is greater than or equal to the threshold value, the image processing apparatus 3 determines that light has been applied to a part of the driver's face from the lateral direction, and when the absolute value of the difference is less than the threshold value, It is determined that a part of the light is not illuminated from the lateral direction. By the way, if it is determined that light is being applied to a part of the driver's face from the lateral direction, it is determined that light is applied to the larger edge peak by comparing the left edge peak and the right edge peak. You can also

  FIG. 2 shows an example of a captured image I around the face of the driver, and the face area A updated in the previous process and the vertical edge histogram G obtained in the current process are added to the captured image I. In the case of this example, since the sunlight hits the left side of the driver's face, the right side of the face is blackish (the portion indicated by the hatching). In addition, the face of the driver this time is almost within the previous face area A, and the boundary between the driver's face and the background is located near the left and right frames of the face area A. A left edge peak LP exists near the left frame, and a right edge peak RP exists near the right frame. In the case of this example, the change in luminance is clearer at the boundary between the left side of the face that is exposed to sunlight and the background than the boundary on the right side. Therefore, the left edge peak LP is clearly larger than the right edge peak RP, and the difference is remarkable.

  When a part of the face is exposed to light from the lateral direction, a portion of the driver's face in the captured image that is exposed to light may be crushed white and a portion that is not exposed to light may be crushed in black by the shutter control of the camera 2. is there. Each part such as the eyes and nose of the face cannot be accurately extracted from such a captured image, and the face area and the face orientation are erroneously detected. Therefore, a captured image captured in an optical environment where light is applied to a part of the face from the lateral direction is unsuitable for performing these processes.

  Therefore, when it is determined that a part of the driver's face is exposed to light from the lateral direction, the image processing apparatus 3 stops the face area detection process and the face direction detection process for the current frame. On the other hand, if it is determined that a part of the driver's face is not exposed to light from the lateral direction, the image processing apparatus 3 performs face area detection processing and face orientation detection processing for the current frame. The face area detection process and the face direction detection process are performed using a conventional method, and the method used is not particularly limited. When the image processing device 3 detects the face area, the face area value is updated.

  With reference to FIG. 1, the operation of the face orientation detection device 1 will be described. In particular, the processing in the image processing apparatus 3 will be described along the flowchart of FIG. FIG. 3 is a flowchart showing the flow of processing in the image processing apparatus of FIG. In the face orientation detection device 1, the following operation is repeated at regular intervals.

  The camera 2 receives near-infrared light at regular time intervals, captures a near-infrared image, and transmits the captured near-infrared image to the image processing device 3 as an image signal. The image processing device 3 receives an image signal at regular intervals, acquires a captured image of the current frame, and stores it in a predetermined area of the RAM.

  When the captured image of the current frame is acquired, the image processing apparatus 3 determines whether or not the face area value has been updated in the previous process (S1).

  When it is determined that the image is updated in S1, the image processing device 3 generates a vertical edge image from the captured image of the current frame (S2). Then, the image processing apparatus 3 creates a vertical edge histogram from the vertical edge image (S3). Subsequently, the image processing apparatus 3 searches for the left edge peak from the vertical edge histogram with reference to the left frame of the face area value of the previous frame (S4) and uses the right frame of the face area value of the previous frame as a reference. The right edge peak is searched from the vertical edge histogram (S5). Then, the image processing device 3 determines whether or not the difference (absolute value) between the left edge peak and the right edge peak is equal to or greater than a threshold value (S6).

  If it is determined in S1 that the update has not been performed, or if it is determined in S6 that the threshold is greater than or equal to the threshold, the image processing apparatus 3 detects a face area from the captured image of the current frame (S7). Further, the image processing apparatus 3 detects the face orientation from the face area image (S8). Then, the image processing apparatus 3 updates the face area value (S9) and ends the process for the current frame. Finally, the image processing device 3 transmits the face orientation information as a face orientation signal to the driving support device.

  If it is determined in S6 that the threshold is less than the threshold value, the image processing apparatus 3 ends the process for the current frame without updating the face area value (S10). Further, the image processing device 3 does not transmit the current face orientation signal to the driving support device.

  According to this face orientation detection device 1, it is possible to determine whether or not a part of the driver is exposed to light from the horizontal direction by comparing left and right edge peaks of the vertical edge histogram with reference to the previous face area value. Can be determined. In particular, since changes in luminance are compared using a histogram of vertical edges indicating boundaries, more accurate determination can be performed. Further, by searching for the left and right edge peaks using the previous face area value, the vicinity of the boundary between the face and the background in the current frame can be searched with high accuracy, and the search load can be reduced.

  Furthermore, according to the face direction detection apparatus 1, since the face area detection process and the face direction detection process are not performed for a frame that is not suitable for the light environment, erroneous detection of the face area and the face direction can be reduced. Furthermore, before performing face area detection processing and face orientation detection processing with a very large processing load, it is determined whether the current frame is suitable for each processing using the vertical edge image and its histogram. Does not perform extra processing and can reduce the processing load.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, the present invention is applied to a face orientation detection device that detects the face orientation of a driver of a vehicle, but the present invention can also be applied to other fields, and other devices such as a collision prevention device and an aside look determination device. It is good also as an apparatus built in or judging only a light environment.

  In the present embodiment, it is determined whether or not a part of the object (face) is exposed to light from the lateral direction. However, it is determined whether or not light is applied from other directions such as the vertical direction. It is good also as composition to do. In the case of the vertical direction, a horizontal edge histogram may be created from the horizontal edge image, and the difference between the upper edge peak and the lower edge peak may be determined using a threshold value.

  In this embodiment, the determination is made based on the difference in the number of pixels (edge peaks) that are the edges in the left and right vertical lines. However, the determination is made based on the difference in luminance values in the left and right vertical lines. Also good.

  In the present embodiment, the left and right edge peaks are searched using the previous face area. However, a predetermined face area (for example, an average driver's face position is obtained in advance through experiments or the like). May be used, or the left and right edge peaks may be searched without using the face area.

It is a block diagram of the face direction detection apparatus which concerns on this Embodiment. It is an example of the captured image which added the face area and the vertical edge histogram. 3 is a flowchart showing a flow of processing in the image processing apparatus of FIG. 1.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Face direction detection apparatus, 2 ... Camera, 3 ... Image processing apparatus

Claims (2)

An image processing apparatus that determines a light environment for a face from a captured image of the face ,
Edge detection means for detecting the edge of the boundary between the face and the background based on the captured image;
If the difference in the number of pixels that are the edges between the left and right borders of the face and background based on the edge of the border between the face and background detected by the edge detection means is lateral to a part of the face the image processing apparatus characterized by comprising determination means for determining that light is hitting a. The determination unit is configured to apply a lateral direction to a part of the face when a difference in the number of pixels that are edges between edge portions that are symmetrical with respect to the left and right sides of the face region based on the edge detected by the edge detection unit is equal to or greater than a threshold value. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined to be exposed to light.

Images