JP6299103B2 - Object recognition device, object recognition program used for the object recognition device, and moving body control system - Google Patents

Description

  The present invention relates to an object recognition apparatus, an object recognition program used for the object recognition apparatus, and a moving body control system.

  Recently, a moving vehicle as an object such as a traveling vehicle or a pedestrian in front of the host vehicle is recognized, and a dangerous state such as an interruption of another vehicle between the host vehicle and the preceding vehicle, a pedestrian jumping out, etc. Object recognition devices have been developed that detect early, warn drivers, and prevent danger.

  A technique for recognizing a pedestrian and a two-wheeled vehicle using feature quantities such as color and spatial frequency is known as an object recognition device (see, for example, Patent Document 1). However, when the background becomes complicated, the number of objects having the same color and the same spatial frequency as the subject increases in the background, so that the recognition rate of the target moving object decreases and the false recognition rate increases.

  Accordingly, parallax is calculated from an image obtained by the camera to obtain a parallax image (also referred to as a distance image), and the parallax image and the luminance image are used to improve the identification efficiency and identification accuracy of the moving object as the target object. A technique has been proposed (see Patent Document 2). A parallax image is an image having a parallax value as a pixel value.

However, even with the one disclosed in Patent Document 2, when the background becomes complicated, the recognition accuracy is lowered and the misrecognition rate is increased.
The present invention has been made in view of the above circumstances. An object recognition apparatus capable of improving object recognition accuracy and reducing object misrecognition, an object recognition program used for the object recognition apparatus, and movement It is to provide a body control system.

The object recognition apparatus of the present invention sequentially stores, for each frame, a stereo image input unit to which stereo image signals obtained by a stereo camera are sequentially input and at least one of the stereo images output from the stereo image input unit. A luminance image construction unit that constructs a luminance image frame by holding the parallax image frame by calculating parallax using a stereo image output in synchronization with an output from the stereo image input unit to the luminance image construction unit; A parallax image construction unit to be constructed, and an object recognition unit for recognizing an object using the luminance image frame and the parallax image frame. The object recognition unit calculates a luminance feature amount and the luminance feature amount from the luminance image frame. An object feature amount calculation unit for calculating a parallax feature amount based on a parallax value of a pixel used to An object determination processing unit for determining the object said calculates an evaluation value for determining the object by the luminance feature amount and the parallax features and of learning data and the luminance feature quantity and said parallax feature amounts includes,
The object feature amount calculation unit calculates a parallax average value and a parallax standard deviation as the parallax feature amount, and the stored data is a weighting factor for weighting the luminance feature amount for obtaining the evaluation value And a weighting coefficient for weighting the parallax feature quantity for obtaining the evaluation value,
An arithmetic expression for obtaining the evaluation value is:
An object recognition device characterized by being.
Where F (x) is an evaluation function for obtaining the evaluation value, h _t (x) is a luminance feature amount, α _t is a coefficient for weighting the luminance feature amount h _t (x), and α _d is A coefficient for weighting the parallax average value, α _σd is a coefficient for weighting the parallax standard deviation.

  According to the present invention, the luminance feature amount for recognizing the object, the parallax feature amount obtained from the parallax value of the pixel used to calculate the luminance feature amount, the luminance feature amount, and the parallax feature amount Because it was decided to determine the object based on the learning data learned in advance, the recognition accuracy of the object can be improved and the erroneous recognition of the object can be reduced.

FIG. 1 is a schematic view showing a vehicle as a moving body on which an object recognition device according to the present invention is mounted. FIG. 2 is a block diagram showing a hardware configuration of the object recognition apparatus shown in FIG. FIG. 3 is a block diagram for explaining the function of the object recognition apparatus shown in FIG. FIG. 4 is a schematic diagram for explaining the relationship between the structure of the stereo camera and the parallax. FIG. 5 is an explanatory diagram of left and right stereo images and parallax images acquired by a stereo camera, where (a) is an image by the left photographic lens system, and (b) is an example of an image by the right photographic lens system. (C) shows an example of a parallax image obtained from this stereo image. FIG. 6 is a diagram schematically showing a rectangular block cut out from the luminance image. FIG. 7 is an explanatory diagram schematically illustrating an example of a division pattern used for obtaining a luminance feature amount of a rectangular block and an example of obtaining a luminance feature amount using the division pattern. FIG. 8 shows an example of a learning object for learning the learning data regarding the luminance feature quantity and the parallax feature quantity, (a) shows the luminance image of the learning object, and (b) shows the learning object. The parallax images are shown. FIG. 9 is a diagram for explaining the concept of the hierarchical structure of the object classifier. FIG. 10 is a diagram for explaining the recognized object. FIG. 11 is a flowchart for explaining the operation of the embodiment of the object recognition processing apparatus according to the present invention.

Hereinafter, an object recognition processing apparatus according to the present invention will be described with reference to the drawings.
FIG. 1 is a schematic view showing the appearance of a vehicle as a moving body equipped with an object recognition processing apparatus according to the present invention.

  In FIG. 1, reference numeral 1 denotes a host vehicle as a moving body, 2 denotes an object recognition device, and S denotes an object (moving object) as a tracking target. The object recognition device 2 is provided in the main body of the host vehicle.

FIG. 2 is a block diagram showing a hardware configuration of the stereo camera and the object recognition device 2.
The object recognition device 2 includes a circuit that processes a signal output from the in-vehicle stereo camera shown in FIG. FIG. 2 is a block diagram showing a hardware configuration of the stereo camera. In FIG. 2, 1a and 1b indicate left and right photographing lens systems, and 2a and 2b indicate CMOS (Complementary Metal Oxide Semiconductor) as left and right imaging elements.
The object recognition device 2 may be included in a stereo camera.

The subject light is imaged on the CMOS 2a and 2b by the left and right photographing lens systems 1a and 1b, respectively.
The CMOS 2a, 2b converts the optical image formed on the imaging surface into an electric signal and outputs it as analog image data to the left and right CDS (Correlated Double Sampling) circuits 3a, 3b.

  The CDS circuits 3a and 3b remove the noise components of the analog image data output from the CMOSs 2a and 2b, and output them to the A / D converters 4a and 4b. The A / D converters 4a and 4b convert the analog image data into digital image data and output it to the image processing circuits 5a and 5b.

  The CMOSs 2a and 2b, the CDS circuits 3a and 3b, and the A / D converters 4a and 4b are timing-controlled by a timing signal generator 6 that generates a timing signal. The timing signal generator 6 is controlled by a CPU (Central Processing Unit) 7. The CPU 7 also controls the image processing circuits 5a and 5b, an image compression / decompression circuit (to be described later), and a memory card.

  The image processing circuits 5a and 5b use an SDRAM (Synchronous DRAM) 8 that temporarily stores image data, and various images such as Y, Cr, Cb conversion processing, white balance processing, contrast correction processing, edge enhancement processing, and color conversion processing. Process.

  The white balance process is an image process that adjusts the color density of image information, and the contrast correction process is an image process that adjusts the contrast of image information. The edge enhancement process adjusts the sharpness of image information, and the color conversion process is an image process that adjusts the hue of image information.

  The image information subjected to the signal processing and the image processing is recorded in the memory card 10 via the compression / decompression circuit 9. The compression / decompression circuit 9 compresses the digital image data output from the image processing circuits 5a and 5b and outputs the compressed data to the memory card 10, and also decompresses the image information read from the memory card 10 to the image processing circuits 5a and 5b. Has a function to output.

  The CPU 7 performs various arithmetic processes in accordance with a computer-readable object recognition program, a ROM (Read Only Memory) 11 as a read-only memory storing the program, a work area used in various processing steps, and various data A RAM (Random Access Memory) 12 as a readable / writable memory having a storage area and the like is built in, and these are interconnected by a bus line. Reference numeral 13 denotes an operation unit of the object recognition apparatus 2. Here, the object recognition apparatus 2 executes the function by the operation of the operation unit 13.

  An outline of processing functions of the CPU 7 of the object recognition apparatus 2 will be described with reference to FIG. Here, the CMOS 2a and 2b, the CDS circuits 3a and 3b, and the A / D converters 4a and 4b of the stereo camera have a function as a stereo image output unit that sequentially outputs a stereo image signal.

  The image processing circuits 5a and 5b receive the stereo image output from the stereo image output unit, and function as the stereo image input unit 20 that sequentially stores and holds the stereo image for each frame, and at least one of the stereo images. And a function as a luminance image constructing unit 21 that constructs a luminance image frame for each image frame.

  The CPU 7 loads a program stored in the ROM 11, calculates a parallax using a stereo image output in synchronization with an output from the stereo image input unit 20 to the luminance image construction unit 21, and generates a parallax image frame Functions as a parallax image construction unit 22 for constructing

  Further, the CPU 7 functions as an object recognition unit 23 that recognizes an object using the luminance image frame from the luminance image construction unit 21 and the parallax image frame from the parallax image construction unit 22.

  Here, the object recognizing unit 23 calculates the luminance feature amount obtained from the luminance value of the pixel in the rectangular block obtained from the luminance image frame and the parallax feature amount of the pixel used for obtaining the luminance feature amount. The object feature amount calculation unit 23a to be obtained, the object recognition dictionary memory unit (recording medium) 23c for storing the weighting factor related to the feature amount obtained by learning in advance to determine the object, and the object recognition dictionary memory unit 23c. And an object determination processing unit 23b for determining an object based on the luminance feature amount and the parallax feature amount obtained by the weighting factor and the object feature amount calculation unit 23a. The object recognition dictionary memory unit 23c is composed of the ROM 11, for example.

FIG. 4 is a diagram schematically showing the relationship between the object S and the image obtained by the stereo camera.
As shown in FIG. 4, the imaging target point O of the object S is imaged on the imaging surfaces of the CMOSs 2a and 2b by the taking lens systems 1a and 1b.

  The focal length of the photographic lens systems 1a and 1b is f, the distance between the optical axes (base line length) of the photographic lens systems 1a and 1b is D, the distance from the stereo camera to the object (moving object) S is Z, and the distance from the imaging center. If the shift amounts are Δ1 and Δ2, respectively, the parallax Δ is Δ = Δ1 + Δ2. Note that the shift amounts Δ1 and Δ2 are obtained from the imaging position of the imaging target point O on the CMOSs 2a and 2b. In this embodiment, the symbol Δ is also used to mean the parallax value.

The distance Z to the object (moving object) S can be obtained from the equation Z = D × (f / Δ).
The parallax image construction unit 22 calculates a parallax value for each pixel using the parallax Δ, and the object distance calculation unit 25a uses an arithmetic expression of Z = D × (f / Δ) to calculate the imaging target point of the object S. The distance to O can be calculated.

  In FIG. 4, the symbol s' indicates an image of the object (moving object) S formed on the imaging surfaces of the CMOSs 2a and 2b. Also, here, the optical axes of the photographing lens systems 1a and 1b are assumed to be parallel to the horizontal direction (lateral direction or horizontal direction), and the pixel lines are shifted in the vertical direction for the left and right images with respect to the horizontal optical axis. Shall not. The optical axes of the photographing lens systems 1a and 1b may be parallel to the vertical direction.

  FIG. 5 shows a stereo image and a parallax image of the object S formed on the imaging surfaces of the CMOSs 2a and 2b. (A) is a left image G (G1), and (b) is a right image G (G2). Is shown. FIG. 5C shows a schematic diagram of the parallax image G3 obtained from the stereo image. The left image G1 and the right G2 have different imaging positions on the imaging surface due to parallax. These images G1 and G2 are temporarily stored in the memory area of the RAM 12, for example.

  A luminance image frame and a parallax image frame are generated from the images G1 and G2, and the luminance images G1 and G2 and the parallax image G3 captured at the same time, that is, a plurality of images in which the luminance image frame and the parallax image frame are synchronized. Successive image frames are generated. The coordinates on the image of the luminance image frame and the coordinates on the image of the parallax image frame have a one-to-one correspondence.

The parallax image is obtained by, for example, the method described below.
First, the parallax when the horizontal lines having heights y1 and y2 shown in FIG. 5 cross the object S ′ and the road surface area RO will be described.
The parallax value of the pixel for the horizontal line of height y1 is, for example, “..., 5, 4,..., 12, 12,. Here, the parallax value “12, 12,...” Means, for example, the object S ′, and the parallax value “1, 1,...” Means the road surface region R0.

  That is, here, the parallax value “12” of the pixel on the horizontal line at the height y1 means, for example, the preceding vehicle as the object S located at about 10 m from the host vehicle 1 shown in FIG. “1” means a road surface farther away than the vehicle ahead.

  Further, the parallax values of the pixels with respect to the horizontal line having the height y2 are, for example, “..., 25, 25,..., 24,. The parallax value of the pixel on the horizontal line at the height y2 means the parallax value of the road surface region R0 that is closer to the host vehicle 1 than to the preceding vehicle.

  In order to create a parallax image, as shown in FIGS. 5A and 5B, the left and right images G1 and G2 are divided by the block Ir, and when the block difference is the smallest, the block Ir is matched. As a result, the parallax value is obtained. Note that the optimum value of the size of the block Ir is set after being determined through experiments.

  For example, 1280 × 960 pixel images G1 and G2 are divided by a block Ir of 5 × 5 size, and block matching of the block Ir of the left image G1 and the block Ir of the right image G2 by this block Ir will be described.

  It is assumed that the block Ir of the left image G1 is located at the position (x1, y1) with respect to the coordinate origin position (0, 0) for the horizontal line having the height y1. In this example, the block Ir of the left image G1 is located at the boundary between the left front landscape and the white portion on the left side of the image S1 'of the object S. For example, the sum of the luminance values of the block Ir of the left image G1 is Ir (x1, y1).

  On the other hand, the block Ir of the right image G2 is moved from the position (0, y1) to the position (x1 + Δ, y1) for the horizontal line having the height y1, for example. The sum of the luminance values of the block Ir at the position (0, y1) is Ir (0, y1), and the sum of the luminance values of the block Ir at the position (x1 + Δ, y1) is Ir (x1 + Δ, y1).

  The block Ir of the left image G1 is located at the boundary of the object image S1 ', and the left half of the area surrounded by the block Ir is dark and the right half is bright. In contrast, the area surrounded by the block Ir existing at the position (0, y1) of the right image G2 is entirely dark. Since the difference between the total luminance value Ir (x1, y1) of the block Ir of the left image G1 and the total luminance value Ir (0, y1) of the block Ir of the right image G2 is large, it is determined that block matching has been performed. Not.

  The block Ir of the right image G2 is moved from the position (0, y1) toward the position (x1 + Δ, y1), and the sum Ir (x1, y1) of the luminance values of the block Ir of the left image G1 and the right The difference from the sum Ir (x, y1) of the luminance values of the block Ir of the image G2 is sequentially obtained.

  Then, when the block Ir of the right image G2 is positioned at the position (x1 + Δ, y1), the sum Ir of the luminance values of the block Ir of the left image G1 (x1, y1) and the block Ir of the right image G2 The difference from the total luminance value Ir (x1 + Δ, y1) is minimized. When this difference is minimum, it is determined that block matching has been performed, and parallax Δ is obtained. The parallax Δ is “12”, for example, because the block Ir exists at the boundary of the image S1 ′ of the object S.

  The block Ir is sequentially moved from the horizontal line having the height y1 toward the horizontal line having the height y2. Next, in each horizontal line of height y, block Ir is moved in the x direction to perform block matching processing. Then, the parallax Δ is sequentially obtained at the boundary portion of the image S1 ′ of the object S.

  Further, for example, it is assumed that the block Ir of the left image G1 is located at the position (x1, y2) of the horizontal line having the height y2. That is, it is assumed that the block Ir of the left image G1 is located at the boundary of the white line.

  On the other hand, the block Ir of the right image G2 is moved from the position (0, y2) to the position (x1 + Δ, y2), for example, on the horizontal line having the height y2. The sum of the luminance values of the block Ir at the position (0, y2) is Ir (0, y2), and the sum of the luminance values at the position (x1 + Δ, y2) is Ir (x1 + Δ, y2).

  The block Ir of the left image G1 is located at the boundary of the white line, and the left half of the area surrounded by the block Ir is dark and the right half is bright. In contrast, the area surrounded by the block Ir existing at the position (0, y2) of the right image G2 is entirely dark. Therefore, the difference between the sum Ir (x1, y1) of the luminance values of the block Ir of the left image G1 and the sum of the luminance values Ir (0, y2) of the block Ir of the right image G2 is large, and block matching is performed. Is not determined.

  The block Ir of the right image G2 is moved from the position (0, y2) toward the position (x1 + Δ, y2), and at each position (x, y2) of the block Ir of the right image G2, the left image G1 The difference between the total luminance value Ir (x1, y2) of the block Ir and the total luminance value Ir (x, y2) of the block Ir of the right image G2 is sequentially obtained.

  Then, when the block Ir of the right image G2 is positioned at the position (x1 + Δ, y2), the sum Ir (x1, y2) of the luminance values of the block Ir of the left image G1 and the block Ir of the right image G2 The difference from the sum of the luminance values Ir (x1 + Δ, y2) is minimized. When this difference is minimum, it is determined that block matching has been performed, and parallax Δ is obtained. The parallax Δ is, for example, “24” or “25” because the block Ir exists at the boundary of the white line. A matching point is sometimes obtained for portions other than the white line of the horizontal line of height y2.

  The image thus obtained is a parallax image G3, and the parallax image G3 is schematically shown in FIG. In this parallax image G3, the one with a larger parallax value is represented by a higher luminance value, and the one with a smaller parallax is represented by a lower luminance value. In the portion where the contrast of the image is large, since the number of matching blocks Ir is large, the number of bright spots is large in the boundary portion.

The object recognition unit 23 determines an object S such as a vehicle or a pedestrian using the luminance image frame obtained by the luminance image construction unit 21.
An object recognition dictionary is created in advance by using learning data of the object S by a machine learning method. Depending on the type of object S, it is necessary to create a dictionary for object recognition separately.

  For example, in order to recognize a vehicle, it is necessary to create a dictionary that recognizes the vehicle as an object. In order to recognize a pedestrian as the object S, it is necessary to create a dictionary that recognizes the pedestrian. Therefore, an object recognition dictionary is created for each target object S.

    In order to recognize this object S, first, a rectangular block BR1 is set within the range of the luminance image G from the luminance image G (G1 or G2) as shown in FIG. The coordinates (Xs, Ys) of the upper left corner point and the coordinates (Xe, Ye) of the lower right corner point of the rectangular block BR1 are determined by the position of the rectangular block BR1 in the luminance image G and the size of the rectangular block. The rectangular block BR1 is selected in order from the largest size to the smallest size.

  In this embodiment, since normalization of the rectangular block BR1 is performed, the processing time for generating from the large size rectangular block BR1 to the small size rectangular block BR2 is the same. In the luminance image G, the number of candidates for the large-sized rectangular block BR1 is small, and the number of small-sized rectangular blocks BR2 is considered to be large. This is because it is considered that there are few moving objects in front of the host vehicle. Note that the rectangular block BR2 is described as corresponding to the object S 'shown in FIGS. 5 (a) and 5 (b).

  The number of large-sized rectangular blocks BR1 is small on the luminance image. Therefore, if the search is performed using the rectangular block BR2 having a smaller size in order from the rectangular block BR1 having a larger size, a moving object to be tracked can be detected more quickly. Also, when an image of a large object S is detected, the sensation speed increases when output.

For example, as shown in FIG. 6, a rectangular block BR1 ′ having a large size is scanned in the directions of arrows Arx and Ary in the luminance image G (G1, G2) to search the image for a recognition target object.
Next, scanning is performed using a rectangular block BR1 having a slightly smaller size, and a search is made in the image for the object to be recognized.

    The object feature amount calculation unit 23a calculates a luminance feature amount and a parallax feature amount. For example, the luminance feature amount of a black and white rectangular area existing in the rectangular block BR2 is calculated.

  FIGS. 7A to 7D schematically show examples of four characteristic division patterns for obtaining the luminance feature amount for the rectangular block BR2 corresponding to the object S ′. This division pattern is for obtaining a luminance feature amount for the vehicle.

  FIG. 7A shows a case where the rectangular block BR2 is divided into a white rectangular area BR3 and a black rectangular area BR4 in the horizontal direction. FIG. 7B shows a case where the rectangular block BR2 is divided into a white rectangular area BR3 and a black rectangular area BR4 in the vertical direction. FIG. 7C shows a case where the rectangular block BR2 is divided in the horizontal direction so that the black rectangular area BR4 exists between the white rectangular area BR3 and the white rectangular area BR3 which are spaced apart in the horizontal direction. Yes. FIG. 7D shows a case where the rectangular block BR2 is divided so that a white rectangular area BR3 and a black rectangular area BR4 exist so as to cross diagonally.

  The white rectangular area BR3 and the black rectangular area BR4 shown in FIGS. 7A to 7D are superimposed on the rectangular block BR2, respectively, and the luminance values of the respective pixels existing in the overlapping rectangular area are summed up. Find the value.

Then, the difference between the sum of the luminance values of the pixels of the rectangular block BR2 that overlaps the white rectangular region BR3 and the sum of the luminance values of the pixels of the rectangular block BR2 that overlaps the black rectangular region BR4 is obtained, and this is obtained. Luminance feature amount h _t (x).

Hereinafter, an example of the calculation of the luminance feature amount h _t (x) for the rectangular block BR2 will be described.
For example, it is assumed that the rectangular block BR2 corresponds to the object S ′ shown in FIG. This object S1 'has a substantially rectangular shape, and the lower body part is white and the upper part has a window and appears black. When the luminance feature amount h _t (x) is calculated for this block BR2 using, for example, the division pattern shown in FIG. 7B, the result is as follows.

For example, as schematically shown in FIG. 7E, it is assumed that the luminance value of the pixel in the upper part of the rectangular block BR2 is Br = 0 (black). Further, it is assumed that the luminance value of the lower half pixel of the rectangular block BR2 is Wa = 255 (white). In this case, the sum of the luminance values of the rectangular block BR2 overlapping the black rectangular area BR4 is ΣBr. On the other hand, the sum of the luminance values of the rectangular block BR2 overlapping the white rectangular area BR3 is ΣWa. ΣBr is the product of the number of pixels having the luminance value Br and the luminance value Br, and here, ΣBr = 0. Similarly, ΣWa is the product of the number of pixels having the luminance value Wa and the luminance value Wa = 255. Therefore, the feature quantity h _t (x) = ΣWa−ΣBr is much larger than zero.

Further, when the luminance feature amount h _t (x) is calculated for the object S ′ shown in FIG. 5A using the division pattern shown in FIG. As shown in FIG. 7F, the sum of the luminance values of the rectangular block BR2 that overlaps the white rectangular area BR3 in the left half is, for example, Σ (Wa + Br). Similarly, the sum of the luminance values of the rectangular block BR2 overlapping the black rectangular area BR4 in the right half is Σ (Br + Wa). In FIG. 7F, since the rectangular block BR2 has symmetrical luminance values, the feature value h _t (x) is h _t (x) = 0.

As described above, as shown in FIGS. 7 (a) to 7 (d), the size and position of the white rectangular area BR3 and the black rectangular area BR4 for obtaining the characteristic amount h _t (x) and the division pattern are used. The quantity h _t (x) is determined. Then, a luminance evaluation value f (x) for recognizing the object is obtained from the luminance feature value h _t (x).

Here, a luminance evaluation value f (x) of feature amount weighting as shown in the equation (1) is calculated using the calculated luminance feature amount h _t (x). As shown in Expression (1), the luminance feature value h _t (x) in the block is calculated, and the luminance evaluation value f (x) as the evaluation function is calculated in consideration of the weighting coefficient α _t .

The evaluation function has a luminance feature amount h _t (x) and a weighting coefficient αt. The weighting coefficient αt for the luminance feature quantity ht (x) is obtained by calculation in advance by learning. That is, learning data is collected and learned for the object recognition target, and a weighting coefficient α _t for the luminance feature amount h _t (x) is obtained.

Further, the object feature amount calculation unit 23a obtains a parallax feature amount. The object feature quantity computing unit 23a computes the parallax average value d and the parallax standard deviation value σd in order to obtain the parallax feature quantity.
The parallax average value d is obtained as described below, for example. For example, as shown in FIG. 7E, the disparity value of each pixel in the area surrounded by the white rectangular area BR3 and the area surrounded by the black rectangular area BR4 is di, and the white rectangular area BR3 and the black If the total number of pixels existing in the rectangular area BR4 is N,
The parallax average value d is d = Σdi / N
Is required.

The parallax standard deviation value σd is calculated by statistical mathematics.

The parallax average value d and the parallax standard deviation value σd are parallax feature amounts of the object.
The object determination unit 23b determines an object based on the weighting coefficient stored in the object recognition dictionary memory unit 23c, the luminance feature amount obtained by the object feature amount calculation unit 23a, and the parallax feature amount.

In order to determine the object, the object determination processing unit 23b uses the luminance feature quantity h _t (x), the parallax feature quantity d, σd, and the weighting factor to evaluate the evaluation value using the evaluation function F (x) shown below. Ask for.

Here, α _d and α _σd are weighting coefficients of the parallax feature amount.
The object recognition dictionary memory unit (recording medium) 23c has weighting factors α _t , α _d , and luminance feature amounts h _t (x), parallax feature amounts d, and σd obtained in advance for determining an object. α _σd is stored.
In the object recognition dictionary memory unit (recording medium) 23c, the coordinates of the intersection of the upper left corner among the intersections of the four corners where two sides of the rectangular block rectangular block shown in FIGS. 7A to 7D intersect. Value, the horizontal and vertical width (rectangular size) of the rectangular block, the coordinate value of the intersection of the upper left corner of the four corners where the two sides of the white area (or black area) intersect, the white area and the black area The size, the threshold value of the evaluation value of f _t (x), the weight coefficient α _t of the feature amount, and the like are stored.
Note that the symbol t is the number of luminance feature amounts, and the symbol T is the total number of luminance feature amounts.

FIG. 8 shows an example of the learning object, (a) shows the luminance image G4 of the learning object, and (b) shows the parallax image G5 of the learning object.
A luminance image region is divided from the luminance image G4 of the learning object by a rough division pattern and a fine division pattern to obtain a weighting coefficient αt for the luminance feature amount h _t (x) of the object of the luminance image G4, This is stored in the object recognition dictionary memory unit 23c.
The coordinate values (xi, yi) of the pixels for calculating the luminance feature value h _t (x) are stored with the upper left corner as the origin (0, 0), for example, and the coordinate values (xi, yi) of these pixels are stored. ) And the division pattern, the luminance feature amount h _t (x) is calculated.

Similarly, weight coefficients for the parallax average value d and the parallax standard deviation value σd of the pixels corresponding to the pattern of the luminance image G4 are obtained from the parallax image G5, and are stored in the object recognition dictionary memory unit 23c.
The weighting coefficient for the parallax feature amount is also stored as a table as shown in Table 1 for each object to be recognized in the object recognition dictionary memory unit 23c, for example.

For example, as a parallax feature amount and a weighting coefficient of an object, a parallax average value d1 = 100 (m), a weighting coefficient α _d1 = 0.1, a parallax standard deviation value σd1 = 1/10, and a weighting coefficient α _σd1 = 0.01 , Parallax average value d2 = 80 (m), weight coefficient α _d2 = 0.2, parallax standard deviation value σd2 = 1/20, weight coefficient α _σd2 = 0.02,..., Parallax average value di = 1 (m) ,... Are stored in the object recognition dictionary memory unit 23c.

Object recognition is performed using an object classifier having a hierarchy shown in FIG. Each hierarchy has an evaluation function F (x) shown in Expression (4). When the value of the evaluation function F (x) is smaller than a preset threshold value, it is determined that the object is not an object, and the evaluation of the rectangular block BR2 is stopped.
An evaluation value F (x) is calculated at each layer (n1 to n1; n is a positive integer). The rectangular block BR1 that is not determined to be an object in the last hierarchy n1 is determined to be an object candidate area.

  For example, the object feature amount calculation unit 23a first calculates the luminance evaluation value f (x) using the luminance feature amount ht (x) and the weighting coefficient αt for calculating the evaluation value F (x) in the hierarchy 11 first. . Next, the parallax average value di and the standard deviation σdi are obtained from the parallax values of the pixels used for obtaining the luminance feature quantity ht (x).

Then, an evaluation function F (x) is created using the weight coefficient.
For example, in order to determine whether or not there is a car-like feature in the hierarchy 11, the luminance evaluation value f (x) is calculated using the patterns shown in FIGS. 7 (a) to 7 (d). Next, the parallax average value di and the standard deviation σdi are obtained from the parallax values of the pixels used for obtaining the luminance feature quantity ht (x). Then, an evaluation function F (x) is created using the weight coefficient. The evaluation value obtained by the evaluation function F (x) is compared with the threshold value of the hierarchy 11.

  When the division pattern shown in FIG. 7A is used, the luminance evaluation value f (x) for the object S ′ shown in FIG. 5A is f (x) = 0. Next, the evaluation function F (x) is created using the parallax feature amount. Since the evaluation value based on the evaluation function F (x) is smaller than the threshold value, in the hierarchy 11, the division pattern shown in FIG. When it is used, the object S ′ is determined as a non-object block, and its processing is stopped.

  Then, the process proceeds to the process using the division pattern shown in FIG. In this case, the luminance evaluation value f (x) for the object S ′ shown in FIG. 5A is larger than the threshold value. Next, an evaluation function F (x) is created using the parallax feature amount. When the evaluation function F (x) is larger than the threshold value, the level 21 is entered.

  In the hierarchy 21, a finer division pattern is prepared. The number is larger than the number of division patterns prepared in the hierarchy 11. Using these division patterns, an evaluation value F (x) for the object S ′ shown in FIG. When the evaluation value F (x) exceeds the threshold value, the process moves to the next hierarchy 31. Such processing is repeated, and the rectangular block BR2 that is not determined to be an object in the last hierarchy n1 is determined to be an object. An example of the object thus determined is shown in FIG.

Next, an outline of the operation of the object recognition apparatus 2 will be described with reference to the flowchart shown in FIG.
First, the CPU 7 executes a stereo image input step in which stereo image signals obtained by the stereo camera are sequentially input (S.1).

  Next, the CPU 7 executes a luminance image construction step in which at least one of the stereo images obtained in the stereo image input step is sequentially stored and held for each frame as a luminance image to construct a luminance image frame (S.2). .

  Next, the CPU 7 executes a parallax image construction step (S.3) in which the parallax image frame is constructed by calculating the parallax using the stereo image output in synchronization with the output of the luminance image frame in the stereo image input step. .

  Next, the CPU 7 executes an object recognition step using the luminance image frame obtained by the luminance image construction step (S.2) and the parallax image frame obtained by the parallax image construction step (S.3) (S .Four). This object recognition step (S.4) calculates the luminance feature amount from the parallax image frame, and also calculates the parallax feature amount from the parallax value of the pixel used to obtain the luminance feature amount (S. .41), an input step (S.41) for inputting a weighting factor learned from the object recognition dictionary memory unit 23c, and an object determination process (S.41) for determining an object based on the luminance feature amount, the parallax feature amount, and the weighting factor. 42).

When the object is determined by the object determination process (S.42), the CPU 7 outputs an object recognition result to perform a process described later, for example, an object tracking process (S.5).
The object recognition processing device 2 is, for example, a device such as a steering device mounted on a vehicle as a moving body to avoid a collision with a recognized object, an alarm device mounted on a vehicle as a moving body, or the like. It is used as a part of a mobile control system that controls equipment.

20 ... Stereo image input unit 21 ... Luminance image construction unit 22 ... Parallax image construction unit 23 ... Object recognition unit 23a ... Object feature amount calculation unit 23b ... Object determination processing unit

JP 2008-146549 A JP 2012-226689 A

Claims (2)

Builds a luminance image frame by sequentially storing and holding at least one of the stereo image input unit to which the stereo image signal obtained by the stereo camera is sequentially input and the stereo image output from the stereo image input unit for each frame. A luminance image constructing unit, a parallax image constructing unit that constructs a parallax image frame by calculating parallax using a stereo image output in synchronization with an output from the stereo image input unit to the luminance image constructing unit, An object recognizing unit for recognizing an object using the luminance image frame and the parallax image frame, and the object recognizing unit calculates the luminance feature amount from the luminance image frame and the parallax of the pixel used to calculate the luminance feature amount An object feature amount calculation unit for calculating a parallax feature amount based on a value, and a stored luminance feature amount and a parallax feature amount An object determination processing unit for determining the object by calculating an evaluation value for determining the object憶data and the luminance feature amount and by the parallax feature amount includes,
The object feature amount calculation unit calculates a parallax average value and a parallax standard deviation as the parallax feature amount, and the stored data is a weighting factor for weighting the luminance feature amount for obtaining the evaluation value And a weighting coefficient for weighting the parallax feature quantity for obtaining the evaluation value,
An arithmetic expression for obtaining the evaluation value is:
An object recognition device characterized by being.
Where F (x) is an evaluation function for obtaining the evaluation value, h _t (x) is a luminance feature amount, α _t is a coefficient for weighting the luminance feature amount h _t (x), and α _d is A coefficient for weighting the parallax average value, α _σd is a coefficient for weighting the parallax standard deviation. A moving body control system for controlling a device mounted on a moving object using the object recognition device according to claim 1 .