JP7014122B2 - Object detection device and object detection method - Google Patents

Description

The present invention relates to an object detection device and an object detection method.

A moving object such as a vehicle is equipped with an object detection device for detecting an object. The object detection device described in Patent Document 1 detects the position of an object captured by a stereo camera from a parallax image obtained by the stereo camera. The object detection device divides the parallax image into a plurality of sections and generates a parallax histogram for each section. The object detection device pays attention to the parallax having a frequency higher than the threshold value from the distribution of the parallax obtained by the histogram, and calculates the position of the object using this parallax. By paying attention to the parallax having a frequency equal to or higher than the threshold value, the position can be detected for each object when a plurality of objects are overlapped in the divided division of the parallax image.

Japanese Unexamined Patent Publication No. 2008-64628

In Patent Document 1, a parallax histogram is generated for each category and the frequency of parallax is obtained, but the position of the object is calculated using any one of the parallax equal to or higher than the threshold value. The detection accuracy of the position of an object depends on one parallax, and it cannot be said that the detection accuracy is sufficiently high.

An object of the present invention is to provide an object detection device and an object detection method capable of improving the detection accuracy of the position of an object.

The object detection device that solves the above-mentioned problems is vertically separated into a stereo camera, a parallax image acquisition unit that acquires a parallax image in which parallax is associated with each pixel from the image of the stereo camera, and the parallax image. A division unit that divides into a plurality of division areas arranged in the horizontal direction, a histogram generation unit that generates a parallax histogram associated with each pixel included in the division area for each of the plurality of division areas, and the above. Whether or not the extraction unit that extracts the parallax whose frequency is equal to or higher than the threshold from the histogram, the maximum parallax that is the largest parallax among the parallax extracted by the extraction unit, and the most frequent maximum frequency parallax are adjacent to each other. When the determination unit for determining whether or not is adjacent to the maximum parallax and the maximum frequency parallax, the maximum frequency parallax is set as the attention parallax, and when the maximum parallax and the maximum frequency parallax are not adjacent to each other. In the case of the attention parallax setting unit in which the maximum parallax is the attention parallax and the parallax in which 1 is added to the attention parallax as the added parallax, the number of digits after the decimal point is 1 or more based on the following equation (1). Adopted parallax calculation unit that calculates parallax,

前記採用視差を用いて、前記複数の分割領域毎に物体の位置を演算する演算部と、を備える。

A calculation unit for calculating the position of an object for each of the plurality of divided regions using the adopted parallax is provided.

According to this, the position of the object is calculated using the adopted parallax. Assuming that the parallax can be expressed by a value having one or more digits after the decimal point, the parallax caused by the object closest to the stereo camera among the objects in the divided region is the value between the attention parallax and the additive parallax. .. Then, the ratio of the frequency of the attention parallax and the frequency of the additive parallax corresponds to the value after the decimal point of the parallax. For example, if the value after the decimal point of the parallax is closer to the attention parallax, the frequency of the attention parallax is higher than the frequency of the added parallax, and if the value after the decimal point of the parallax is closer to the added parallax, the frequency of the added parallax is more noticeable. More than the frequency of parallax. Therefore, the adopted parallax calculated from the ratio of the frequency of attention parallax and the frequency of additive parallax expresses the parallax caused by the object closest to the stereo camera among the objects in the divided area with one or more digits after the decimal point. Can be understood as. Since the adopted parallax has one or more digits after the decimal point, the resolution of the position of the object can be reduced as compared with the case where the position of the object is calculated using the parallax which is an integer. Therefore, the accuracy of detecting the position of the object can be improved.

The object detection device may include a road surface parallax removing unit that removes a pre-calculated road surface parallax from the parallax image.
According to this, it is possible to suppress the deterioration of the detection accuracy of the position of the object due to the influence of the parallax of the road surface.

Regarding the object detection device, the calculation unit may calculate the position of an object at a plurality of points in the divided region having different lateral coordinates.
The divided region includes a plurality of pixels in the horizontal direction. By calculating the position of the object at multiple points with different lateral coordinates, it is easy to recognize the width occupied by the object in the real space.

The object detection method that solves the above problem acquires a parallax image in which parallax is associated with each pixel from an image of a stereo camera, and the parallax image is divided in the vertical direction and a plurality of divided regions arranged in the horizontal direction. A parallax histogram associated with each pixel included in the divided area is generated for each of the plurality of divided areas, and the parallax having a frequency equal to or higher than the threshold is extracted from the histogram, and the extracted parallax is obtained. When it is determined whether or not the maximum parallax, which is the largest parallax among the above parallaxs, and the most frequent maximum frequency parallax are adjacent to each other, and the maximum parallax and the maximum frequency parallax are adjacent to each other. The maximum frequency parallax was defined as the attention parallax, the maximum parallax was defined as the attention parallax when the maximum frequency parallax and the maximum frequency parallax were not adjacent to each other, and the parallax obtained by adding 1 to the attention parallax was defined as the additional parallax. In this case, calculate the adopted parallax in which the number of digits after the decimal point is 1 or more based on the following equation (1).

前記採用視差を用いて、前記複数の分割領域毎に物体の位置を演算する。

Using the adopted parallax, the position of the object is calculated for each of the plurality of divided regions.

According to this, it is possible to improve the detection accuracy of the position of the object.

According to the present invention, the accuracy of detecting the position of an object can be improved.

Side view of a forklift equipped with an object detection device. Schematic diagram of a forklift and an object detection device. The figure which shows an example of the environment where a forklift is used. A flowchart showing an object detection process. The figure which shows the parallax image. The figure which shows the parallax image which the parallax of the road surface was removed. The figure which shows the detection range and the division area. The figure which shows the histogram of the parallax obtained from one of the division areas. The figure which shows the histogram of the parallax when the image is taken by the stereo camera in the state where a plurality of objects are overlapped. The figure which shows the histogram of the parallax when the image is taken by the stereo camera in the state where the objects do not overlap. The figure which plotted the position of the object obtained by the object detection process.

Hereinafter, an embodiment of the object detection device and the object detection method will be described.
As shown in FIG. 1, the forklift 10 includes a vehicle body 11 and a cargo handling device 12 provided on the vehicle body 11. The forklift 10 may be one in which the traveling operation and the cargo handling operation are automatically performed, or the forklift 10 may be one in which the traveling operation and the cargo handling operation are performed by the operation by the passenger.

As shown in FIG. 2, the forklift 10 includes a main controller 20, a traveling motor M1, a traveling control device 23 for controlling the traveling motor M1, and a vehicle speed sensor 24. The main controller 20 controls the traveling operation and the cargo handling operation. The main controller 20 includes a CPU 21 and a memory 22 in which programs for performing various controls and the like are stored.

The CPU 21 of the main controller 20 gives a command of the rotation speed of the traveling motor M1 to the traveling control device 23 so that the vehicle speed of the forklift 10 becomes the target speed. The travel control device 23 of this embodiment is a motor driver. The vehicle speed sensor 24 of the present embodiment is a rotation speed sensor that detects the rotation speed of the traveling motor M1. The vehicle speed sensor 24 outputs the rotation speed of the traveling motor M1 to the traveling control device 23. The travel control device 23 controls the travel motor M1 so that the rotation speed of the travel motor M1 matches the command based on the command from the main controller 20.

The forklift 10 is equipped with an object detection device 30. The object detection device 30 includes a stereo camera 31 and an image processing unit 41 that performs image processing of the image captured by the stereo camera 31. As shown in FIG. 1, the stereo camera 31 is arranged so that the road surface on which the forklift 10 travels can be seen from above the forklift 10, such as the upper part of the vehicle body 11. The vehicle body 11 includes a base provided with seats and members related to traveling, and a head guard provided on the upper part of the seat. The stereo camera 31 of the present embodiment images the rear of the forklift 10. Therefore, the object detected by the object detection device 30 is the object behind the forklift 10.

An object in front of the forklift 10 may be detected by using a stereo camera that captures an image of the front of the forklift 10. Further, an individual stereo camera that captures images in front of and behind the forklift 10 may be used to detect objects on both sides in front of and behind the forklift 10. That is, by changing the arrangement of the stereo camera, it is possible to detect an object in any direction. When a stereo camera for capturing the front of the forklift 10 is provided, for example, the stereo camera is provided on the upper part of the vehicle body 11 or the upper part of the cargo handling device 12.

As shown in FIG. 2, the stereo camera 31 includes two cameras 32 and 33. As the cameras 32 and 33, for example, a CCD image sensor or a CMOS image sensor is used. The cameras 32 and 33 are arranged so that their optical axes are parallel to each other. In this embodiment, the two cameras 32 and 33 are arranged side by side in the horizontal direction. Of the two cameras 32 and 33, one is the first camera 32 and the other is the second camera 33. Assuming that the image captured by the first camera 32 is the first image and the image captured by the second camera 33 is the second image, the same object appears laterally displaced in the first image and the second image. Become. More specifically, when the same object is imaged, the lateral pixel [px] of the object captured in the first image and the object captured in the second image are displaced according to the distance between the cameras 32 and 33. It will be. The first image and the second image have the same number of pixels, and for example, an image of 640 × 480 [px] = VGA is used. The first image and the second image are images in RGB format.

The image processing unit 41 includes a CPU 42 and a storage unit 43 including a RAM, a ROM, and the like. The storage unit 43 stores various programs for detecting an object from the image captured by the stereo camera 31. The image processing unit 41 may include dedicated hardware for executing at least a part of various processes, for example, an integrated circuit for a specific application: ASIC. The image processing unit 41 may be configured as one or more processors operating according to a computer program, one or more dedicated hardware circuits such as an ASIC, or a circuit including a combination thereof. The processor includes a CPU and a memory such as RAM and ROM. The memory stores a program code or a command configured to cause the CPU to execute the process. Memory, or computer-readable medium, includes anything that can be accessed by a general-purpose or dedicated computer.

Hereinafter, the object detection process performed by the image processing unit 41 will be described together with the object detection method. The object detection process is performed when the forklift 10 is in the activated state. The activated state is a state in which the forklift 10 can perform a traveling operation and a cargo handling operation. In the following description, the object detection process when the environment shown in FIG. 3 is imaged by the stereo camera 31 will be described as an example. In the environment shown in FIG. 3, obstacles OB1, OB2, and OB3 that hinder the progress of the forklift 10 are arranged on the road surface, and a wall W exists on the side of the road surface. The image processing unit 41 detects the position of an object such as an obstacle OB1, OB2, OB3 or a wall W.

As shown in FIGS. 4 and 5, in step S1, the image processing unit 41 acquires the parallax image dp. The parallax image dp is an image in which the parallax [px] is associated with the pixels. The parallax is obtained by comparing the first image and the second image and calculating the difference in the number of pixels between the first image and the second image for the same feature point appearing in each image. The feature point is a part that can be recognized as a boundary, such as an edge of an object. The feature points can be detected from the luminance information and the like.

The image processing unit 41 acquires the first image and the second image of the same frame from the image captured by the stereo camera 31. The image processing unit 41 converts RGB to YCrCb using a RAM that temporarily stores each image. The image processing unit 41 may perform distortion correction, edge enhancement processing, and the like. The image processing unit 41 performs stereo processing for calculating the parallax by comparing the similarity between each pixel of the first image and each pixel of the second image. As the stereo processing, a method of calculating the parallax for each pixel may be used, or a block matching method of dividing each image into blocks containing a plurality of pixels and calculating the parallax for each block may be used. .. For comparison of the similarity of each pixel, for example, SAD: Sum of Absolute Difference, SSD: Sum of Squared Difference and the like are used. The image processing unit 41 acquires a parallax image dp with the first image as a reference image and the second image as a comparison image. The image processing unit 41 extracts the pixels of the second image most similar to each pixel of the first image, and disparates the difference in the number of pixels in the horizontal direction between the pixels of the first image and the pixels most similar to the pixels. Calculated as. As a result, it is possible to acquire a parallax image dp in which parallax is associated with each pixel of the first image which is a reference image. The parallax image dp does not necessarily require display, and indicates data in which parallax is associated with each pixel in the parallax image dp. In this embodiment, it can be said that the image processing unit 41 functions as a parallax image acquisition unit.

FIG. 5 shows the parallax image dp obtained from the first image and the second image. In the parallax image dp shown in FIG. 5, the magnitude of the parallax is expressed by shading. The parallax increases as the position is closer to the stereo camera 31, and decreases as the distance from the stereo camera 31 increases. Hereinafter, the horizontal coordinates of the parallax image dl will be referred to as X coordinates, and the vertical coordinates of the parallax image dl will be referred to as Y coordinates. The X coordinate indicates the pixel position in the horizontal direction, and the Y coordinate indicates the pixel position in the vertical direction. For example, if the parallax image dp is 640 × 480 [px], the center coordinate O of the parallax image dp can be expressed as the X coordinate 320 and the Y coordinate 240. The X coordinate of the center coordinate O corresponds to the center of the forklift 10 in the width direction.

As shown in FIGS. 4 and 6, in step S2, the image processing unit 41 removes the parallax on the road surface from the parallax image dp. The road surface is the surface on which the forklift 10 is present. The parallax of the road surface is obtained in advance by the following equation (2).

Ｍ_０（ｙ）は路面により生じる視差である。ｙは、視差画像ｄｐにおけるＹ座標である。Ｂは第１カメラ３２と第２カメラ３３との離間距離＝基線長であり、詳細にいえば、第１カメラ３２の光軸と第２カメラ３３の光軸との離間距離である。Ｈはステレオカメラ３１の設置高さであり、詳細にいえば、路面からステレオカメラ３１までの距離である。θは、ステレオカメラ３１の設置角度であり、ステレオカメラ３１の光軸が水平方向に延びている場合を０°とした場合の角度である。Ｆは焦点距離である。（２）式によって求められた路面の視差は、記憶部４３に記憶されている。

M ₀ (y) is the parallax caused by the road surface. y is the Y coordinate in the parallax image dp. B is the separation distance between the first camera 32 and the second camera 33 = the baseline length, and more specifically, the separation distance between the optical axis of the first camera 32 and the optical axis of the second camera 33. H is the installation height of the stereo camera 31, and more specifically, the distance from the road surface to the stereo camera 31. θ is the installation angle of the stereo camera 31, and is the angle when the optical axis of the stereo camera 31 extends in the horizontal direction as 0 °. F is the focal length. The parallax of the road surface obtained by the equation (2) is stored in the storage unit 43.

As shown in FIG. 6, the image processing unit 41 can acquire the parallax image dl from which the parallax of the road surface is removed by removing the parallax of the road surface from the parallax image dl. That is, it is possible to obtain a parallax image dp obtained by extracting the parallax caused by an object existing at a position higher than the road surface. In the present embodiment, the image processing unit 41 functions as a road surface parallax removing unit.

As shown in FIGS. 4 and 7, in step S3, the image processing unit 41 divides the parallax image dp into a plurality of division regions A1. In the present embodiment, the detection range A for detecting the object in the parallax image dp is defined, and the detection range A is divided. The detection range A is a range in which the number of pixels in the horizontal direction is the same as the number of pixels in the horizontal direction of the parallax image dl, and the number of pixels in the vertical direction is smaller than the number of pixels in the vertical direction of the parallax image df. The parallax image dp includes a region outside the detection range A, and no object is detected in the region outside the detection range A. The areas outside the detection range A are positions that are excessively close to the stereo camera 31 and areas that are excessively distant from the stereo camera 31. The detection range A is a range in which an object should be detected as the forklift 10 advances.

The divided area A1 is an area in which the parallax image dp is vertically divided. Each divided region A1 is a region including a plurality of pixels, and can be said to be a region including a plurality of parallax associated with each pixel. The divided regions A1 are adjacent to each other in the horizontal direction. In the present embodiment, the image processing unit 41 functions as a division unit.

As shown in FIG. 4, in step S4, the image processing unit 41 determines whether or not the divided region A1 that has detected the position of the object among the plurality of divided regions A1 is the last divided region A1. The image processing unit 41 detects the position of the object in each of the divided regions A1 in the processing after step S5. When the processing of step S4 is performed after the position of the object is detected in the processing of steps S5 and subsequent steps and the determination result of step S4 is negative, the image processing unit 41 steps on the divided region A1 for which the detection of the object has not been completed. The processing after S5 is performed. That is, the position of the object is sequentially detected for each divided region A1. The determination result in step S4 is negative until the position of the object is detected for all the divided regions A1. If the determination result in step S4 is affirmative, the image processing unit 41 performs the processing in step S1. As a result, the detection of the object is repeated every predetermined frame.

In step S5, the image processing unit 41 generates a parallax histogram associated with the pixels included in the divided region A1. For example, as shown in FIG. 8, when a parallax histogram is generated for one of the divided regions A1, the distribution of the frequency for each parallax = the appearance frequency of the parallax can be obtained. When an object exists in the portion corresponding to the divided region A1 in the first image, parallax occurs depending on the object. The parallax generated in the same object has substantially the same value, although some variations occur due to the distance to the object, the size of the object, the shape of the object, the calculation accuracy of the parallax, and the like. Therefore, when an object appears in the portion corresponding to the divided region A1 in the first image, the frequency of parallax caused by the object increases. In the present embodiment, the image processing unit 41 functions as a histogram generation unit.

As shown in FIG. 4, in step S6, the image processing unit 41 extracts the parallax whose frequency is equal to or higher than the threshold value from the histogram. In the example shown in FIG. 8, the parallax 6 [px] and the parallax 7 [px] are parallax equal to or greater than the threshold value. The threshold value is set to a value at which it can be determined that the object exists by parallax. Even when an object does not exist, parallax may appear due to the accuracy of stereo processing, or parallax may appear due to an object of a small size. In such a case, the threshold value is set so that it can be considered that there is no object that hinders the running of the forklift 10 although the parallax exists. In the present embodiment, the image processing unit 41 functions as an extraction unit.

In step S7, the image processing unit 41 extracts the maximum parallax having the largest value among the parallax whose frequency is equal to or higher than the threshold value. In the example shown in FIG. 8, 7 [px] is extracted as the maximum parallax.

In step S8, the image processing unit 41 extracts the most frequent parallax among the parallax whose frequency is equal to or higher than the threshold value. In the example shown in FIG. 8, 6 [px] is extracted as the maximum frequency parallax.

In step S9, the image processing unit 41 determines whether or not the maximum parallax and the maximum frequency parallax are adjacent to each other. Since the parallax is calculated as an integer, the image processing unit 41 can determine that the maximum parallax and the maximum frequency parallax are adjacent to each other when the absolute value of the value obtained by subtracting the maximum frequency parallax from the maximum parallax is 1. .. In the present embodiment, the image processing unit 41 functions as a determination unit.

As described above, since the parallax becomes larger as it is closer to the stereo camera 31, the maximum parallax can be said to be the parallax caused by the object closest to the stereo camera 31 among the objects in the divided region A1.

As shown in FIG. 9, when the maximum parallax and the maximum frequency parallax are not adjacent to each other, it can be determined that the maximum parallax and the maximum frequency parallax are parallax caused by different objects. When a plurality of objects overlapping in the depth direction are shown in the images captured by the cameras 32 and 33, a plurality of maximum values may occur in the histogram due to the difference in distance from the stereo camera 31 to each object. .. When multiple maxima occur in the histogram, the maximum parallax and the maximum frequency parallax are not adjacent.

As shown in FIG. 10, when the maximum parallax and the maximum frequency parallax are adjacent to each other, the maximum parallax and the maximum frequency parallax can be regarded as the parallax caused by the same object. Even if the object that caused the maximum parallax and the object that caused the maximum frequency parallax are different, if the maximum parallax and the maximum frequency parallax are adjacent to each other, the two objects are placed close to each other. It can be said that it has been done. Therefore, it is considered that there is no practical problem even if both objects are regarded as the same object.

As shown in FIG. 4, when the determination result in step S9 is affirmative, the image processing unit 41 sets the maximum frequency parallax as the parallax of interest in step S10. Note: The visual parallax is a typical parallax among the parallax caused by the object closest to the stereo camera 31. When the determination result in step S9 is affirmative, the maximum parallax and the maximum frequency parallax can be determined to be the parallax caused by the object closest to the stereo camera 31. Therefore, the dominant maximum frequency parallax among the parallax generated by the object closest to the stereo camera 31 is set as the attention parallax.

If the determination result in step S9 is negative, the image processing unit 41 sets the maximum parallax as the parallax of interest in step S11. If the determination result in step S9 is negative, it can be determined that the parallax caused by a plurality of objects is included in the divided region A1. In this case, it is the object closest to the stereo camera 31 that hinders the progress of the forklift 10. Therefore, when the determination result in step S9 is negative, the position of the object closest to the stereo camera 31 is detected by setting the maximum parallax as the attention parallax. In the present embodiment, the image processing unit 41 functions as a parallax setting unit of interest.

In step S12, the image processing unit 41 extracts the frequency of the additive parallax, which is the parallax of attention parallax + 1. In the example of FIG. 8, the additive parallax is 7 [px].
In step S13, the image processing unit 41 calculates the adopted parallax [px] from the following equation (1). The adopted parallax is a parallax used to calculate the position of an object. The number of digits after the decimal point of the adopted parallax is 1 or more. In this embodiment, the number of digits after the decimal point of the adopted parallax is 1.

図８に示す例では、注目視差＝６［ｐｘ］、加算視差＝７［ｐｘ］、注目視差の頻度＝１００、加算視差の頻度＝８０である。算出される採用視差は、６．４４４≒６．４となる。本実施形態において、画像処理部４１は、採用視差算出部として機能している。

In the example shown in FIG. 8, attention parallax = 6 [px], additive parallax = 7 [px], attention parallax frequency = 100, and additive parallax frequency = 80. The calculated adopted parallax is 6.444 ≈ 6.4. In the present embodiment, the image processing unit 41 functions as an adopted parallax calculation unit.

In step S14, the image processing unit 41 calculates the distance from the stereo camera 31 to the object, that is, the position of the object, using the adopted parallax. The adopted parallax is obtained by using the parallax generated by the object closest to the stereo camera 31, and the position of the object calculated by using the adopted parallax is the position of the object in each divided region A1. It is the relative position between the object closest to the stereo camera 31 and the stereo camera 31. The position of the object calculated by using the adopted parallax can be said to be the position of the object reflected in the pixel when each divided region A1 is regarded as one pixel. The image processing unit 41 calculates the position Yw of the object with respect to the direction in which the optical axis extends in the horizontal direction from the following equation (3). In the present embodiment, the direction in which the optical axis extends in the horizontal direction is the rear direction of the forklift 10.

焦点距離は、ステレオカメラ３１の固有の値であり、記憶部４３に記憶されている。なお、位置Ｙｗは、視差画像ｄｐ中のＹ座標を実空間上の座標＝ワールド座標に変換した値ともいえる。

The focal length is a value peculiar to the stereo camera 31, and is stored in the storage unit 43. The position Yw can be said to be a value obtained by converting the Y coordinate in the parallax image dp into the coordinate in the real space = the world coordinate.

The image processing unit 41 calculates the position Xw of the object with respect to the direction orthogonal to the optical axis in the horizontal direction from the following equation (4). In the present embodiment, the horizontal direction orthogonal to the optical axis is the left-right direction of the forklift 10. The left and right are the left and right when the direction in which the stereo camera 31 is facing is the front.

ｕは視差画像ｄｐ中における分割領域Ａ１のＸ座標＝横方向の画素位置である。ｕ０は視差画像ｄｐにおける中心座標ＯのＸ座標である。なお、位置Ｘｗは、視差画像ｄｐ中のＸ座標を実空間上の座標＝ワールド座標に変換した値ともいえる。分割領域Ａ１には、複数の横方向の画素が含まれているため、ｕの取り得る範囲は、分割領域Ａ１の横方向の画素数と同一となる。ｕは分割領域Ａ１に含まれるＸ座標から任意の値とすることができるが、ｕを１つのみとした場合、フォークリフト１０の左右方向に対する物体の位置がｕの値によって変化する。詳細にいえば、分割領域Ａ１は、横方向の画素を複数含んでいるため、分割領域Ａ１に含まれる物体は、最大で横方向の画素数に対応する幅を実空間上で占有する。しかしながら、ｕを１つのみとした場合、分割領域Ａ１のＸ座標のうち１点のみしか位置の演算が行われず、分割領域Ａ１の全てのＸ座標について位置の演算を行う場合に比べて、実空間上で物体が占有する幅を認識しにくい。これは、ステレオカメラ３１から離れるほど、即ち、採用視差が小さくなるほど顕著となる。

u is the X coordinate of the divided region A1 in the parallax image dp = the pixel position in the horizontal direction. u0 is the X coordinate of the center coordinate O in the parallax image dp. The position Xw can be said to be a value obtained by converting the X coordinate in the parallax image dp into the coordinate in the real space = the world coordinate. Since the divided region A1 includes a plurality of pixels in the horizontal direction, the range that u can take is the same as the number of pixels in the horizontal direction of the divided region A1. u can be any value from the X coordinate included in the divided region A1, but when u is only one, the position of the object in the left-right direction of the forklift 10 changes depending on the value of u. More specifically, since the divided region A1 includes a plurality of pixels in the horizontal direction, the object included in the divided region A1 occupies a width corresponding to the number of pixels in the horizontal direction at the maximum in the real space. However, when u is set to only one, only one point of the X coordinates of the divided area A1 is calculated, and the position is actually calculated for all the X coordinates of the divided area A1. It is difficult to recognize the width occupied by an object in space. This becomes more remarkable as the distance from the stereo camera 31 increases, that is, as the adopted parallax becomes smaller.

In this embodiment, u is plural, and the position of the object is calculated for a plurality of points. More specifically, as shown in FIG. 8, among the X coordinates included in the divided area A1, the center X coordinate is um, the rightmost X coordinate among the X coordinates at both ends of the divided area A1 is ur, and the left end X coordinate. Let be ul. Then, the position of the object is calculated from the following equations (5), (6) and (7) in which u = um, ur and ul of the equation (4).

位置Ｘｗｍは、分割領域Ａ１内の物体の左右方向での中心部分の実空間上の位置Ｘｗである。位置Ｘｗｒは、分割領域Ａ１内の物体の右端部分の実空間上の位置Ｘｗである。位置Ｘｗｌは、分割領域Ａ１内の物体の左端部分の実空間上の位置Ｘｗである。位置Ｘｗｒは分割領域Ａ１の右端、位置Ｘｗｌは分割領域Ａ１の左端であるため、位置Ｘｗｍと位置Ｘｗｌは、物体が実空間上で占有する最大の幅を再現していると捉えることができる。位置Ｘｗｍ，Ｘｗｒ，Ｘｗｌが負の値の場合、位置Ｘｗはフォークリフト１０の幅方向の中心よりも左側に位置しているといえる。位置Ｘｗｍ，Ｘｗｒ，Ｘｗｌが正の値の場合、物体の位置はフォークリフト１０の幅方向の中心よりも右側に位置しているといえる。本実施形態において、画像処理部４１は、演算部として機能している。上記したように、画像処理部４１で実現される視差画像取得部、路面視差除去部、分割部、ヒストグラム生成部、抽出部、判定部、注目視差設定部、採用視差算出部及び演算部は、画像処理部４１のＣＰＵ４２が所定のプログラムを実行することで実現されているといえる。

The position Xwm is the position Xw in the real space of the central portion of the object in the divided region A1 in the left-right direction. The position Xwr is the position Xw in the real space of the right end portion of the object in the divided region A1. The position Xwl is the position Xw in the real space of the left end portion of the object in the divided region A1. Since the position Xwr is the right end of the divided area A1 and the position Xwl is the left end of the divided area A1, the position Xwm and the position Xwl can be regarded as reproducing the maximum width occupied by the object in the real space. When the positions Xwm, Xwr, and Xwl are negative values, it can be said that the position Xw is located on the left side of the center of the forklift 10 in the width direction. When the positions Xwm, Xwr, and Xwl are positive values, it can be said that the position of the object is located on the right side of the center in the width direction of the forklift 10. In the present embodiment, the image processing unit 41 functions as a calculation unit. As described above, the parallax image acquisition unit, the road surface parallax removal unit, the division unit, the histogram generation unit, the extraction unit, the determination unit, the attention parallax setting unit, the adopted parallax calculation unit, and the calculation unit realized by the image processing unit 41 are It can be said that this is realized by the CPU 42 of the image processing unit 41 executing a predetermined program.

As shown in FIG. 11, a bird's-eye view plot can be obtained by plotting the results of obtaining the positions Yw and Xw for each divided region A1 of the parallax image dp. FIG. 11 is a coordinate system in which the origin is the current position of the forklift 10, the vertical axis is the distance [m] with respect to the rear direction of the forklift 10, and the horizontal axis is the distance [m] with respect to the left-right direction of the forklift 10. In the figure, circles indicate Xwl, squares indicate Xwm, and rhombuses indicate Xwr. Since three points are plotted for each divided area A1, the number of plotted points is the number of divided areas A1 × 3. For convenience of explanation, the number of the divided regions A1 is underexpressed in FIG. 7, but FIG. 11 shows a bird's-eye view plot obtained when the number of the divided regions A1 is larger than that of the divided region A1 shown in FIG. ing.

As can be seen from FIG. 11, an obstacle OB1 is located near 3 [m] behind the forklift 10, an obstacle OB2 is located 5 [m] behind the forklift 10, and an obstacle OB2 is located near the left side of the forklift 10. It can be detected that the obstacle OB3 is present in the rear of the m] and in the vicinity of the left side of the 3 [m]. Further, it can be detected that a wall W extending rearward exists on the right side of the forklift 10.

When the position of the object is detected by the object detection process, the image processing unit 41 transmits the detection result of the position of the object, that is, the coordinates of the object to the main controller 20. The main controller 20 performs deceleration processing of the forklift 10 according to the distance between the forklift 10 and the object while grasping the coordinates of the object. The deceleration process includes a process of stopping the forklift 10, a process of gradually reducing the speed of the forklift 10 from the current speed, a process of limiting the speed of the forklift 10 and restricting running at a speed exceeding the limit speed, and the like. , Various processing can be mentioned.

The operation of this embodiment will be described.
The object detection device 30 detects the position of the object closest to the forklift 10 among the objects included in each divided region A1. Since the parallax becomes larger as it is closer to the forklift 10, the position of the object closest to the forklift 10 can be detected by using the maximum parallax among the parallax whose frequency is equal to or higher than the threshold value.

Here, when calculating the position of the object using only the maximum parallax in order to detect the position of the object closest to the forklift 10, the adopted parallax, which is the denominator of the equations (3) and (4), is used as the maximum parallax. The position of the object is calculated by the replaced formula. Since the parallax is calculated as an integer, the position of the object is calculated using the maximum parallax which is an integer. Assuming that the object exists at a position equivalent to a parallax of 6.5 in the real space, the maximum parallax is 6 [px] or 7 [px], and the position of the object is calculated using this maximum parallax. It will be. As can be understood from equations (3) and (4), parallax has a large effect on the detection accuracy of the position of an object, and when parallax, which is an integer, is used, the resolution of the position of the object = the distance to the object is It is large and the detection accuracy of the position of the object is low.

Assuming that the parallax can be expressed by a value having one or more digits after the decimal point, the parallax of the object closest to the stereo camera 31 among the objects in the divided region A1 is the value between the attention parallax and the additive parallax. Become. Then, the ratio of the frequency of the attention parallax and the frequency of the additive parallax corresponds to the value after the decimal point of the parallax. For example, if the value after the decimal point of the parallax is closer to the attention parallax, the frequency of the attention parallax is higher than the frequency of the added parallax, and if the value after the decimal point of the parallax is closer to the added parallax, the frequency of the added parallax is more noticeable. More than the frequency of parallax. If the position of the object in the real space is equivalent to a parallax of 6.5, it is considered that the frequency of the parallax 6 [px] and the frequency of the parallax 7 [px] are close to 5: 5. Similarly, if the position of the object corresponds to a parallax of 6.3, it is considered that the frequency of the parallax 6 [px] and the frequency of the parallax 7 [px] are close to 7: 3. That is, it is considered that the value after the decimal point of the parallax appears as the ratio of the frequencies of the adjacent parallax.

Therefore, the adopted parallax calculated from the ratio of the frequency of attention parallax and the frequency of additive parallax expresses the parallax caused by the object closest to the stereo camera 31 among the objects in the divided region A1 with one or more digits after the decimal point. It can be regarded as something that has been done.

The effect of this embodiment will be described.
(1) Since the adopted parallax has one or more digits after the decimal point, the resolution of the position of the object can be reduced as compared with the case where the position of the object is calculated using the parallax which is an integer. Therefore, the accuracy of detecting the position of the object can be improved.

It is also conceivable to detect the position of the object by performing a sub-pixelation process for subdividing each pixel of the parallax image dp. However, the sub-pixelation process increases the load on the object detection device 30. By using the adopted parallax calculated from the parallax histogram as in the present embodiment, it is possible to improve the detection accuracy of the position of the object while suppressing the increase in the load of the object detection device 30.

(2) The parallax of the road surface is removed from the parallax image dp, and the parallax histogram is generated using the parallax image df after the parallax of the road surface is removed. When a histogram is generated from a parallax image dp including the parallax of the road surface, the frequency of the parallax on the road surface may become dominant, and the parallax generated by the object may not be the parallax of interest. On the other hand, by using the parallax image dp after removing the parallax on the road surface, the parallax of interest can be appropriately set.

(3) The position of the object is calculated using three points in the divided area A1 having different X coordinates. If the position of the object is calculated only for one point in the divided region A1, it becomes difficult to recognize the width occupied by the object in the real space. By calculating the position of the object for each of the three points in the divided area A1, it is easy to recognize the width occupied by the object in the real space.

The embodiment can be modified and implemented as follows. The embodiments and the following modifications can be implemented in combination with each other within a technically consistent range.
○ The position of the object may be calculated for only one X coordinate of the divided area A1. In this case, since the position of the object in the divided area A1 is calculated only at one point, the left and right positions of the object depend on the X coordinate, and it may be difficult to recognize the width occupied by the object in the real space. be. Therefore, it is preferable to consider the width of the object, such as determining that the object exists within a predetermined range from the calculated position of the object. For example, when the position of the object in the divided region A1 is calculated using um as the X coordinate, it is preferable to determine that the object exists over a predetermined range on both the left and right sides of the calculated coordinate of the object.

○ The image processing unit 41 may generate a histogram from the parallax image dp in which the parallax on the road surface is not removed. Since the road surface is flat, the parallax of the road surface gradually decreases as the distance from the stereo camera 31 increases. Therefore, the frequency of parallax on the road surface can be predicted in advance. In consideration of this, by setting the threshold value of the parallax frequency, the position of the object can be calculated from the parallax image dp in which the parallax on the road surface is not removed.

○ The image processing unit 41 may divide the entire parallax image dp into a plurality of division areas A1 without defining the detection range A. That is, the image processing unit 41 may detect the position of the object over the entire parallax image dp.

○ Even if the parallax image acquisition unit, road surface parallax removal unit, division unit, histogram generation unit, extraction unit, judgment unit, attention parallax setting unit, adopted parallax calculation unit, and calculation unit are each configured by individual control devices. good.

○ The first camera 32 and the second camera 33 may be arranged side by side in the vertical direction.
○ The number of pixels of the first image and the number of pixels of the second image may be different. For example, the number of pixels of the second image as the comparison image may be the same as the number of pixels of the parallax image, and the number of pixels of the first image as the reference image may be larger than the number of pixels of the second image.

○ The stereo camera 31 may include three or more cameras.
○ The forklift 10 may be driven by an engine. In this case, the travel control device is a device that controls the fuel injection amount to the engine and the like.

○ The object detection device 30 may be mounted on a moving body such as an industrial vehicle or a passenger car other than the forklift 10 such as a construction machine, an automatic guided vehicle, or a truck. When the object detection device 30 is mounted on a passenger vehicle, the object detection device 30 may transmit the position of the detected object to a control device that controls, for example, an active suspension system, an automatic braking system, or the like. .. When the object detection device 30 is mounted on the automatic carrier, the object detection device 30 may transmit the position of the detected object to, for example, a control device for mapping or a control device for generating a traveling route. .. As described above, the position of the object detected by the object detection device 30 can be used for various processes.

30 ... Object detection device, 31 ... Stereo camera, 41 ... Image processing unit (parallax image acquisition unit, road surface parallax removal unit, division unit, histogram generation unit, extraction unit, determination unit, attention parallax setting unit, adopted parallax calculation unit, and Calculation unit).

Claims (4)

With a stereo camera
A parallax image acquisition unit that acquires a parallax image in which parallax is associated with each pixel from the image of the stereo camera, and a parallax image acquisition unit.
A division portion that divides the parallax image into a plurality of division areas that are vertically divided and arranged in the horizontal direction.
A histogram generation unit that generates a parallax histogram associated with each pixel included in the divided area for each of the plurality of divided areas.
An extraction unit that extracts parallax whose frequency is equal to or higher than the threshold value from the histogram,
A determination unit for determining whether or not the maximum parallax, which is the largest parallax among the parallax extracted by the extraction unit, and the most frequent maximum frequency parallax are adjacent to each other.
When the maximum parallax and the maximum frequency parallax are adjacent to each other, the maximum frequency parallax is regarded as the attention parallax, and when the maximum parallax and the maximum frequency parallax are not adjacent to each other, the maximum frequency parallax is referred to as the attention parallax. Attention parallax setting unit and
When the parallax obtained by adding 1 to the attention parallax is used as the added parallax, the adopted parallax calculation unit for calculating the adopted parallax having one or more digits after the decimal point based on the following equation (1)

An object detection device including a calculation unit that calculates the position of an object for each of the plurality of divided regions using the adopted parallax. The object detection device according to claim 1, further comprising a road surface parallax removing unit that removes a pre-calculated road surface parallax from the parallax image. The object detection device according to claim 1 or 2, wherein the calculation unit calculates the position of an object at a plurality of points in the divided region having different lateral coordinates. Obtain a parallax image in which the parallax is associated with each pixel from the image of the stereo camera.
The parallax image is divided into a plurality of divided areas that are vertically divided and arranged in the horizontal direction.
For each of the plurality of divided areas, a histogram of the parallax associated with each pixel included in the divided area is generated.
Parallax with a frequency greater than or equal to the threshold is extracted from the histogram.
It is determined whether or not the maximum parallax, which is the largest parallax among the extracted parallax above the threshold value, and the most frequent maximum frequency parallax are adjacent to each other.
When the maximum parallax and the maximum frequency parallax are adjacent to each other, the maximum frequency parallax is defined as the attention parallax, and when the maximum parallax and the maximum frequency parallax are not adjacent to each other, the maximum frequency parallax is defined as the attention parallax. ,
When the parallax obtained by adding 1 to the attention parallax is used as the added parallax, the adopted parallax having one or more digits after the decimal point is calculated based on the following equation (1).

An object detection method for calculating the position of an object for each of the plurality of divided regions using the adopted parallax.

Images