JP2829537B2 - Mobile vehicle environment recognition device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to an environment recognizing device for a mobile vehicle such as a car or a robot provided with an image input unit for recognizing an external environment. The present invention relates to an environment recognition device for a mobile vehicle having a function of detecting a positional shift between two images sequentially input in a moving vehicle.

(Prior Art) Conventionally, as shown in, for example, Japanese Patent Application Laid-Open No. 64-26913, an image input means such as a television camera is mounted on a moving vehicle, and based on an image input from the image input means. There has been proposed a technology for recognizing an external environment and automatically controlling the traveling of a mobile vehicle according to the recognition result.

When recognizing the external environment as described above, matching is often performed between two images that are sequentially input with time. That is, for example, by calculating a mismatch measure between these two images, it is possible to detect the amount of movement of a moving object (for example, another vehicle) in the images.

(Problems to be Solved by the Invention) In order to correctly recognize the external environment by such a matching method, it is naturally assumed that there is no positional displacement between the two images. However, in a moving vehicle such as an automobile, the above-described positional deviation is likely to occur due to vertical swinging (pitching) due to riding on an obstacle or rolling at the time of turning.

Although the case where matching is performed between two images has been described above, even when environmental recognition is performed for each image without using such a matching method, photosensitive recognition is performed from an image in which positional displacement has occurred. When you do
There is a high possibility that the environment will be mistakenly recognized.

In order to solve such a problem, it is only necessary to correctly detect the displacement and correct the displacement based on the detection result. For this purpose, conventionally, sensors for detecting the behavior of the moving vehicle such as pitching and rolling described above are provided, and the displacement of the image is estimated based on the detected behavior of the moving vehicle.

However, in such a case, it is necessary to correctly take correspondence between the behavior of the moving vehicle and the image position shift. In many cases, the above-mentioned sensors have problems of characteristic variation and aging, so it is actually quite difficult to correctly take the above-mentioned measures in any case. Naturally, the image position shift is incorrectly detected.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a mobile vehicle environment recognition device capable of directly and accurately detecting an image position shift from an input image signal. Is what you do.

(Means and Actions for Solving the Problems) The first environment recognition apparatus for a mobile vehicle according to the present invention is, as described above, an environment recognition apparatus for a mobile vehicle having image input means for external world recognition. Means for obtaining a difference image between a first image at a specific time and a second image after a predetermined time from among the images sequentially input by the image input means, wherein each of the first image, the second image and the difference image ,
Means for obtaining a projection on the substantially horizontal x-axis and a projection on the y-axis perpendicular to the x-axis; and a method for determining the projection based on the correlation between the projection on the first image or the second image and the projection on the difference image. One image and the second
Means for detecting the presence / absence and the amount of positional displacement between images.

The image captured for environmental recognition is generally an image of the situation on the front side of the traveling road, that is, an image that can be seen by the occupant in the normal driver's seat (see (1) in FIG. 2). Therefore, as long as there is no pitching or rolling of the moving vehicle, the first image and the second image are substantially the same as shown in (2) and (3) of FIG. Therefore, in this case, as shown in FIG. 4D, the difference images are basically images in which nothing is photographed, and are projected on the x-axis (hereinafter, referred to as x-projection).
Also, the projection on the y-axis (hereinafter referred to as y-projection) is 0
A set of density values close to (zero).

In contrast, for example, when pitching occurs in a moving vehicle, the first image, the second image, and a difference image thereof are
These are as shown in (5), (6) and (7) of FIG. 2, respectively. In other words, when this pitching occurs, the second
The image is displaced in the y-axis direction with respect to the first image.
The x-projection of the difference image is significantly different from that of the difference image when there is no displacement, and the first image or the second image
As compared with the x-projection of the image, the value becomes almost twice at each position. Therefore, by examining the correlation between the x-projection of the difference image and the x-projection of the first image or the second image, it is determined whether there is a displacement in the y-axis direction between the first image and the second image. It is possible to detect whether or not.

In that case, a significant difference occurs between the y-projection of the difference image and the y-projection of the first image or the second image in accordance with the displacement between the first image and the second image. , These two
By examining the correlation between the two projections, it is possible to detect the amount of the positional shift.

Here, as can be seen from (5) and (6) in FIG. 2, when there is a displacement between the first image and the second image, the y-projection of the first image and the y-projection of the second image are performed. And a significant difference corresponding to the positional deviation occurs. The present invention
Another object of the present invention is to provide a device capable of detecting the amount of displacement using this fact.

That is, the second mobile vehicle environment recognition device according to the present invention relates to the projection and difference image relating to the first image or the second image instead of the means for detecting the presence / absence and amount of displacement in the first device. Based on the correlation with the projection, the presence or absence of a positional shift between the first image and the second image is detected, and based on the correlation between the projection on the first image and the projection on the second image, It is characterized in that means for detecting the amount are provided.

In the above, the case where the displacement in the y-axis direction has occurred between the first image and the second image has been described as an example. However, the presence or absence of the displacement in the x-axis direction can be determined by comparing the difference image with the y-projection. , Can be detected by examining the correlation of the first image or the second image with the y-projection. The amount of displacement in that case is the first
In the second device, the detection can be performed by examining the correlation between the x-projection of the difference image and the x-projection of the first image or the second image. In the second device, the first image and the second image can be detected.
It can be detected by examining the correlation of each x-projection with the image.

Further, if the detection of the presence / absence and the amount of the displacement in the x-axis direction and the detection of the presence / absence and the amount of the displacement in the y-axis direction as described above are performed in combination, the displacement in any direction can be determined. Even if it occurs, its direction and amount can be detected.

(Examples) Hereinafter, the present invention will be described in detail based on examples shown in the drawings.

FIG. 1 shows a basic configuration of a mobile vehicle environment recognition apparatus according to a first embodiment of the present invention. The device of this embodiment is applied, for example, for automatic driving of an automobile. As shown in the figure, a television signal V output from a television camera 10 for photographing an external environment of a car is input to an image input device 11 constituting image input means together with the television camera 10. The television camera 10 can be mounted on the roof 6 at a position near the windshield of the automobile 5 as shown in FIG. The glass may be fitted into a part and attached to the rear side. As the TV camera 10, for example, a CCD
A device provided with an image sensor such as described above is preferably used.

The image input device 11 converts the television signal V sent from the television camera 10 into A / D after performing preprocessing such as smoothing and binarization, and converts the digital image signal S into a digital image signal S for external environment recognition. Is output. This image signal S has, for example, 256 pixels.
× 256 or more, and the number of gradations is about 8 bits. Note that the photographing by the television camera 10 is performed every 1/30 second as an example.
Output as an indication of the image.

The image signal S is sent to the environment recognition means 12. The environment recognizing means 12 is composed of, for example, a known computer system, and processes the image signal S according to a predetermined algorithm, thereby extracting an edge, a region, and the like indicating a traveling road. Information J such as edges and regions extracted in this way is input to the local map generating means 13, where the information J
, A local map, that is, a map indicating the state of the outside world near the vehicle 5 is created.

The information K indicating the local map is determined by the optimum route determining means.
14, the optimal route to the destination is determined based on the information K and the information of the destination and the like input in advance, and the traveling direction and traveling speed for traveling the optimal route are determined. You. The information L on the traveling direction and the traveling speed is input to the controller 15, and the controller 15 controls the steering and the speed of the automobile 5 based on the information L.

On the other hand, the image signal S is input to the video switching circuit 16 in order to detect the relative movement of another moving body with respect to the automobile 5. The video switching circuit 16 outputs the image signal S indicating the external environment at a certain time t to the memory 17a as a first image signal S1 and then outputs the image signal S indicating the external environment at a time t + Δt after a predetermined time Δt to the second image signal 2 is input to the memory 17b and temporarily stored. At time t + 2Δt, the image signal S, which was the second image signal S2 at time t + Δt, is input to the memory 17a as the first image signal S1. Also, as mentioned above,
In the present embodiment, Δt = 1/30 seconds.

When the image captured by the television camera 10 is, for example, as shown in (1) of FIG. 2, the image signal S
The first image and the second image carried by S1 and S2 are as shown in (2) and (3) of FIG.

First image signals stored in the memories 17a and 17b
Both S1 and the second image signal S2 are read out every Δt time and input to the relative movement detecting means 18. The relative movement detecting means 18 matches the first image and the second image indicated by the image signals S1 and S2, or matches corresponding labeled regions in the first and second images, and calculates a mismatch measure between the two. Car 5 by request
The relative movement amount, movement speed, and the like of an object (for example, another automobile 9 shown in FIG. 2A) relative to the object are obtained. Note that the second image signal S2 is input to the relative movement detecting means 18 through the correcting means 19, which will be described later.

The information M indicating the relative movement amount, the movement speed, and the like of the moving object thus obtained is input to the above-described optimum route determination means 14. The optimal route determination means 14 determines the optimal traveling direction and speed using the behavior of the moving object indicated by the information M as a judgment factor in addition to the local map indicated by the information K described above.

Here, if there is a displacement between the first image and the second image due to the pitching or the like described above, the mismatch measure obtained by the relative movement detecting means 18 is as follows:
It does not correspond to the actual movement of the moving object. In other words, if this displacement occurs, the mismatch measure changes by itself.

Hereinafter, the point of preventing the above-described problem will be described. The first image signal S1 and the second image signal S2 read from the memories 17a and 17b pass through sign switches 29a and 20b, respectively.
It is also input to the difference detector 21. The difference detector 21 calculates a difference between the first image signal S1 and the second image signal S2 for each signal of a corresponding pixel. The difference image signal S3 thus obtained is input to the displacement calculator 22. In this embodiment, in particular, only the vertical image position shift due to the pitching of the automobile 5 is determined, and the position shift calculator 22 calculates the position shift in this direction. Note that the sign switches 20a and 20b are used to calculate each image signal for calculating the difference.
A predetermined code is given to S1 and S2 so that the difference can be detected by a single operation.

When the vehicle 5 is traveling on a general road, when the predetermined time Δt is about 1/10 to 1/30 seconds, the first image and the second image carried by the signals S1 and S2 respectively are those of the vehicle 5 If pitching does not occur, they are almost the same as shown in (2) and (3) of FIG. 2, while if pitching occurs, they are shown in (5) and (6) of FIG. Thus, they are shifted from each other in the vertical direction. Therefore, the difference image carried by the difference image signal S3 is as shown in FIG. 2 (4) in the normal case where the pitching has not occurred, and when the pitching has occurred, the difference image shown in FIG. It will be as shown.

In order to detect the positional deviation between the first image and the second image, the positional deviation computing unit 22 first sets the horizontal x on the image for each of the first image signal S1, the second image signal S2, and the difference image signal S3. Projection onto the axis (x-projection) and y perpendicular
Projection onto the axis (y-projection). Next, the displacement calculator 22 compares the x-projection Px (Δt) of the difference image signal S3 with the x-projection Px (t) of the first image signal S1, and for each position in the x-axis direction, the former is the latter. If the value is approximately twice as large as, it is considered that there is a position shift, while if the former has a value that is significantly smaller than the latter, it is considered that there is no position shift. The reason why the presence / absence of the displacement can be detected is as described in detail above.

When detecting the presence / absence of a positional shift as described above, instead of the x-projection Px (t) of the first image signal S1 as a comparison target with the x-projection Px (Δt) of the difference image signal S. , X-projection Px (t + Δt) of the second image signal S2 may be used.

When it is determined that there is no displacement as described above, the displacement calculator 22 outputs a displacement signal N indicating that there is no displacement.
Is sent to the correction means 19. When the correction means 19 receives such a displacement signal N, the correction means 19 does not perform any correction processing on the second image signal S2 and directly inputs the second image signal S2 to the relative movement detection means 18.

On the other hand, when it is determined that there is a position shift as described above, the position shift calculator 22 calculates the y-projection Py (Δ
Based on the correlation between t) and the y-projection Py (t) of the first image signal S1, the amount of this displacement is obtained. That is, for example, as shown in of FIG. 2 (5), when the region R ₁ that takes a specific large value over the width y ₁ in the first image signal S1 y- projection Py (t) is present, the difference image Y-projection Py of signal S3
In (Delta] t), as shown in FIG. 7, a region R ₁
The region R ₃ which has a specific large value corresponding to the width y ₃
, Δy = 2y ₁ −y ₃ is the displacement amount.

Also, one ends of the two regions R ₁ and R ₃ are aligned with each other,
Although the other ends will not be aligned, the direction of the displacement can also be known by checking which side the aligned end is located on. For example, FIG. 2 (5), (7)
In the example, when the end portion of the upper in the figure in the region R ₁ and R ₃ are aligned, the second image will be the image is shifted downward with respect to the first image.

The displacement calculator 22 sends a displacement signal N indicating the direction and amount of the displacement detected in this way to the correcting means 19. The correcting unit 19 corrects the second image signal S2 so as to eliminate the position shift indicated by the position shift signal N. This correction is performed in such a manner that the image carried by the second image signal S2 is translated in the y-axis direction. The correction means 19 has a built-in buffer memory for temporarily storing the second image signal S2, and when the displacement signal N is input, reads out the second image signal S2 from the buffer memory, and To make the above correction.

If the second image signal S2 is corrected as described above, the relative movement detecting means 18 does not affect the position of the second image and the second image in determining the mismatch measure.
Basically, a mismatch measure based on the relative position of each element in both images is obtained.

When detecting the direction and the amount of the displacement as described above, the correlation between the difference image signal S3 and the y-projection Py (Δt) is examined as the y-projection Py of the first image signal S1.
Instead of (t), the y-projection Py (t +
Δt) may be used.

Next, a second embodiment of the present invention will be described with reference to FIG. In FIG. 4, the same elements as those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted.

The position shift calculator 32 in the apparatus shown in FIG.
The presence / absence of a displacement is detected in the same manner as the displacement calculator 22 of the apparatus shown in the figure, but the detection of the displacement is determined by another process. That is, in addition to the difference image signal S3 and the first image signal S1, the second image signal S2 is also input to the displacement calculator 32. The displacement calculating unit 32, for example, y- and region R ₁ as described above in the projection Py (t) of the first image signal S1, the area y- in the projection Py (t + Δt) of the second image signal S2 corresponding thereto R ₂ (see FIG. 2 (6)), and the direction and amount of displacement between the first and second images are determined based on the direction and amount of displacement between the one ends.

In the two embodiments described above, the displacement of the first image and the second image in the y-axis direction, that is, in the vertical direction is detected, but as described above, the position in the x-axis direction or even in the oblique direction is detected. A shift can also be detected.

Further, in the above embodiment, the position shift correction is performed when matching the two images based on the detected position shift amount (including the direction). When the environment recognizing unit 12 recognizes the environment from each input image, it can be used to correct the positional shift of each image.

(Effects of the Invention) As described in detail above, the environment recognition device for a mobile vehicle according to the present invention is configured to directly detect a positional shift between two images from an input image signal.
There is no need to use sensors for detecting the behavior of the mobile vehicle such as shaking. Therefore, according to this device,
It is possible to always detect the image position deviation with high accuracy, irrespective of the problems such as the characteristic variation of the sensors and the change with time.

[Brief description of the drawings]

The first is a block diagram showing the basic configuration of the device of the first embodiment of the present invention. FIG. 2 is the first image, the second image and their difference images related to the present invention, the x-axis of each image, FIG. 3 is a schematic view showing an example of projection on the y-axis, FIG. 3 is a perspective view showing an example of a state in which a television camera constituting an image input means according to the present invention is mounted on a moving vehicle, and FIG. It is a block diagram which shows the basic structure of 2nd Example apparatus. Reference numeral 5: automobile, 9: other automobile 10: television camera 11, image input device 12: environment recognition means 16, video switching circuit 17a, 17b memory, 18 relative movement detection means 19: Correcting means, 20a, 20b: Sign switch 21: Difference detector, 22, 32: Position shift calculator

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-63-186308 (JP, A) JP-A-3-179598 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G05D 1/02 G06T 1/00 G08G 1/16

Claims (2)

(57) [Claims] 1. An environment recognizing device for a mobile vehicle having an image input means for recognizing the outside world, wherein, among images sequentially input by the image input means,
Means for obtaining a difference image between the first image at a specific time and the second image after a predetermined time from the first image, projecting each of the first image, the second image, and the difference image onto a substantially horizontal x-axis; Means for determining a projection on the y-axis perpendicular to the axis, and a displacement of the first image and the second image based on a correlation between the projection on the first image or the second image and the projection on the difference image. Means for detecting presence / absence and an amount of the environment. 2. An environment recognition apparatus for a mobile vehicle having image input means for external world recognition, wherein:
Means for obtaining a difference image between the first image at a specific time and the second image after a predetermined time from the first image, projecting each of the first image, the second image, and the difference image onto a substantially horizontal x-axis; Means for determining a projection on the y-axis perpendicular to the axis, and a displacement of the first image and the second image based on a correlation between the projection on the first image or the second image and the projection on the difference image. Means for detecting presence / absence and detecting the amount of the displacement based on a correlation between the projection on the first image and the projection on the second image. .