JP4469757B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for calculating three-dimensional position information of an object by performing image processing on images taken at different frame times using a plurality of cameras.

  Various researches and developments have been made on visual processing systems for mobile robots that follow people. When the mobile robot moves to follow a moving body such as a person, the visual processing system first images the moving body and measures its three-dimensional position by image processing. According to this measurement result, the robot calculates a control amount of the drive system for movement.

  Conventionally, there are known human detection techniques based on background differences used in surveillance systems and the like, and techniques for recognizing a person's head through image processing, and these techniques can be applied to the visual processing system of a mobile robot. Conceivable. However, since the background image changes greatly due to the movement of the robot, the background subtraction method is not suitable for the visual processing system of the mobile robot. In addition, since the head to be tracked is not always captured in the field of view of the robot due to the positional relationship with the person or due to shielding, the human head recognition technology is not effective. Considering that a mobile robot is active in a human living space, it is necessary to reduce the data to be processed and process it at high speed.

Here, a technique is known in which three-dimensional position information of an object is calculated at high speed by performing image processing on images taken at different frame times (shutter timings) using a plurality of cameras (for example, the following non-patent documents). Reference 1).
Shoichi Shimizu and Hironobu Fujiyoshi, “High-speed three-dimensional position estimation using shift in shutter timing between cameras”, Proc. of MIRU 2004, vol. 1, pp. 428-433, July, 2004.

  In a conventional configuration in which a moving object is imaged with a plurality of cameras in synchronism with the imaging time, and the three-dimensional position information of the moving object is obtained by performing stereo image processing on a pair of images obtained at the same frame time. There is a problem in that the throughput of image processing is limited depending on the frame rate of the camera that captures the moving object. In a system that uses a processing result as input information for a control system, the throughput of image processing is an important factor that determines the reaction performance with respect to the movement of the object.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of obtaining three-dimensional position information of a moving body with high throughput without being limited by the throughput of an imaging camera.

  An image processing apparatus according to an aspect of the present invention includes: a first imaging unit that images a moving body; a second imaging unit that images the moving body from a different viewpoint from the first imaging unit; A first image picked up by the first image pickup means at the frame time, a second image picked up by the second image pickup means at a second frame time after the first frame time, and the first image An image acquisition unit that acquires a third image captured by the first imaging unit at a third frame time after the second frame time, and the first image and the second image in stereo. The first three-dimensional position information of the moving object is calculated by image processing, and the second three-dimensional position information of the moving object is processed by stereo image processing of the second image and the third image. Stereo image processing means for calculating Image processing apparatus comprising a calculating means for calculating the estimated three-dimensional position information of the movable body from a first three-dimensional position information and the second three-dimensional position information.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which can obtain the three-dimensional position information of a moving body with a high throughput without being restrict | limited to the throughput of an imaging camera can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an image processing apparatus (visual processing system) according to an embodiment of the present invention. In this embodiment, the present invention is applied to a robot 1 as shown in FIG. The robot 1 includes a plurality of imaging devices (cameras) 101a to 101b, an arm 3, wheels 4 and the like, and these are controlled by a control unit (not shown). When such a robot 1 performs a task such as grasping an object with the arm 3 or continuing to chase a specific person, the robot 1 needs to obtain three-dimensional position information of these moving bodies (objects). . In the present embodiment, the generation of the three-dimensional position information is realized by a relatively simple device configuration and image processing.

  In implementing the present invention, a plurality of imaging devices are required. In the configuration of FIGS. 1 and 2, two imaging devices 101a to 101b are used. It is assumed that the positional relationship between the two imaging devices is known. The imaging apparatus 101b images the moving body from different viewpoints with respect to the imaging apparatus 101a that images the moving body.

  The video signal switching unit 102 alternately switches video signals output from each of the two imaging devices 101 a to 101 b and supplies the video signals to the image acquisition unit 103. The image acquisition unit 103 receives the video signal from the video signal switching unit 102 and converts it into digital image data that can be easily processed by a computer. Thereby, a plurality of images of the object having different frame times are taken into the image processing apparatus of the present embodiment.

  Specifically, the image data of the first image captured by the imaging device 101a at the first frame time in the first imaging cycle and the second imaging having a time lag with respect to the first imaging cycle. Image data of the second image captured by the imaging device 101b at the second frame time in the cycle, and imaging by the imaging device 101a at the third frame time after the second frame time in the first imaging cycle. The image data of the third image thus obtained is continuously acquired.

  The image recording unit 104 temporarily records image data obtained via the image acquisition unit 103. The image recording unit 104 performs a recording and holding operation so that at least one frame of previous image data captured by each of the imaging devices 101a to 101b can be referred to.

  The stereo image processing units 105 and 106 calculate the three-dimensional position information of the moving object by performing stereo image processing on the image data obtained via the image acquisition unit 103. In stereo image processing, the parallax between images taken by the two imaging devices 101a to 101b is measured. In parallax measurement, a corresponding point search between images is performed by block matching processing.

  The estimated three-dimensional position information calculation unit 107 calculates the estimated three-dimensional position information of the moving object based on the three-dimensional position information of the moving object calculated by the stereo image processing units 105 and 106, respectively. In this calculation, the first distance represented by one of the three-dimensional position information calculated by the stereo image processing unit 105 and the second distance represented by the other three-dimensional position information calculated by the stereo image processing unit 106. The average value is used as the estimated three-dimensional position information of the moving body. The control unit 109 obtains the control amount of the robot 1 from the estimated three-dimensional position information of the moving object obtained by the estimated three-dimensional position information calculation unit 108.

  In FIG. 1, the components excluding the two imaging devices 101a to 101b and the video signal switching unit 102 are realized as a computer 110 as shown in FIG.

  FIG. 3 is a conceptual diagram of asynchronous stereo image processing according to an embodiment of the present invention.

  For example, when two imaging devices 101a to 101b are provided as in the present embodiment, image capturing is performed so that a time shift of a half cycle (1/2 cycle) occurs between the imaging devices as described above. In the asynchronous image sequence thus captured, a set of the image f1 at the frame time t of the imaging device 101a and the image f2 at the frame time t + 0.5Δt of the imaging device 101b, and the image f2 and the imaging at the frame time t + 0.5Δt of the imaging device 101b. A set of the image f3 at the frame time t + Δt of the device 101a, a set of the image f3 at the frame time t + Δt of the imaging device 101a and the image f4 at the time t + 1.5Δt of the imaging device 101b. . . Thus, stereo image processing is performed by combining pairs of images with different frame times.

  Then, the results D1, D2, D3... Of the three-dimensional coordinates obtained by stereo viewing in the order from the imaging device 101a to the imaging device 101b. . . Is farther than the true value obtained in the case of synchronous shooting, and results D4, D5,... In the order from the imaging device 101b to the imaging device 101a. . Becomes closer (or the result of reversing the perspective is determined by the positional relationship between the imaging devices and the moving direction of the object).

  Therefore, two consecutive stereo results (for example, D1 and D4, D4 and D2, D2 and D5) are averaged to estimate the three-dimensional coordinates E1, E2, E3, E4. . . And Thereby, when there are two imaging devices, stereo image processing at a substantially double frame rate is possible. Note that the difference between the estimated trajectory of the moving object and the actual trajectory is exaggerated in FIG.

  FIG. 4 is a flowchart showing an asynchronous stereo image processing procedure according to an embodiment of the present invention. Here, a procedure for calculating one estimated three-dimensional position information from the captured asynchronous image series will be described.

  First, in ST1, image data is acquired from the imaging devices 101a to 101b via the video signal switching unit 102. The video signal switching unit 102 captures image data while alternately switching the imaging devices one by one. Since a certain time is required for switching the video signal, there is a “deviation” between the shooting times (frame times) of the two image data obtained from the two imaging devices 101a to 101b. It is assumed that the time required for signal switching is always constant. As a result, images with frame times t, t + 0.5Δt, and t + Δt are acquired.

  FIG. 5 is a diagram illustrating the shooting timing (time) of the imaging devices 101a and 101b. As shown in FIG. 5, two images having a time shift are used as stereo image pairs to perform stereo image processing. In general stereo image processing, images captured by a plurality of imaging devices at the same time are used. It is premised on using data, and if there is a difference in the photographing time, an error occurs, and a correct stereo image processing result cannot be obtained. Therefore, in the embodiment of the present invention, this error is reduced so that stereo image processing can be appropriately performed even when image pairs having a time lag are used.

  In ST2 of FIG. 4, the obtained image data is temporarily stored by the image recording unit 104. Here, at least one image data before the latest image is stored. Image data before that is sequentially deleted.

  In subsequent ST3, the parallax is measured between a pair (first image pair) of the image captured by the imaging device 101a at the frame time t and the image captured by the imaging device 101b at the frame time t + 0.5Δt. Specifically, the parallax is measured as follows.

  FIG. 6 shows the image data 501 at the frame time t and the image data 502 at the frame time t + 0.5Δt. Now, when obtaining the parallax of the point p in the image data 501, a small region w1 centered on p is created. Then, block matching is performed on the image data 502, and a small region having the highest similarity is searched. The search range is within a certain range from the coordinates of the point p, and the similarity is the sum of the square errors of the pixel values in the block. In the case of FIG. 6, the small area w2 is most similar to the small area w1. The difference in coordinates between the area obtained by this search and the original area corresponds to parallax. Such parallax measurement is performed at all points where three-dimensional position information is desired and recorded as parallax information.

In ST4, the stereo image processing unit 105 calculates first three-dimensional position information from the parallax information. The method of obtaining corresponding points between images taken with different imaging devices and acquiring 3D position information is described in the reference (by Xugang and Saburo Tsubasa, “3D Vision”, Chapter 7 Binocular Vision). However, if the optical characteristics of the imaging device and the positional relationship between the imaging devices are known, a point on the image 902 of the imaging device 101a as shown in FIG. From p ″ and the point p ′ on the image 901 of the image pickup apparatus 101b, the three-dimensional position p where the site indicated by these originally exists can be obtained.
Next, in ST5, in the same manner as in ST3, a pair of an image captured by the imaging device 101b at the frame time t + 0.5Δt and an image captured by the imaging device 101a at the frame time t + Δt ( The parallax is measured between the second image pair). In subsequent ST6, the same calculation as in ST4 is performed to calculate the second three-dimensional position information.

  FIG. 8 shows shooting timings of the two imaging devices 101a and 101b. As a result of ST3 to ST6, the stereo image processing results (first three-dimensional position information) of two images respectively captured at time t and time t + 0.5Δt surrounded by one dotted line, and the other dotted line Stereo image processing results (second three-dimensional position information) of two images captured at time t + 0.5Δt and time t + Δt, respectively, are obtained.

  In ST7, estimated three-dimensional position information (corresponding to, for example, E1 in FIG. 3) is calculated from the first three-dimensional position information and the second three-dimensional position information. Specifically, the first three-dimensional position information represents one distance (for example, D1 in FIG. 3) from the imaging device to the moving body, and the second three-dimensional position information represents the distance from the imaging device to the moving body. The other distance (for example, D4 in FIG. 3) is shown. The estimated three-dimensional position information calculation unit 107 calculates an average value of the first distance and the second distance as estimated three-dimensional position information of the moving object.

  In ST8, the control unit 109 calculates the following control amount for the robot 1 based on the obtained estimated three-dimensional position information. For example, the rotation amount of the wheel is determined so as to keep the distance to the target to be tracked constant, and the joint is moved to grasp the target object. Alternatively, the rotation amount and the rotation speed of the movement drive control such as the servo motor are determined so that the three-dimensional distance from the imaging device to the object approaches the target.

  Finally, in the determination process of ST9, it is determined whether or not to continuously measure the three-dimensional position information. When the process is continued, the process returns to ST1 to acquire image data.

  According to the embodiment described above, the result of two consecutive stereo visions is averaged to obtain an estimated value of three-dimensional coordinates. Thereby, when there are two imaging devices, stereo image processing at a substantially double frame rate is possible. In the case of the present embodiment, since three-dimensional position information can be obtained only from an image taken by one imaging device and images taken by other imaging devices before and after that, no error is accumulated. Further, it can be easily realized by slightly changing the conventional stereo vision framework. Further, the present embodiment is also suitable for a hardware configuration that has poor image processing performance.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  For example, in the above-described embodiment, the number of image pickup devices is two, but the number of image pickup devices—one stored three-dimensional position information and the number of image pickup devices is n even when the number of image pickup devices is two or more. By obtaining an average with the latest three-dimensional position information, an equivalent effect can be obtained.

  In the above-described embodiment, the shift in the image capturing time between the image capturing apparatuses is set to 0.5 Δt. However, for any time shift, an equivalent result can be obtained by obtaining a weighted average according to the capturing time. It is possible to obtain. For example, when the difference in imaging time is 0.25Δt, the image captured by the imaging device 101a at the frame time t (first frame time) and the imaging device 101b at the frame time t + 0.25Δt (second frame time). The above-described processing is performed using the image captured by the image capturing apparatus 101a and the image captured by the image capturing apparatus 101a at the frame time t + Δt (third frame time). The weighting value corresponding to the ratio (1/4: 3/4) of the time from the first frame time to the second frame time and the time from the second frame time to the third frame time is the first distance. By multiplying and adding to the second distance, the same result as described above can be obtained.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. The figure which shows the external appearance of the robot which concerns on one Embodiment of this invention. 1 is a conceptual diagram of asynchronous stereo image processing according to an embodiment of the present invention. The flowchart which shows the asynchronous stereo image processing procedure which concerns on one Embodiment of this invention. The figure which shows the imaging timing (time) by two imaging devices The figure which shows the newest image data and the image data of 1 frame before obtained with the same imaging device A diagram showing a two-dimensional image pair and three-dimensional position information The figure which shows each imaging | photography timing of the two imaging devices which concern on one Embodiment of this invention.

Explanation of symbols

101a-101b ... imaging device;
102 ... Video signal switching unit;
103 ... Image acquisition unit;
104: Image recording unit;
105, 106 ... stereo image processing unit;
107: Estimated three-dimensional position information calculation unit;
109 ... control unit;
110: Computer;

Claims (4)

First imaging means for imaging a moving body;
Second imaging means for imaging the moving body from a different viewpoint from the first imaging means;
A first image picked up by the first image pickup means at a first frame time and a second image picked up by the second image pickup means at a second frame time after the first frame time; Image acquisition means for respectively acquiring a third image captured by the first imaging means at a third frame time after the second frame time;
The first parallax measured by searching the first corresponding point on the second image corresponding to an arbitrary point on the first image, to the moving object based on said first parallax Second parallax by calculating first three-dimensional position information indicating a first distance and searching for a second corresponding point on the third image corresponding to an arbitrary point on the second image was measured, and the stereo image processing means for calculating a second three-dimensional position information indicating the second distance to the moving object based on said second parallax,
An image processing apparatus comprising: a calculation unit that calculates estimated three-dimensional position information of the moving object from the first three-dimensional position information and the second three-dimensional position information.   The image acquisition means uses, as the second image, an image captured by the second imaging means at a second frame time corresponding to an intermediate time between the first frame time and the third frame time. The image processing apparatus according to claim 1, which is acquired. Before SL calculating means, the image processing apparatus according to claim 1 or 2, wherein an average value of said second distance and said first distance as an estimated three-dimensional position information of the movable body. Pre Symbol image processing apparatus according to any one of claims 1 to 3 each of the first and second corresponding point is searched by block matching.

Images