JP5776212B2 - Image processing apparatus, method, program, and recording medium - Google Patents

Description

  The present invention relates to a calibration technique for a stereo camera, and more particularly to an image processing apparatus, method, program, and recording medium that improve the ranging accuracy of a stereo camera.

  In recent years, stereo cameras have been used in various systems such as driving support systems and monitoring systems. A stereo camera is a device composed of a pair of monocular cameras, and uses the difference in the imaging position of the object to be imaged on each of the monocular cameras with different viewpoint positions, and the distance between objects. And position can be measured.

  Stereo cameras have external parameters such as the distance between each monocular camera (baseline length) and rotation deviation of each camera with respect to the vertical axis of the installation surface, and the position of the image formed by the lens built into each monocular camera (image Center), the distance between the lens and the image plane (focal length), and internal parameters unique to the camera, such as distortion caused by the lens. The calibration accuracy of these parameters affects the measurement accuracy of the stereo camera. give. For this reason, it is important to improve the calibration accuracy with respect to the parameters of the stereo camera.

  Patent Document 1 discloses a calibration method for adjusting optical distortion and positional deviation of a stereo camera by image processing. In this method, for a pair of image data output by imaging a chart having a predetermined pattern with a stereo camera, a correction parameter for correcting distortion of the image data and a calibration parameter for calibrating the deviation of the pair of images Is calculated.

  However, the calibration method disclosed in Patent Document 1 has a problem that the distance measurement accuracy of the stereo camera is low because the calibration accuracy of the parallax direction parameter that most affects the distance measurement accuracy is insufficient.

  The present invention has been made in view of the above-described problems of the prior art, and increases the calibration accuracy of parallax direction parameters that affect the ranging accuracy, and improves the ranging accuracy of the stereo camera. The object is to provide a program and a recording medium.

  In order to solve the above-described problem, the image processing apparatus of the present invention acquires a captured image pair captured by a stereo camera, parallelizes the captured image pair using conversion information, and searches for corresponding points. Then, using the corresponding points, the parallax value of the captured image pair is calculated, and when it is determined that the conversion information needs to be calibrated, the conversion information is calibrated to the conversion information that takes into account the rotational deviation between the cameras constituting the stereo camera. Thereby, the image processing apparatus of the present invention can calibrate the parallax direction parameter that most affects the ranging accuracy, that is, the rotational deviation with high accuracy, using the parallax values at different angles of view. The ranging accuracy of the stereo camera can be improved.

  In addition, the present invention enables a highly accurate calibration of the rotational deviation that most affects the ranging accuracy using parallax values at different angles of view, and can improve the ranging accuracy of the stereo camera, A program and a recording medium are provided.

1 is a diagram illustrating an image processing system according to an embodiment. 1 is a diagram illustrating a functional configuration of an image processing apparatus according to an embodiment. The figure which shows the rotation gap of the camera which comprises the stereo camera of this embodiment. 6 is a flowchart illustrating calibration processing executed by the image processing apparatus according to the present embodiment. The figure which shows embodiment of the picked-up image image | photographed with the stereo camera of this embodiment. The figure which shows the parallax value of the picked-up image pair image | photographed with the different imaging angles using the stereo camera of this embodiment.

  Hereinafter, although this invention is demonstrated with embodiment, this invention is not limited to embodiment mentioned later. FIG. 1 is a diagram illustrating an image processing system 100 according to the present embodiment. The image processing system 100 includes a target 110, a stereo camera 120, an angle adjustment device 130, an image processing device 140, and an output device 150.

  The target 110 is an object photographed by the stereo camera 120 and is an object whose distance from the stereo camera is measured. In the present embodiment, it is preferable to employ a target color scheme that easily detects parallax when searching for corresponding points of a captured image pair, which will be described later, for example, white and black. Moreover, in order to reduce the influence by the installation environment of the stereo camera 120, it is preferable to arrange the target 110 at a far position where the influence is small. Note that the target 110 is arranged at a distance that can be displayed on the captured image of the stereo camera 120.

  The stereo camera 120 is a camera device composed of a pair of cameras, and can photograph the same object simultaneously using two cameras. The stereo camera 120 provides a pair of captured images of the target 110 captured by each camera to the image processing device 140.

  The angle adjustment device 130 is a device that rotates horizontally with the installation surface and adjusts the shooting angle of the stereo camera 120. A stereo camera 120 is placed on the upper surface of the angle adjusting device 130, and the stereo camera 120 is rotated by rotating the angle adjusting device 130, so that the photographing angle can be adjusted. In the present embodiment, the angle adjustment device 130 is manually rotated. However, in other embodiments, a stage controller may be connected to the adjustment device to control the rotation operation. In still another embodiment, the angle adjusting device 130 and the stage controller may be integrated to employ the device.

  The image processing apparatus 140 is an image processing apparatus that performs calibration using a captured image of the stereo camera 120 that is a calibration target. In the present embodiment, the image processing apparatus 140 may employ an image processing apparatus such as a notebook computer or a desktop computer.

  The image processing apparatus 140 includes a processor such as a PENTIUM (registered trademark) processor or a compatible processor, and an OS such as a WINDOWS (registered trademark) series, a MAC (registered trademark) OS, a UNIX (registered trademark), or a LINUX (registered trademark). Under the control of the above, the program of this embodiment described in a programming language such as assembler, C, C ++, Java (registered trademark), JavaScript (registered trademark), PERL, RUBY, or PYTHON is executed.

  In addition, the image processing apparatus 140 includes a RAM that provides an execution space for executing a program, a hard disk device that continuously stores programs and data, and the like. The functional means is realized on the image processing apparatus by executing the program. The program of the present invention can be distributed by being stored in a device-readable recording medium such as HDD, CD-ROM, MO, flexible disk, EEPROM, EPROM, etc., and in a format readable by other devices via a network. Can be transmitted.

  The output device 150 is a device that displays image data and information processed by the image processing device 140. The output device 150 is connected to the image processing device 140 and displays image data transferred from the image processing device 140, various information, and the like. In the present embodiment, various display devices such as a liquid crystal display and an organic EL display can be employed as the output device 150.

  FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. The image processing device 140 includes a control unit 200, a distance acquisition unit 202, an image acquisition unit 204, a storage device 206, and an image buffer 208.

  The control unit 200 is a functional unit that performs overall control of the image processing apparatus 140, and executes various processes by appropriately calling each functional unit described later.

  The distance acquisition unit 202 is a means for acquiring a target distance that is an actual distance from the stereo camera 120 to the target 110. In the present embodiment, the distance acquisition unit 202 stores a numerical value designated by the user as a target distance in the storage device 206 via a user interface that can be specified by the display distance displayed on the output device 150 by the display control unit 224 described later. save.

  The image acquisition unit 204 is a means for acquiring an image captured by the stereo camera 120 (hereinafter referred to as “captured image”). The image acquisition unit 204 stores the captured image pair acquired from the stereo camera 120 in the image buffer 208. The image buffer 208 stores a plurality of photographed image pairs photographed at different photographing angles.

  In addition, the image processing device 140 includes a correction unit 209, a parallelization unit 210, a corresponding point search unit 212, a parallax calculation unit 214, a calibration unit 216, and storage devices 218 and 220.

  The correction unit 209 is a functional unit that performs image correction on one or a plurality of captured image pairs acquired from the stereo camera 120. The correction unit 209 corrects internal parameters unique to the camera, such as the position of the image formed by the lens included in the monocular camera constituting the stereo camera 120, the focal length, and distortion caused by the lens. For example, as shown in Non-Patent Document 1, the correction unit 209 performs an optimization process using a difference between a calibration reference point captured by a stereo camera and a calibration reference point calculated by a camera model on a computer as an evaluation function. Thus, the optimized parameter can be calculated and calibrated.

  The parallelizing unit 210 is a functional unit that parallelizes the captured image pair of the stereo camera 120. The parallelizing unit 210 performs perspective transformation using a perspective transformation matrix that is transformation information stored in the storage device 218 in advance, and rotates and / or translates the captured image pair to parallelize the captured images. To do. In the present embodiment, it is possible to employ a perspective transformation matrix M represented by a homogeneous coordinate system as shown in Equation 1 below.

Here, (s ₀ , t ₀ ) indicates position coordinates before perspective transformation, and (s ₁ , t ₁ ) indicates position coordinates after perspective transformation. S indicates the value of s ₀ after matrix transformation, and T indicates the value of t ₀ after matrix transformation. A shows the coordinate showing a magnification. Refer to Non-Patent Document 2 for details of the perspective transformation matrix M.

  The corresponding point search unit 212 is a functional unit that specifies a corresponding point by performing a corresponding point search (stereo matching) on the parallelized captured image pair. The corresponding point search unit 212 can specify the corresponding points using various methods such as a correlation method based on a sum of absolute differences, a phase-only correlation method, and block matching.

  The parallax calculation unit 214 is means for calculating the parallax value of the captured image pair. The parallax value is a value indicating the degree of parallax of the captured image pair, and the parallax calculation unit 214 obtains the difference between the position coordinates of the corresponding points of the captured image pair identified by the corresponding point search, thereby obtaining the corresponding points. Can be calculated. The parallax calculation unit 214 stores the calculated parallax value in the storage device 220. The storage device 220 stores parallax values for each pair of captured images with different shooting angles.

  The calibration unit 216 is a functional unit that corrects and calibrates conversion information used when a captured image pair is parallelized. The calibration unit 216 uses the parallax values of a plurality of photographed image pairs with different photographing angles acquired from the stereo camera 120 to calculate a rotational deviation amount with respect to the vertical axis of the installation surface of each camera constituting the stereo camera 120. Specifically, the calibration unit 216 calculates the amount of deviation of the parallax values of the plurality of captured image pairs, and calculates the amount of rotational deviation with respect to the vertical axis using Equation 2 below. In the present embodiment, the difference between the maximum value and the minimum value of the parallax values of the plurality of photographed image pairs is set as a deviation amount of the parallax values of the photographed image pairs.

  Here, reference is made to FIG. 3 showing the rotational deviation of the cameras constituting the stereo camera 120. FIG. 3 shows a state in which the right camera constituting the stereo camera 120 is rotationally shifted with respect to the left camera.

In the present embodiment, the rotational deviation amount of the stereo camera 120 is θy_s. Also, let f be the focal length between the center point of the virtual imaging plane 300, 302 of the right camera and the center point of the lens 310. Further, the position coordinates of the target on the virtual imaging plane 302 parallelized with the conversion information before calibration are (i ₀ , j ₀ ), and the position coordinates of the target on the virtual imaging plane 304 parallelized with the conversion information after calibration are used. Is (i, j). Further, an angle formed by a straight line connecting the center point of the virtual imaging plane 300 and the center point of the lens 310 and a straight line connecting the position coordinates of the target imaged on the imaging plane 300 from the center point of the lens 310 is defined. Let φ. The following formula 2 is derived from the relationship shown in FIG.

Here, Δd indicates a shift amount of the parallax value of the captured image pair.

  The calibration unit 216 calculates a perspective transformation matrix M that takes into account the amount of camera rotation deviation calculated using Equation 2, and stores it in the storage device 218. Refer to Non-Patent Document 2 for a method of calculating the perspective transformation matrix M using the rotational deviation amount.

  Further, the image processing apparatus 140 illustrated in FIG. 2 includes a distance calculation unit 222 and a display control unit 224.

  The distance calculation unit 222 is a functional unit that calculates the distance between the target 110 and the stereo camera 120. The distance calculation unit 222 uses the parallax value of the captured image pair, the distance between the cameras of the stereo camera 120 (baseline length), and the focal length between the camera lens and the imaging plane to determine the target 110 and the stereo camera 120. Calculate the distance.

  The display control unit 224 is a functional unit that controls the display screen of the output device 150. In addition to the above-described user interface that can specify the target distance, the display control unit 224 includes a user interface for acquiring a captured image from the stereo camera 120, a captured image, feature points and parallax values of the captured image, conversion information, and the like. A user interface capable of displaying information processed by the image processing device 140 can be displayed on the output device 150.

  FIG. 4 is a flowchart showing a calibration process executed by the image processing apparatus of this embodiment. The conversion information calibration process executed by the image processing apparatus 140 will be described below with reference to FIG.

  The process of FIG. 4 starts when the control unit 200 of the image processing apparatus 140 receives an instruction of a captured image via a user interface for acquiring a captured image in step S400. In step S <b> 401, the control unit 200 calls the image acquisition unit 204, and the image acquisition unit 204 acquires a captured image pair from the stereo camera 120 and stores it in the image buffer 208.

  In step S <b> 402, the control unit 200 calls the correction unit 209, and the correction unit 209 performs image correction on one or a plurality of captured image pairs acquired from the stereo camera 120. In step S <b> 403, the control unit 200 calls the parallelizing unit 210, and the parallelizing unit 210 performs a perspective transformation process on one or a plurality of captured image pairs that have undergone image correction to parallelize the pair. In step S <b> 404, the corresponding point search unit 212 specifies a corresponding point by stereo matching the parallelized captured image pair. In step S405, the parallax calculation unit 214 calculates the parallax value of the captured image pair using the corresponding points.

  In step S406, the control unit 200 determines whether calibration is necessary for the conversion information used in the parallelization process in step S403.

  In the present embodiment, it is possible to determine whether or not calibration is necessary by comparing the parallax values of captured image pairs having different imaging angles. In this embodiment, it is determined that calibration is not necessary when the parallax values of these captured image pairs match. Alternatively, a threshold may be set, and it may be determined that calibration is not necessary when the parallax values of these captured image pairs substantially match.

  In another embodiment, the control unit 200 causes the distance calculation unit 222 to calculate the distance from the stereo camera 120 to the target 110 using the parallax values of one or a plurality of captured image pairs. Then, the control unit 200 can determine whether calibration is necessary by comparing the distance with the actual target distance stored in the storage device 206. In the present embodiment, if the calculated distance matches the actual target distance, it is determined that calibration is not necessary. Alternatively, a threshold may be set, and it may be determined that calibration is not required when the calculated distance and the actual target distance substantially match.

  If it is determined in step S406 that calibration is not required (no), the process branches to step S409 and ends. On the other hand, if it is determined that calibration is necessary (yes), the process branches to step S407.

  In step S <b> 407, the control unit 200 calls the calibration unit 216, and the calibration unit 216 calculates the amount of rotational deviation with respect to the vertical axis of the cameras constituting the stereo camera 120. In step S408, the calibration unit 216 corrects the conversion information using the rotational deviation amount calculated in step S407, stores it in the storage device 218, and the process ends in step S409.

  The image processing apparatus 140 of the present invention corrects the conversion information by the calibration process using the captured image of the target. Then, the photographed image pair is made parallel using the corrected conversion information, the corresponding point is identified using the parallelized photographed image pair, the parallax value is calculated, and the distance to the target is calculated. be able to.

  FIG. 5 is a diagram illustrating an embodiment of a captured image captured by the stereo camera of the present embodiment. The captured image 510 is an image captured with the stereo camera 120 in the state indicated by 514 in FIG. 5, that is, the state in which the stereo camera 120 is directly facing the target and the capturing angle is “0 °”, and the center The target is reflected in the department. The photographed image 512 is an image photographed by the stereo camera 120 in the state indicated by 516 in FIG. 5, that is, in a state where the photographing angle is “15 °”, and the target is shown on the right side thereof. In the captured images 510 and 520, the targets are shown at different angles of view.

  FIG. 6 shows a parallax value obtained by correcting the parallax value at each angle of view of a pair of captured images taken at different shooting angles by the stereo camera of the present embodiment according to the magnitude of the shooting angle. FIG. When the accuracy of this correction is low, the corrected parallax value varies depending on the shooting angle as shown in FIG. The parallax value can be derived using Equation 3 below.

Here, d indicates the parallax value when the stereo camera is directly facing the distance measuring target, and d ′ indicates the parallax value when the stereo camera is not directly facing the distance measuring target. θ is the straight line connecting the stereo camera and the ranging target when the stereo camera is directly facing the ranging target, and the rotation center of the stereo camera when the stereo camera is not facing the ranging target. And an angle formed by a line perpendicular to the base line of the stereo camera.

  In the embodiment illustrated in FIG. 6, the parallax values at the positions of the field angles of the captured image pair corresponding to the imaging angles of “−16 °” to “16 °” are represented. It should be noted that since the shooting angle corresponds to the angle of view that can be shot with a monocular camera constituting a stereo camera, the shooting angle is not limited to these in other embodiments.

  The sample point 610 indicates the parallax value at the shooting angle “−16 °”, that is, the parallax value of the shot image pair in which the target is captured at the position of the field angle corresponding to the end of the shot image. The sample point 612 is the parallax value at the shooting angle “0 °”, that is, the parallax of the shot image pair in which the target is captured at the position of the angle of view corresponding to the center of the shot image, as in the shot image 510 of FIG. The value is shown.

  In the embodiment shown in FIG. 6, the parallax value at the position of the view angle corresponding to the shooting angle “0 °” at which the parallax value is maximum, and the view angle corresponding to the shooting angle “−16 °” at which the parallax value is minimum. The shift amount (Δd) of the parallax value can be calculated using the parallax value at the position.

  Although the present embodiment has been described so far, the present invention is not limited to the above-described embodiment, and those skilled in the art can change or delete the configuration requirements of the above-described embodiment, add other configuration requirements, and the like. It can be changed within the range that can be conceived, and any aspect is included in the scope of the present invention as long as the effects and effects of the present invention are exhibited.

  DESCRIPTION OF SYMBOLS 100 ... Image processing system, 110 ... Target, 120 ... Stereo camera, 130 ... Angle adjustment apparatus, 140 ... Image processing apparatus, 150 ... Output device

Japanese Patent No. 4109077 Z. Zhang, “A Flexible New Technique for Camera Calibration”, Technical Report MSR-TR-98-71, Microsoft Research, 1998 Koichiro Deguchi, “Image and Space Geometry of Computer Vision”, Shosodo, 1991

Claims (12)

An image processing apparatus that processes a captured image of a stereo camera, wherein the image processing apparatus includes:
Means for acquiring a pair of captured images of a subject to be photographed by the stereo camera;
Parallelizing means for parallelizing the captured image pair using conversion information for translating and / or rotating the captured image pair;
Corresponding point search means for searching for corresponding points of the parallelized captured image pair;
Parallax calculating means for calculating a parallax value of the captured image pair using the corresponding points;
Means for determining whether calibration of the conversion information is necessary;
Calibration means for calibrating the conversion information when it is determined that calibration of the conversion information is necessary, and
The calibration means includes a straight line connecting the center point of the virtual imaging plane of the camera constituting the stereo camera to the center point of the lens and the position coordinates of the target imaged on the virtual imaging plane from the center point of the lens. Rotation of the camera using an angle formed by a straight line connecting the lines, a shift amount of parallax values of a plurality of captured image pairs, and a position coordinate of the target on the virtual imaging plane parallelized by the conversion information An image processing apparatus that calculates a shift and calibrates the conversion information in consideration of the rotation shift. The calibration means calculates the angle using a focal length between a center point of the virtual imaging plane and a center point of the lens, and a position coordinate of the target on the virtual imaging plane parallelized by the conversion information. calculated, it calculates the difference between the maximum value and the minimum value of the disparity values of the plurality of captured image pairs as the shift amount, an image processing apparatus according to claim 1.   The determination unit compares parallax values of a pair of photographed images with different photographing angles, and determines that the conversion information needs to be calibrated when the parallax values of the plurality of photographed image pairs are different. 2. The image processing apparatus according to 2. The image processing apparatus includes:
A distance calculating means for calculating a distance from the stereo camera to the object to be photographed using the parallax value;
The determining means compares the distance calculated by the distance calculating means from the parallax values of one or a plurality of photographed image pairs with the actual distance to the photographing object stored in a storage device, and calculates the distance. The image processing apparatus according to claim 1, wherein when the distance is different from the actual distance, it is determined that the conversion information needs to be calibrated. The image processing apparatus includes:
A correction means for correcting the captured image pair;
The image processing apparatus according to claim 1, wherein the parallelizing unit parallelizes the captured image pair corrected by the correcting unit. A method executed by an image processing apparatus that processes a captured image of a stereo camera, wherein the image processing apparatus includes:
Obtaining a photographed image pair of a photographing object photographed by the stereo camera;
Parallelizing the captured image pair using transformation information that translates and / or rotates the captured image pair;
Searching for corresponding points of the parallelized captured image pair;
Calculating a parallax value of the captured image pair using the corresponding points;
Determining whether calibration of the conversion information is necessary;
Calibrating the conversion information when it is determined that calibration of the conversion information is necessary, and
The step of calibrating comprises:
A straight line connecting the center point of the virtual imaging plane of the camera constituting the stereo camera to the center point of the lens and a straight line connecting the position coordinates of the target imaged on the virtual imaging plane from the center point of the lens Calculating a rotational shift of the camera using an angle formed, a shift amount of a parallax value between a plurality of captured image pairs, and a position coordinate of the target on the virtual imaging plane parallelized by the conversion information. When,
Calibrating the conversion information in consideration of the rotational deviation. The step of calibrating comprises:
Calculating the angle using the focal length of the center point of the virtual imaging plane and the center point of the lens, and the position coordinates of the target on the virtual imaging plane parallelized by the conversion information ;
The method according to the steps of calculating a difference between the maximum value and the minimum value of the disparity values of the plurality of captured image pairs as the shift amount including, in claim 6.   The step of determining includes a step of comparing parallax values of captured image pairs with different shooting angles and determining that the conversion information needs to be calibrated when the parallax values of the plurality of captured image pairs are different. Item 8. The method according to Item 6 or 7. In the method, the image processing apparatus includes:
Further comprising calculating a distance from the stereo camera to the object to be photographed using the parallax value;
The determining step compares the distance calculated from the parallax value of one or a plurality of photographed image pairs in the step of calculating the distance with an actual distance to the photographing object stored in a storage device, The method according to claim 6, further comprising the step of determining that the conversion information needs to be calibrated when the calculated distance is different from the actual distance. In the method, the image processing apparatus includes:
Further comprising the step of correcting the captured image pair;
The method according to any one of claims 6 to 9, wherein the parallelizing step parallelizes the captured image pair corrected in the correcting step.   The computer-executable program for an image processing apparatus to perform each step of any one of Claims 6-10.   A computer-readable recording medium on which the program according to claim 11 is recorded.