JP5087264B2 - Image processing apparatus and method - Google Patents

Description

  The present invention relates to an image processing apparatus and a method thereof. Specifically, the present invention relates to an image processing apparatus and method for tracking a dynamic image when an imaging apparatus moves relative to an object and measuring coordinates of the imaging apparatus or the object.

  There has been a technique for measuring the position of an imaging apparatus by continuously imaging an object while performing relative movement. However, when shooting is actually performed with a shooting device mounted on a moving body such as an automobile, a stable image is not necessarily obtained due to the rocking of the automobile, and vertical movement, inclination, etc. are corrected for each frame. In addition, for example, another car, an asuka, a fallen leaf, or the like may enter between the photographing apparatus and the object, and the feature point may be hidden behind and disappear or may be restored again. Therefore, it is necessary to perform such processing of swinging of the photographing apparatus and processing of feature points that disappear and are reproduced. On the other hand, when a stationary object is imaged with a stationary imaging device, a three-dimensional measurement technique that automatically searches for and measures corresponding points with high accuracy is disclosed. (See Patent Document 1)

Japanese Patent Application No. 2002-64094 (paragraphs 0018 to 0073, FIGS. 1 to 11, etc.)

  Therefore, it is necessary to develop and apply the technique for capturing a stationary object with a stationary imaging device when either of them moves, to process the oscillation of the imaging device, or to process the feature points that disappear or reappear. Even in such a case, it is necessary to provide a technique capable of accurately measuring the shooting position, posture, or coordinates of an object of the shooting device from a moving image or a shot image that changes little by little. .

  The present invention provides a shooting position of a photographing device from a moving image or a photographed image that changes little by little, even in a case where processing of swinging of the photographing device and processing of feature points to disappear / reproduce are required. An object of the present invention is to provide a technique capable of accurately measuring the posture or the coordinates of an object.

In order to solve the above-described problem, the image processing apparatus 100 according to the first aspect of the present invention sequentially acquires captured images that change little by little with respect to a relatively moving object, for example, as illustrated in FIG. The captured image acquisition unit 2, the feature extraction unit 3 that extracts feature points from the captured image obtained by the captured image acquisition unit 2, and a plurality of captured images acquired before or after the captured image from which the feature points have been extracted. A candidate corresponding point corresponding to the feature point for the feature point tracking unit 4 for tracking the feature point, and for the candidate corresponding point, a corresponding point is determined from the tracking result, and the determined corresponding point is used. And a processing unit 5 for calculating the coordinates of the photographing position or the object.

  Here, the photographed image that changes little by little is a photographed image that changes little by little in terms of time or space, and means that the object is common to most of the image. More specifically, it refers to a captured image that changes so as to include at least four common feature points. It is preferable to take as many common feature points as possible. It may be a moving image continuously shot with a video camera, may be taken sequentially with a single camera by changing the time or place little by little, all the frames taken sequentially may be acquired, and acquired every few frames. May be. Typically, either one of the object or the photographing device moves and the other image is taken in a stationary state. However, the photographing may be performed in a state in which both move relatively. In addition, feature point tracking starts from a feature point and searches for corresponding candidate corresponding points and corresponding points sequentially. In a broad sense, candidate corresponding points and corresponding points are also collectively feature points. In this way, feature points are sometimes used in such a broad sense. A candidate corresponding point refers to a corresponding point before being determined. The photographing position is typically the position coordinate of the imaging lens, and the coordinate of the object is typically the coordinate of each feature point attached to the object or the position coordinate of the object gravity center. With such a configuration, even when processing of swinging of the imaging device or processing of feature points to disappear / reproduce is required, imaging of the imaging device can be performed from a moving image or a captured image that changes little by little. An image processing apparatus that can accurately measure the position, orientation, or coordinates of an object can be provided.

Further, according to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the relative movement of the screen obtained by projective transformation using two photographed images sequentially acquired as a tracking result. A movement trajectory obtained by sequentially connecting an amount or a movement vector of a feature point between two captured images or a screen relative movement amount or a movement vector is used.

  Here, the relative movement amount of the screen is a relative movement amount on the screen between the photographing apparatus and the photographing object (including feature points), and the movement vector is the relative of each feature point on the two-dimensional photographed image. A moving vector. As this movement vector, it is preferable to use a vector obtained by correcting the swing of the photographing apparatus by projective transformation, but it includes a case where a vector without correction is used. In addition, although adjacent images are typically used as the two captured images, the present invention is not limited to this, and images separated by several frames may be used. If comprised in this way, the candidate corresponding | compatible point which is not suitable as a corresponding | compatible point can be accurately selected using a screen relative movement amount, a movement vector, or these movement locus | trajectories.

Further, according to a third aspect of the present invention, in the image processing apparatus according to the second aspect , the feature point tracking unit includes a captured image from which the feature points have been extracted and a plurality of captured images sequentially acquired before or after that. Search for candidate corresponding points corresponding to the feature points in the image, form a movement vector from the positional relationship between the searched candidate corresponding points and the feature points, and correspond to the feature points based on the movement vector and the screen relative movement amount Candidate corresponding points judged to have a high probability are selected, candidate corresponding points corresponding to feature points and corresponding points are searched for a plurality of captured images sequentially acquired before or after that, formation of movement vectors, candidate corresponding points The selection of candidate points and the search for candidate corresponding points are repeated sequentially.
If comprised in this way, the reliability of a candidate corresponding | compatible point can be improved using the tracking result of a feature point.

In addition, according to a fourth aspect of the present invention, in the image processing apparatus according to the second aspect , the feature point tracking unit 4 determines the movement vector and screen of each feature point from the tracking result of each feature point between adjacent captured images. The relative movement amount is obtained, the movement vector of each feature point is compared with the estimated position of each feature point estimated from the screen relative movement amount, and those having large differences are excluded as defective points.
If comprised in this way, the candidate corresponding point which is not suitable as a corresponding point can be removed easily and with high precision.

Further, according to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect , the processing unit 5 detects disappearance of feature points or corresponding points, and a predetermined number or more disappearances, or a blank area that exceeds a predetermined range. When this occurs, the feature extraction unit 3 is instructed to newly establish a feature point.
Here, the predetermined number and the predetermined range may be determined as appropriate values within a range in which template matching or the like can be sufficiently performed, for example, 1/4 of the feature points displayed on the screen or 1/4 of the screen. With this configuration, an appropriate number of feature points on the screen can always be held.

According to a sixth aspect of the present invention, in the image processing device according to the second aspect , the processing unit 5 moves the screen relative movement amount or the movement vector sequentially connected to each feature point of the first photographed image. An estimated image related to the second captured image obtained by separating a predetermined amount (time, position) using the trajectory is created, the second captured image is compared with the estimated image related thereto, and the candidate corresponding points are compared. Corresponding points are determined, and the imaging position or the coordinates of the object are calculated using the determined corresponding points.
Here, since the predetermined amount (time, position) changes depending on the degree of change of the acquired captured image, an amount that is sufficiently effective to determine the corresponding point may be determined. Further, the predetermined amount is not limited to one value and has an appropriate range. With this configuration, reliable corresponding points can be selected using the estimated image.

Further, according to a seventh aspect of the present invention, in the image processing apparatus according to the sixth aspect , the first photographed image is photographed before or after the second photographed image or in front or behind, The movement trajectory follows time or position in the traveling direction or the backward direction.
If comprised in this way, the estimation of any forward / reverse direction is also possible.

Further, according to an eighth aspect of the present invention, in the image processing apparatus according to the sixth aspect , the processing unit 5 is once excluded from the candidate corresponding points, but is reproduced with a captured image acquired before or after that. About the candidate corresponding point to be estimated, the captured image acquired before or after the second captured image is used as a second captured image, and the second captured image is compared with the estimated image related thereto, and the probability that the candidate corresponding point is a candidate corresponding point is high. The candidate corresponding points determined to be restored as candidate corresponding points again.
With this configuration, the candidate corresponding points can be restored.

According to a ninth aspect of the present invention, in the image processing apparatus according to the fourth aspect , the processing unit 5 has a relative screen movement amount between adjacent images or between the first image and the second image. Adjust the acquisition of adjacent images to be quantitative.
Here, the predetermined amount is the same as in the case of claim 6. If comprised in this way, the comparison of the movement vector in a long-term visual field will always become possible.

According to a tenth aspect of the present invention, in the image processing apparatus according to the fourth aspect , the processing unit 5 has a predetermined number of pixel shifts between adjacent images or between the first image and the second image. Adjust so that
Here, since the predetermined number also changes depending on the degree of change of the acquired captured image, it is sufficient to empirically determine an effective amount sufficient to determine the corresponding point. Further, the predetermined number is not limited to one value and has an appropriate range. If comprised in this way, extraction of a feature point or a candidate corresponding point will become easy.

According to an eleventh aspect of the present invention, in the image processing apparatus according to the sixth aspect, any one of an overlap ratio, a base line length, and a movement vector between captured images obtained by the captured image acquisition unit 2 is used. And an image selection unit that selects the second image.
If comprised in this way, a comparison image can be selected stably.

An image processing method according to a twelfth aspect of the present invention includes a captured image acquisition step of sequentially acquiring captured images that change little by little with respect to a relatively moving object, and a captured image obtained in the captured image acquisition step. A feature extraction step for extracting feature points, and a feature point for tracking feature points by searching for candidate corresponding points corresponding to feature points for a plurality of photographed images acquired before or after the photographed image from which the feature points are extracted. A tracking step and a calculation processing step of determining the corresponding point from the tracking result for the candidate corresponding point and calculating the imaging position or the coordinates of the object using the determined corresponding point.
With such a configuration, even when processing of swinging of the imaging device or processing of feature points to disappear / reproduce is required, imaging of the imaging device can be performed from a moving image or a captured image that changes little by little. It is possible to provide an image processing method that can accurately measure the position, orientation, or coordinates of an object.

  According to the present invention, even when processing of swinging of the photographing device or processing of feature points to disappear / reproduce is required, the photographing position of the photographing device can be determined from a moving image or a photographed image that changes little by little. Thus, it is possible to provide a technique capable of accurately measuring the posture or the coordinates of the object.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram for explaining the concept in the present embodiment. This is an example in which a camera is mounted on an automobile, the position of the automobile is changed little by little, and an urban area that is an object is photographed, and the position coordinates of the camera, that is, the locus of the automobile are obtained from the tracking results in these captured images. This makes it possible to continuously display the position of the car in the car navigation, but it is also significant to be used complementarily in a section where GPS radio waves cannot be received. Feature points with movement and feature points with large fluctuations are removed because they are inappropriate for obtaining the position coordinates of the camera.

FIG. 2 shows a configuration example of the image processing apparatus 100 in the present embodiment. In the figure, reference numeral 1 denotes an operation unit that controls each unit of the image processing apparatus 100 to function as an image processing apparatus. Specifically, the operation start / stop instruction to the captured image acquisition unit 2 and the feature extraction unit 3 Change setting, feature point tracking unit 4 setting change, start / stop instruction, determination start instruction to processing unit 5, orientation execution instruction, and the like.
Reference numeral 2 denotes a captured image acquisition unit that sequentially acquires captured images such as moving images. In addition to acquiring captured images, the captured image acquisition unit 2 outputs the captured images to the feature extraction unit 3, stores the captured images in the moving image memory 10, and the like. Note that the captured image may be acquired by communication from another imaging apparatus without capturing the captured image.

Reference numeral 3 denotes a feature extraction unit that extracts feature points from sequentially acquired captured images, and extracts feature points from the captured image input from the captured image acquisition unit 2 and a feature point tracking unit of the extracted feature points. 4 and output to the processing unit 5.
A feature point tracking unit 4 searches for candidate corresponding points corresponding to the feature points input from the feature extraction unit 3 and tracks the feature points. In addition to the tracking process, the tracking result is output to the processing unit 5 Then, the execution start instruction of the processing unit 5, the determination of the arrangement of the candidate corresponding points, the new instruction of the feature point to the feature extraction unit 3, etc. are performed.

A processing unit 5 includes a long-term determination processing unit 6 and an orientation processing / three-dimensional measurement unit 7. The long-term determination processing unit 6 determines the corresponding point from the tracking result in the feature point tracking unit 4, and further deletes and restores the corresponding point based on the tracking result. Estimated position calculation processing for the feature point in the initial frame input from the feature extraction unit 3, pass / fail judgment processing for the candidate corresponding point of the current frame input from the feature point tracking unit 4, correspondence point restoration processing, features of these processing The reflection to the point tracking unit 4 and the recording of the candidate corresponding point information to the corresponding point information memory 9 are performed. The orientation processing / three-dimensional measurement unit 7 performs orientation calculation and three-dimensional measurement using the determined corresponding points. Besides, acquisition of captured images from the moving image memory 10 and candidates from the corresponding point information memory 9 The corresponding point information is acquired, the orientation result and the three-dimensional measurement result are output to the display unit 8, the corresponding point information memory 9 is reflected, the orientation result and the three-dimensional measurement result are output to the outside.
Reference numeral 8 is a display unit for stereoscopically displaying an image of an object subjected to orientation processing or three-dimensional measurement by the processing unit, and 9 is a corresponding point information memory for storing information on feature points and their corresponding points (including candidate corresponding points). Reference numeral 10 denotes a moving image memory for storing captured images.

  FIG. 3 shows a flow example of the image processing method according to the first embodiment. First, the motion of feature points on the image is analyzed by moving image analysis (S10). In moving image analysis, image acquisition, feature extraction, tracking, and feature point association are performed. A subject is photographed by gradually changing the position of a moving body such as an automobile, and feature points are automatically and sequentially associated with adjacent images, and candidate corresponding points are extracted. Further, the corresponding points are determined by continuing the tracking. Next, erroneous correspondence points are removed (S20). Next, a pair of stereo images having a sufficient baseline length is determined, and automatic orientation by mutual orientation or bundle adjustment is performed (S30). Position coordinates, posture, and further three-dimensional coordinates are obtained (S40).

[Video analysis]
FIG. 4 shows a flow example of moving image analysis. Here, a flow example of real-time processing is shown, and the overall flow is shown in FIG. The moving image analysis sequentially processes moving images and captured images that change little by little. First, the captured image acquisition unit 2 acquires a captured image. You may acquire and acquire an image with an own imaging device, and you may acquire the image image | photographed with the other imaging device via a communication line. The operation unit 1 sequentially supplies a photographed image that changes little by little from the photographed image to the feature extraction unit 3. In this embodiment, since a photographing device is mounted on a car and photographed while moving, a photographed image that changes little by little is a photographed image that changes little by little in terms of time or space. It means that things are common. The feature extraction unit 3 extracts feature points from each captured image (S11). Also, noise removal and feature point selection are performed.

  Next, the feature point tracking unit 4 tracks each feature point selected by the feature extraction process (S12). That is, a candidate corresponding point corresponding to a feature point is obtained, a feature point movement vector and a screen relative movement amount are obtained, and these are connected to obtain a movement locus. The screen relative movement amount is the relative movement amount on the screen between the photographing device and the photographing object (including the feature points), and the movement vector is the relative movement vector of each feature point on the two-dimensional photographed image. Say. In tracking feature points, first, template matching is performed on adjacent captured images (S13) to obtain candidate corresponding points corresponding to the feature points. Thereby, the movement vector of each feature point is obtained. Further, by performing projective transformation using the adjacent photographed image (S15), the screen relative movement amount with respect to the photographing apparatus is obtained. Next, the movement vector of each feature point is compared with the relative movement amount of the screen between frames, and the quality of the movement vector is determined (S14). Then, candidate corresponding points that are considered to be erroneous correspondences indicating abnormal movement are deleted (S16). By repeating steps S15 and S16, the accuracy of projective transformation is improved.

  Next, the arrangement of candidate corresponding points is determined (S17). That is, the arrangement of feature points and candidate corresponding points on the captured image is confirmed. If the arrangement of feature points is extremely biased and a blank portion is generated, the feature extraction unit 3 is instructed to newly set a point existing in the newly generated blank portion as a new feature point. Then, the process returns to the feature extraction (S11) again, and the feature extraction (S11) and the tracking process (S12) are sequentially repeated for new adjacent images in real time. If feature extraction has been completed for a series of photographed images, the process returns to template matching (S13), and tracking processing (S12) is sequentially performed for new adjacent images.

[Feature extraction]
Feature extraction (S11) is performed in the feature extraction unit 3. Typically, the initial frame is extracted from the entire screen, and the next frame is extracted from a new screen area that does not overlap the initial frame. In the present embodiment, the feature points in the initial frame are extracted by, for example, a MORAVEC operator (HP Moravec. Towers Visual Visual Avid Aidance. Proc. 5th International Joint Conference 77. Int. Operators such as Harris, Pressy, Susan can be employed as appropriate.

  Whatever feature extraction operator is used, there is a problem that it is easy to react to slight noise on the image (such as noise on the edge). In order to improve this property, noise removal processing of the image is performed using a noise filter such as an average value filter before using the feature extraction operator.

  However, even if noise removal is performed, feature points concentrate on a certain part of the image (such as a tree or grass) depending on the subject to be photographed, which may adversely affect template matching described later. In order to avoid this, a collocation process is performed. For example, a method of limiting the distance between points of each feature point is used for the point selection process. Further, when the maximum number of feature points is designated in advance, the distance between points in the case where the feature points are arranged uniformly over the entire image is obtained, and the feature points are arranged so as not to fall below this distance. Thus, the relative orientation is ensured by arranging the feature points evenly throughout.

[Tracking process]
The feature point tracking unit 4 tracks each feature point selected by the feature extraction process (S12). That is, a candidate corresponding point corresponding to a feature point is obtained, a feature point movement vector and a screen relative movement amount are obtained, and these are connected to obtain a movement locus.

[Template matching]
In this embodiment, template matching is used for tracking feature points (S13). Neighboring images are sequentially selected from the acquired captured images to form a stereo pair, and for example, stereo matching is performed by a technique such as the SSDA method (sequential residual method) to obtain candidate corresponding points (S13). The SSDA method (sequential residual method) is a method for determining similarity using a residual, and a position where a partial matrix residual is minimum is obtained as a candidate corresponding point. SSDA template matching is considered to be relatively fast as template matching and easy to implement in hardware. Other methods such as a normalized correlation method can also be employed. For template matching, it is important to optimally select a template size and a search range, and the search range is set optimally based on the frame rate, moving speed, etc. of the video camera.

[Movement vector pass / fail judgment]
Candidate corresponding points for each feature point are obtained by template matching, but many mismatches also occur. For example, if the value of SSDA template matching is too high (accuracy is low), it is determined that a mismatch has occurred at that point. Note that if the feature points overlap as a result of template matching, priority is given to the one with good accuracy from the result of the correlation value.

  In addition, the movement vector of each feature point is obtained from the candidate corresponding point obtained by template matching. The quality of the movement vector and the suitability of the candidate corresponding point are judged, and the candidate corresponding point generated by the mismatching is removed ( S14). First, the movement vector of each feature point is compared with the relative movement amount of the screen between frames, and candidate corresponding points that are considered to be erroneous correspondences indicating abnormal values are deleted. In this case, a threshold value may be determined and deleted. The relative movement amount of the screen is the relative movement amount on the screen between the photographing device and the photographing target (including the feature point), but the same photographing target also changes depending on the distance and direction from the photographing device. What is necessary is just to compare by the movement amount of most feature points (move almost the same) around the feature points.

  In this embodiment, projective transformation is used to remove candidate corresponding points. Since the overall movement between frames is very short in time, the assumption is made that it can be approximated by projective transformation, and the screen relative movement amount is estimated by projective transformation (S15). In other words, the movement is estimated by projective transformation, the movement vector for each feature point is compared with the screen relative movement amount, and the candidate corresponding point indicating an abnormal value is rejected (S16).

  In addition, since the captured image also swings with the swing of a moving body such as an automobile, a movement vector that removes the rotation and the camera position fluctuation by correcting the rotation and camera position fluctuation using projective transformation, The relative movement amount of the screen can be obtained. The screen relative movement amount is estimated, and at the same time, candidate corresponding points that are considered to be miscorresponding are deleted. Further, the reliability of the candidate corresponding point is improved by performing the projective transformation again after deleting the erroneous corresponding point. In addition, the accuracy of template matching by using candidate corresponding points in which rotation and camera position fluctuation are corrected can be improved, and the reliability of the movement vector can also be improved.

  In the shooting screen, when a feature point is given to a moving object such as a running car, an asuka, a fallen leaf, or when the camera shakes heavily, an erroneous response point may occur. The camera swing can be corrected by projective transformation. On the other hand, an object that moves differently from the object to be photographed causes an erroneous correspondence point. Therefore, by removing the miscorresponding points caused by the movement of the object etc., the reliability of the feature points (including corresponding points and candidate corresponding points) is improved, the accuracy of the mismatching determination is improved, It is possible to cope with upsets.

[Placement of corresponding points]
Next, the arrangement of corresponding points is determined (S17). Here, the arrangement of the candidate corresponding points remaining after the defective ones are removed is confirmed. At the time of this arrangement determination, a new feature point instruction and candidate corresponding points are also restored (S18, see FIG. 7).

  FIG. 5 schematically shows an example of feature points and candidate corresponding points in corresponding point matching. The position of the object is slightly different between the left image and the right image. Although many feature points are taken in the left image, the right image has a small number of candidate corresponding points corresponding thereto. This schematically shows that defective candidate corresponding points are removed.

  FIG. 6 shows an example of new feature points. For example, if the arrangement range of the feature points on the screen is not within the range of 50% of the screen width and height, or the center position is not near the center of the screen, the feature extraction unit 3 is instructed. Then, the feature extraction operator performs a new feature point setting process. According to this criterion, in FIG. 6 (a), the feature points are shifted to the upper right and the center position is outside the above range, so the feature points are newly established, and in FIG. 6 (b), the center position is within the above range. Therefore, a feature point is not newly established. Even when the number of feature points becomes a predetermined number or less, a new feature point instruction is given for stable orientation analysis and the like.

  Also, it is checked whether the candidate corresponding point once disappeared outside the screen appears again in the screen, and if it appears, it is recovered. Specifically, the corresponding point on the current frame is estimated by affine transformation based on the points remaining on the current frame for the candidate corresponding point that has been erased as having disappeared from the screen. Template matching is performed again at the estimated location to restore candidate corresponding points that have obtained good results.

In addition, if the candidate corresponding point is not defective and the arrangement of the candidate corresponding point matches the estimated value by the projective transformation in the arrangement determination, the process returns to feature extraction (S11), and the candidate of the frame having the candidate corresponding point The corresponding candidate corresponding point of the next frame is searched for the corresponding point. This operation is repeated until the final frame of the captured image such as a moving image, and the tracking is continued (see S19, FIG. 7).
In the above description, feature points are extracted using one of a plurality of captured images as an initial frame image, and candidate corresponding points corresponding to the feature points are searched for images acquired after that image. For example, when the present invention is applied to the field of surveying, civil engineering, GIS (Geographic Information System (GIS)), or car navigation, it is used complementarily in sections where GPS radio waves cannot be received. Suitable when That is, when searching for candidate corresponding points in real time. However, on the contrary, candidate corresponding points corresponding to the feature points may be searched for a plurality of images acquired before the captured image from which the feature points are extracted. In that case, for example, the frame of the final image becomes the initial frame. This can be applied, for example, when a plurality of captured images are acquired and then analyzed based on the acquired images to measure the coordinates of the object. In the following description, the reverse search is possible.

[Long-term vector pass / fail judgment]
Next, a long-term vector pass / fail judgment routine will be described. In this routine, candidate corresponding points are determined as corresponding points, and feature points (including corresponding points and candidate corresponding points) are newly created, removed, and reproduced based on long-term tracking data (S201, FIG. 7). reference). The long-term vector pass / fail judgment process (S201) is performed by the long-term judgment processing unit 6.

  FIG. 7 is a diagram for explaining a long-term vector pass / fail judgment routine in moving image analysis. First, the coordinate information of the candidate corresponding points acquired by the feature extraction process (S11, see FIG. 4) in the initial frame is output to the “long-term quality determination process” routine (S21). When the movement amount of the entire screen between each frame is estimated by projective transformation (S15), the projection transformation coefficient (rotation matrix, translation vector, camera external parameter, etc.) related to each frame is “long-term quality determination process”. The data is output to the routine (S22). Next, the feature points on the initial frame obtained in step S21 are subjected to projective transformation for each frame using the projective transformation coefficient output in step S22, and each frame obtained by the projective transformation is applied to each frame obtained by the projective transformation. The estimated position of the candidate corresponding point on each frame is obtained from the corresponding two-dimensional image (S23).

  When a predetermined amount of movement, time, or number of frames has elapsed, candidate corresponding points are determined as corresponding points, and corresponding points are newly established and restored. In other words, in the “tracking process” routine (S12, see FIG. 4), the candidate corresponding points have obtained a sufficient amount of movement to be determined as corresponding points. When certain conditions are satisfied (S202), such as the arrangement of the candidate corresponding points remaining in the distribution is biased, the number of candidate corresponding points is reduced, or a new feature point instruction is issued (S18), This triggers the start of long-term pass / fail judgment execution (S29), and the “long-term pass / fail judgment processing” routine acquires tracking position information of candidate corresponding points in the current frame calculated by the “tracking processing” routine (S12). (S241), the tracking position information is compared with the estimated position of the candidate corresponding point calculated in step S23 (S242), the corresponding point is determined (S251 to S252), and the erroneous corresponding point is removed. And management (S261~S263), but was lost once perform the revival process of the candidates for the corresponding point that has been reproduced in the current frame (S271~S273).

  If the position residual is small between the tracking position of the candidate corresponding point in the current frame and the estimated position of the candidate corresponding point (S251), the candidate corresponding point is determined as the corresponding point (S252). When the position residual between the tracking position of the candidate corresponding point in the current frame and the estimated position of the candidate corresponding point is large (S261), it is considered that there is a high possibility that a miscorrespondence has occurred. Therefore, back matching from the current frame to the initial frame is performed (S262) and the quality is determined. Specifically, template matching is performed using a small area around the candidate corresponding point in the current frame as a template and a search area around the candidate corresponding point in the initial frame. As a result of the template matching, a case where a sufficient correlation is obtained near the candidate corresponding point position in the initial frame is good, and otherwise the candidate corresponding point is deleted as an abnormal point (S263).

  In addition, in the current frame, once disappeared in the “tracking process” routine, but the point reproduced in the current frame (S271), the candidate corresponding point is estimated using the estimated position of the candidate corresponding point in the current frame. Perform revival processing. Specifically, template matching is executed using a small area around the candidate corresponding point in the initial frame as a template and a search area around the candidate corresponding point in the current frame (S272). If sufficient correlation is obtained as a result of template matching, the candidate corresponding points are restored (S273). Further, the processing result of candidate corresponding point deletion and restoration is fed back to the “tracking processing routine” (S28).

[Remove false points]
The feature points are associated with all the frames by the moving image analysis processing in the corresponding point tracking unit 4 and the processing unit 5 described above. However, these associations are based on a two-dimensional analysis, and may actually include miscorresponding points.
Therefore, in this embodiment, further corresponding points are removed (S20). The removal of these miscorresponding points is performed using the results of the following orientation processing and three-dimensional measurement processing.

[Orientation / 3D measurement]
Next, relative orientation and three-dimensional measurement are performed in the orientation processing / three-dimensional measurement unit 7. For each image selected as a stereo pair, orientation calculation processing is performed using the coordinates of the feature points and the corresponding points. By the orientation calculation process, the position, tilt, and three-dimensional position of the corresponding point of the photographed camera can be obtained. The orientation calculation processing is performed by relative orientation for associating captured images selected as stereo pairs, and orientation for multiple or all images is performed by bundle adjustment. When this stereo pair selection is performed, the orientation processing / three-dimensional measurement unit appropriately selects a set of images estimated to have an appropriate baseline length from a plurality of images obtained by the captured image acquisition unit. It is possible to perform accurate orientation processing and three-dimensional measurement.

FIG. 8 is a diagram for explaining the orientation. Model image coordinate system in the stereo image XYZ and camera coordinate system _X 1 _Y 1 _{Z 1,} is an illustration of the relationship X ₂ Y _{2 Z 2.} The origin of the model image coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. The scale is based on the base length (distance between camera coordinate origins) as a unit length. The parameters obtained at this time are the Z-axis rotation angle κ1, the Y-axis rotation angle φ1, the X-axis rotation angle ω1, the right-side camera Z-axis rotation angle κ2, the Y-axis rotation angle φ2, X There are six rotation angles of the shaft rotation angle ω2. These rotation angles are obtained by the orientation calculation process, and the precise three-dimensional coordinates of the camera, the posture, and the three-dimensional coordinates of the object are calculated using the rotation angles. Further, these orientation and three-dimensional measurement can be automated (see Patent Document 1).

  FIG. 9 is a diagram showing the result of the three-dimensional measurement, showing the photographing position where the points arranged in a substantially straight line below the left side surface are obtained, and a little small plot scattered in the three-dimensional diagram is a three-dimensional coordinate The feature points and corresponding points given values are shown.

  As described above, according to the first embodiment, even in the case where the process of swinging the imaging apparatus and the process of feature points to be lost / reproduced are required, from a moving image or a captured image that changes little by little, It is possible to provide a technique that can accurately measure the coordinates of the photographing position and posture of the photographing apparatus. In addition, the processing of moving image analysis and removal of erroneous corresponding points can be realized by a computer, and all processes of the image processing method including automation and orientation that have already been automated can be automated. In addition, the three-dimensional coordinates of the photographing apparatus can be obtained stably with high accuracy.

[Second Embodiment]
In the present embodiment, the feature points of an image are tracked forward (future) and backward (past) for a plurality of screens, and the tracking trajectory is extended, so that the reference point (for example, the initial frame or the current frame) is forward or A back image is estimated. The estimated value and the actually measured value are compared, and candidate corresponding points having the actually measured value far from the estimated value are deleted. Alternatively, the candidate corresponding points that disappeared and reproduced once are traced back (in the past) for several screens and estimated, and compared with the original feature points, candidate corresponding points whose estimated values and measured values almost match are restored. To do. Although such a function is added as compared with the first embodiment, the other points are the same as those of the first embodiment. The reference screen corresponds to the first image, and the estimated image compared with the first image corresponds to the second image.

  FIG. 10 is a diagram for explaining an estimated image. In order to ensure the reliability of the estimated points, continuous checks are performed on the forward and backward images. Aspects are as follows: (1) The screen motion is estimated one by one on the forward side and the backward side using projective transformation, the tracking value of the movement vector for each feature point and the estimated value based on the screen relative movement amount, Are compared, and candidate corresponding points indicating abnormal values are rejected. (2) The estimation is sequentially continued on the forward side and the backward side until a predetermined number of frames and movement amount are ensured, and the validity of the estimated image creation is determined by comparing the estimated value with the actually measured value. The determination is made by template matching, for example. Thereby, the reproduced candidate corresponding points are restored. In particular, when there is little or no overlap between the initial frame and the current frame, the estimated value and the tracking value are compared in a range where there is an overlap, and by connecting these while checking this, Estimates can also be made, which is significant.

[Third Embodiment]
The present embodiment adjusts the amount of movement with software or hardware for the purpose of facilitating the extraction of feature points and the processing of ensuring an appropriate baseline length and facilitating measurement. Although such a function is added as compared with the first or second embodiment, the other points are the same as those of these embodiments. As an example of software adjustment, the processing unit 5 refers to the movement amount data of each feature point in a large number of images so that the movement amount of each feature point between adjacent images becomes a predetermined amount of interval. This is possible by adjusting any one of the movement correction amount and the filtering process having directionality. As an example of hardware adjustment, the captured image acquisition unit 2 refers to the movement amount data of each feature point in a large number of images so that the movement amount of each feature point between adjacent images has a predetermined interval. This is possible by adjusting any one of the shooting timing, shooting position, or shooting direction. It is also possible to adjust the number of pixels so that a predetermined amount of interval is obtained. When selecting an appropriate stereo pair, in the orientation processing / three-dimensional measurement unit, by selecting a set of images estimated as an appropriate baseline length from a plurality of images obtained by the captured image acquisition unit, Appropriate orientation processing and three-dimensional measurement can be performed. In addition, an image selection unit that selects a comparison image based on any one of an overlap rate, a base length, a movement vector, and the like of a plurality of images obtained by the captured image acquisition unit may be provided.

[Fourth Embodiment]
In this embodiment, an edge extraction method is further used as a feature point extraction method before the feature extraction operator is applied. Although such a function is added as compared with the first embodiment, the other points are the same as those of the first embodiment. Since the process of simply extracting feature points (for example, MORAVEC method) is performed on the entire screen, processing time is required. In addition, depending on the parameters, there is a problem that points are extracted too much on the entire screen. As processing for reducing these problems, edge extraction is performed by simple processing before processing by the MORAVEC method, and further feature point extraction is performed on the extracted image. In this way, the number of points where feature points are extracted is drastically reduced, and more reliable feature points can be set than simple feature point extraction, that is, the reliability of feature points can be increased.

FIG. 11 shows an example of edge extraction processing. An edge image can be easily obtained by subtracting images between consecutive frames. In this figure, a portion that does not change like a building wall is zeroed by taking the difference of the slightly changed photographed image, and is displayed as black in the figure. Since the boundary between the wall and the window, the boundary between the wall and the space, etc. are changed, the data remains when the difference between the captured images is taken. Therefore, the boundary lines of the respective parts are displayed as white lines in the figure by the remaining data. Then, feature extraction processing by, for example, the Moravec operator is performed on the edge image to obtain feature points for tracking. For example, an intersection of white lines can be used as a feature point.
The edge extraction process is not a difference between frames but a difference between one image, for example, one pixel up, one pixel down, one pixel left, or one pixel right, and one's own image By doing this, or by combining the vertical and horizontal directions, an edge image with respect to directionality can be acquired. If the camera is moving to the right, it is also possible to make a difference between the images moved by the pixel in the right direction. In this way, even if the image is too far away between the frames, edge extraction, that is, feature point extraction can be performed by simple difference processing. Since this process can be performed only with the difference, it is easy even with a software process, and can be processed in real time if it is implemented as hardware.

[Fifth Embodiment]
In the first embodiment, in order to determine the corresponding point, the estimated position of the candidate corresponding point and the “tracking process” are performed after a predetermined amount of movement or the number of frames has elapsed in the long-term vector pass / fail judgment routine. Compared with the tracking position of the candidate corresponding point in the current frame calculated by the routine, it is confirmed, but in this embodiment, in the tracking processing routine, when a predetermined movement amount, time, or number of frames has elapsed. An example is shown in which candidate corresponding points that remain without being removed are determined as corresponding points. This point is different from the first embodiment, but the other points are the same as in the first embodiment. For example, candidate corresponding points remaining after the seventh routine are determined as corresponding points. In this case, in the long-term vector pass / fail judgment routine, the corresponding point may be lost or reproduced, but in this case, a process of removing or restoring it may be performed. Similarly, for resurrection, for example, candidate corresponding points that once disappeared but remained through the seventh routine may be revived.

[Sixth Embodiment]
In the first embodiment, an example in which the photographing apparatus moves while the photographing target is stationary is described. However, the present embodiment is an example in which the photographing target moves while the photographing apparatus is stationary. Even in this case, in addition to the original object, a moving object may be interrupted between the image capturing apparatus and the object, or the image capturing apparatus may be shaken. It is possible to obtain the three-dimensional coordinates of the feature points related to the object in the changed captured image. In addition, if the object itself rotates, the feature point repeatedly disappears and is restored, so that the present invention can be applied. In addition, a plurality of objects may move differently, and the present invention can be applied to each object even in such a case.

[Seventh Embodiment]
In the first embodiment, an example in which projective transformation is used has been described. However, when the imaging apparatus has no swing and a constant direction (such as going straight forward), the motion vector is not necessarily transformed without performing the projective transformation. The candidate corresponding points can be determined, removed, newly established, and restored by comparing the movement vectors of the respective feature points with the entire screen or the movement vectors around the feature points. It should be noted that more reliable processing can be performed by creating a screen relative movement amount using projective transformation and correcting the fluctuation of the photographing screen.

  The present invention can also be realized as a program for causing a computer to execute the image processing method described in the above embodiment. The program may be stored and used in a built-in memory of the operation unit 1, may be stored and used in a storage device inside or outside the system, and may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and it is obvious that various modifications can be made to the embodiment.

  For example, in the above embodiment, an example has been described in which a captured image is acquired in a state where one of the object or the imaging apparatus is moving and the other is stationary. However, the present invention is applied when both are moving. May be. For example, the present invention is sufficiently applicable when one moving speed and direction are constant. Moreover, although the example of the orthogonal coordinate was demonstrated about the three-dimensional coordinate, a cylindrical coordinate and a spherical coordinate can also be used according to the condition of a target object and an imaging device. It is also possible to omit a part of the process such as removal of the erroneous corresponding point in the process S20. Further, although an example in which the MORAVEC operator is used for feature point extraction and the SSDA template matching is used for template matching has been described, other operators and template matching methods may be used. Moreover, although the example which uses the projective transformation for calculating | requiring the screen relative movement amount was demonstrated, you may use projective transformations, such as an affine transformation and a Fermat transformation. Also, in the first embodiment and the like, an example of comparing the tracking position of the candidate corresponding point in the current frame and the estimated position of the candidate corresponding point with respect to the algorithm for determining the corresponding point and removing or restoring the candidate corresponding point The tracking position and the estimated position may be determined by comparing them twice, or another algorithm such as comparing the movement locus with the locus of the estimated position may be used. Further, an example has been described in which a candidate corresponding point is determined as a corresponding point when a predetermined amount of movement, time, or number of frames has passed, but the predetermined amount of movement, time, or number of frames can be variously determined based on experience. .

  The present invention is used for a photographing apparatus using a moving image or measurement of position coordinates of a photographing target.

It is a figure for demonstrating the concept of this invention. It is a block diagram which shows the structural example of the image processing apparatus in 1st Embodiment. It is a figure which shows the example of a flow of the image processing method in 1st Embodiment. It is a figure which shows the example of a flow of real-time processing among moving image analysis. It is a figure which shows the example of corresponding point matching. It is a figure which shows the example which newly establishes a feature point. It is a figure for demonstrating the quality determination routine of a long-term vector among moving image analysis. It is a figure for demonstrating orientation. It is a figure which shows the result of three-dimensional measurement. It is a figure for demonstrating an estimated image. It is a figure which shows the example of an edge extraction process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Operation part 2 Captured image acquisition part 3 Feature extraction part 4 Feature point tracking part 5 Processing part 6 Long-term judgment processing part 7 Orientation process / three-dimensional measurement part 8 Display part 9 Corresponding point information memory 10 Moving picture memory 100 Image processing apparatus

Claims (12)

A captured image acquisition unit that sequentially acquires captured images that change gradually with respect to a relatively moving object;
A feature extraction unit that extracts feature points from the captured image obtained by the captured image acquisition unit;
A feature point tracking unit that searches for candidate corresponding points corresponding to the feature points in a plurality of captured images acquired before or after the captured image from which the feature points are extracted;
For the candidate corresponding points, and a processing unit for determining the corresponding points from the tracking result, and calculates the coordinates of the photographing position or the object using been established corresponding points;
The feature point tracking unit searches for candidate corresponding points corresponding to the feature points for a plurality of captured images sequentially acquired before or after the captured image from which the feature points have been extracted, and the candidate corresponding points that have been searched The feature point movement vector between adjacent captured images is formed from the positional relationship between the image and the feature point, the screen relative movement amount is obtained from the projective transformation using the adjacent captured image, and the movement vector and the screen relative movement amount are compared. A short-term pass / fail judgment to judge the pass / fail of the movement vector in real time, and delete candidate corresponding points that seem to be false corresponding points indicating abnormal movement based on the short-term pass / fail judgment, before or after sequentially acquired search for candidate corresponding points for a plurality of captured images, formation of the movement vector, the short term quality judgment, erroneous sequentially repeating the deletion candidate corresponding points appear to the corresponding point short-term acceptability judgment routine Done;
The processing unit is processed by the short-term pass / fail judgment routine when the candidate corresponding point has obtained a movement amount sufficient to determine the candidate corresponding point as a corresponding point in the feature point tracking unit. The candidate corresponding points in the current frame are subjected to projective transformation for each frame using the tracking position information of the candidate corresponding points and the estimated projective transformation coefficient related to each frame, and the candidate corresponding points on each frame obtained by the projective transformation are obtained. The estimated position is obtained, the sufficient amount of movement is obtained, and the tracking position information when the time or the number of frames is obtained is compared with the estimated position of the candidate corresponding point to determine the quality of the candidate corresponding point. A long-term pass / fail judgment routine that makes a long-term pass / fail judgment, removes candidate corresponding points that are considered to be erroneous corresponding points with a large residual based on the long-term pass / fail judgment, and determines candidate corresponding points with a small residual as the corresponding points And confirm It calculates the coordinates of the photographing position or the object using the corresponding points;
Image processing device. Using the coordinates of the corresponding points determined by the processing unit, the three-dimensional position of the corresponding points is obtained for each image selected as a stereo pair, and the erroneous corresponding points are removed;
The image processing apparatus according to claim 1. The processing unit detects the disappearance of the feature point or the corresponding point, and instructs the feature extraction unit to newly establish a feature point when a predetermined number of disappearances and a blank area exceeding a predetermined range occur;
The image processing apparatus according to claim 1 . The feature point tracking section, the relative previously from photographed image feature points are extracted or photographed image subsequent to or taken by the forward or backward, to follow the time or position the moving locus in the traveling direction or reverse direction Ask for ;
The image processing apparatus according to any one of claims 1 to 3 . The processing unit temporarily excludes the captured image acquired before or after the candidate corresponding point that is once excluded from the candidate corresponding points, but is assumed to be reproduced in the captured image acquired before or after the second corresponding point . A captured image , the second captured image and the estimated image related thereto are compared, and the candidate corresponding point that is determined to have a high probability of being the candidate corresponding point is restored as a candidate corresponding point again;
The image processing apparatus according to any one of claims 1 to 4 . The processing unit adjusts the acquisition of adjacent images so that the screen relative movement amount between adjacent images or between the captured images acquired before or after the captured image becomes a predetermined amount;
The image processing apparatus according to any one of claims 1 to 5 . The processing unit adjusts so that a pixel shift between adjacent images or between the captured image and the captured image acquired before or after that is a predetermined number;
The image processing apparatus according to any one of claims 1 to 5 . An image selection unit that selects the second image based on any one of an overlap rate between captured images obtained by the captured image acquisition unit, a baseline length, and the movement vector;
The image processing apparatus according to claim 5 . The feature point tracking unit performs an arrangement determination to confirm the arrangement of candidate corresponding points remaining after the defective one is removed,
The processing unit feeds back information on candidate corresponding points removed based on the long-term quality judgment to the placement judgment in the short-term quality judgment;
The image processing apparatus according to claim 1. The processing unit revives a candidate corresponding point that has once disappeared but is reproduced in the current frame by comparing the tracking position information with the estimated position of the candidate corresponding point in the long-term quality determination;
The image processing apparatus according to claim 1. The feature point tracking unit performs an arrangement determination to confirm the arrangement of candidate corresponding points remaining after the defective one is removed,
The processing unit feeds back information on candidate corresponding points restored based on the long-term quality determination to the placement determination in the short-term quality determination;
The image processing apparatus according to claim 10. A captured image acquisition step of sequentially acquiring captured images that are gradually changed by the captured image acquisition unit for an object that moves relative to the captured image acquisition unit ;
A feature extraction step of extracting feature points from the captured image obtained in the captured image acquisition step ;
A feature point tracking step of searching for candidate corresponding points corresponding to the feature points in a plurality of captured images acquired before or after the captured image from which the feature points are extracted; and tracking the feature points;
Wherein the candidate corresponding points, to confirm the corresponding points from the tracking result, and an arithmetic processing step of calculating coordinates of the photographing position or the object using been established corresponding points;
In the feature point tracking step, the candidate corresponding points corresponding to the feature points are searched for the photographed image from which the feature points are extracted and the plurality of photographed images sequentially acquired before or after that, and the candidate corresponding points searched for The feature point movement vector between adjacent captured images is formed from the positional relationship between the image and the feature point, the screen relative movement amount is obtained from the projective transformation using the adjacent captured image, and the movement vector and the screen relative movement amount are compared. A short-term pass / fail judgment to judge the pass / fail of the movement vector in real time, and delete candidate corresponding points that seem to be false corresponding points indicating abnormal movement based on the short-term pass / fail judgment, before or after A short-term pass / fail judgment routine that sequentially repeats the search for candidate correspondence points, formation of movement vectors, short-term pass / fail judgment, and deletion of candidate correspondence points that appear to be false correspondence points for multiple captured images The have;
The calculation processing step is performed by the short-term pass / fail judgment routine when the candidate corresponding point has obtained a movement amount sufficient to determine the candidate corresponding point as a corresponding point in the feature point tracking unit. The candidate corresponding point on each frame obtained by the projective transformation is obtained by performing projective transformation for each frame using the tracking position information of the candidate corresponding point in the current frame and the estimated projective transformation coefficient related to each frame. The estimated position of the candidate corresponding point is obtained by comparing the tracking position information obtained when the time or the number of frames is obtained with the estimated position of the candidate corresponding point. Long-term pass / fail judgment that eliminates candidate corresponding points that are likely to have a large residual based on the long-term pass / fail judgment, and determines candidate corresponding points that have a small residual as corresponding points. Have routines It calculates the coordinates of the photographing position or the object using are corresponding points;
Image processing method.