JP6306996B2 - VIDEO DATA PROCESSING METHOD, VIDEO DATA PROCESSING DEVICE, AND VIDEO DATA PROCESSING PROGRAM - Google Patents

Description

  The present invention relates to a video data processing method, a video data processing device, and a video data processing program.

  With the widespread use of smartphones and other camera-equipped devices, it has become easier for individuals to easily shoot and publish photos and videos. A huge archive is being formed. Along with this, there has been a rapid increase in recent years not only to watch such a large number of moving images but also to use them for new applications.

  A typical application collects landmarks and other images of sightseeing spots taken by individuals for each subject, uses them to estimate the 3D structure of the subject, and uses it as a virtual 3D object from any viewpoint. Some are available for viewing.

  Such an application generally restores three-dimensional information from an image group based on a classical image processing technique. Basically, the position and orientation of the camera used for imaging is estimated and the three-dimensional structure of the subject is restored by searching for corresponding points on an image different from feature point extraction in the image (see, for example, Non-Patent Document 1). . That is, what can be restored by this method is limited to a subject whose position and orientation shape does not change between images. On the other hand, the image group is usually taken asynchronously. Therefore, only a stationary subject is a restoration target.

  In addition, research on technology that expands a moving image as a data source is also underway (see, for example, Non-Patent Document 2). In these methods, the subject existing in the video is divided into a stationary object mainly of the background and other moving objects, and the camera position is restored while restoring the 3D structure using the method described above only for stationary objects. Estimate posture. For the moving object, smooth viewpoint movement is possible by performing simple interpolation processing only when the viewpoint is switched during video viewing without restoring the three-dimensional structure.

E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd, “Generic and real-time structure from motion using local bundle adjustment,” Image Vis. Comput., Vol. 27, no. 8, pp. 1178-1193, Jul. 2009. L. Ballan, G. J. Brostow, J. Puwein, and M. Pollefeys, “Unstructured Video-Based Rendering: Interactive Exploration of Casually Captured Videos,” in ACM SIGGRAPH 2010 papers, 2010, pp. 1-11.

  In order to estimate the position and orientation of multiple cameras using images, the basic matrix representing the relative orientation between cameras and the set of rotation matrix and translation vector are determined by the method described in Non-Patent Document 1. To do. However, the scale of the relative position calculated by the method using the image is indefinite, and the value of the translation vector is a value unrelated to the distance scale in the real space. Therefore, for example, even if the posture FAB between the cameras AB and the posture FBC between the cameras BC among the three cameras A, B, and C are determined, the posture FAC between the cameras AC cannot be obtained from them. In applications such as the above, the posture is determined for each camera combination, or the scale of each posture is adjusted by a method such as using the size of the subject common between the cameras as a reference. Yes.

  However, if each camera does not capture a sufficient common area, or if there is a common area but it is not possible to take enough reliable corresponding points, such a countermeasure cannot be taken. There's a problem. Also, even if the scale can be adjusted between multiple cameras, the actual size of the subject is known or the distance from the camera is known in order to match it with a distance scale in real space. Severe conditions are required.

  The present invention has been made in view of such circumstances, and estimates the absolute three-dimensional information of a subject and the position and orientation between two or more cameras from the obtained video and sensor information. An object of the present invention is to provide a video data processing method, video data processing apparatus, and video data processing program.

The present invention uses a multi-view video composed of a plurality of images acquired by a plurality of imaging means, and sensor information output by a sensor capable of estimating the position and orientation of a reference imaging means, and the position and orientation of the plurality of imaging means. A video data processing method for estimating a reference frame, which is an image at one time of a video obtained by the reference imaging means using the sensor information, and an image at a time different from the reference frame A position / orientation displacement estimation step for estimating a position / orientation displacement of the image pickup means between the reference frame and the reference frame, and the three-dimensional information of the point on the reference frame, the position of the corresponding point on the reference frame and the position / orientation A target that is an image at the same time as the reference frame in the video acquired by the three-dimensional information estimation step that estimates based on the displacement, the reference frame, and other imaging means Between the frame, and having a position and orientation estimation step of estimating the position and orientation of the other imaging means by using the three-dimensional information.

  In the present invention, the position and orientation estimation step estimates a relative position and orientation with an indefinite scale between the reference imaging unit and the other imaging unit between the reference frame and the target frame. The scale of the relative position and orientation with indefinite scale is determined based on the three-dimensional information.

  According to the present invention, in the position and orientation estimation step, a projection error from the reference imaging unit based on the three-dimensional information to the other imaging unit is minimized between the reference frame and the target frame. The relative position and orientation is estimated.

  In the present invention, the three-dimensional information is depth information, and the three-dimensional information estimation step estimates the depth information of corresponding points on the reference frame and the reference frame by triangulation based on the position and orientation displacement. It is characterized by.

The present invention uses a multi-view video composed of a plurality of images acquired by a plurality of imaging means, and sensor information output by a sensor capable of estimating the position and orientation of a reference imaging means, and the position and orientation of the plurality of imaging means. A video data processing apparatus for estimating the reference frame, which is an image at one time of the video obtained by the reference imaging means using the sensor information, and an image at a time different from the reference frame Position and orientation displacement estimation means for estimating the position and orientation displacement of the imaging means between the reference frame and the reference frame, and the three-dimensional information of the point on the reference frame and the position of the corresponding point on the reference frame and the position and orientation 3D information estimation means for estimating based on displacement, the reference frame, and a target frame that is an image at the same time as the reference frame in the video acquired by another imaging means Between, characterized in that it comprises a position and orientation estimation means for estimating the position and orientation of the other imaging means by using the three-dimensional information.

  The present invention is a video data processing program for causing a computer to execute the video data processing method.

  According to the present invention, it is possible to estimate the absolute three-dimensional information of a subject and the position and orientation between two or more cameras from the obtained video and sensor information.

It is a block diagram which shows the structure of the video data processing apparatus by one Embodiment of this invention. 2 is a flowchart showing a processing operation of the video data processing apparatus 100 shown in FIG. 1. It is a figure which shows the method of calculating | requiring the depth information of the point on an image by triangulation. It is a figure which shows angle (gamma) which the surface orthogonal to a light receiving element and the light ray which enters into an i-th pixel make.

  Hereinafter, a video data processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a video data processing apparatus according to the embodiment. As shown in FIG. 1, the video data processing apparatus 100 includes a video input unit 101, an input image memory 102, a sensor information input unit 103, a sensor information memory 104, a reference camera position and orientation displacement estimation unit 105, and a reference camera three-dimensional information estimation. Unit 106, an inter-camera position / orientation estimation unit 107, and a scale estimation unit 108.

  The video input unit 101 inputs a multi-view video to be processed, in particular, a video from a camera different from the reference camera. In the following description, the multi-view video to be processed is referred to as a processing multi-view video, in particular, the reference camera is referred to as a reference camera, the video is referred to as a reference video, another camera is referred to as a target camera, and the video is referred to as a target video. In addition, a frame of a reference video at a time at which processing is performed is referred to as a reference frame, and a frame of a target video at the same time is referred to as a target frame. A frame at another time before and after the processing time of the standard video is referred to as a reference frame. The input image memory 102 stores the processing target multi-viewpoint video input by the video input unit 101.

  The sensor information input unit 103 inputs sensor information corresponding to the reference camera. The sensor information memory 104 stores the sensor information input by the sensor information input unit 103.

  The reference camera position / posture displacement estimation unit 105 uses the sensor information stored in the sensor information memory 104 to estimate the position / posture displacement between the reference frame and the reference frame. The reference camera 3D information estimation unit 106 estimates 3D information of a part of the reference frame using the estimated position and orientation displacement, the reference frame and the reference frame stored in the input image memory 102.

  The inter-camera position / orientation estimation unit 107 estimates the inter-camera position / orientation using a reference frame and a target frame stored in the input image memory 102. The scale estimation unit 108 estimates the scale of the relative position and orientation between cameras with indefinite scale using the reference frame three-dimensional information, and determines the relative position and orientation between cameras.

  Next, the processing operation of the video data processing apparatus 100 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the video data processing apparatus 100 shown in FIG. Here, it is assumed that the reference video and the target video are two videos corresponding to different cameras capturing the same scene of the multi-view video.

  First, the video input unit 101 inputs a reference video and a target video and stores them in the input image memory 102. Also, the sensor information input unit 103 inputs sensor information corresponding to the reference camera and stores it in the sensor information memory 104 (step S101). The sensor information is information acquired by a sensor such as an acceleration sensor or a gyro sensor attached to the reference camera. In the present embodiment, the displacement of the self-position and orientation of the reference camera is determined based on the sensor information, and the sensor may belong to the same device as the reference camera and the position and orientation may be the same as the reference camera. Alternatively, it may be an object that is fixed to the reference camera and whose relative position and orientation with respect to the reference camera are known.

  The sensor information may be information obtained by recording sensor acquisition information at the same frame rate as the video, or may be information obtained by recording the acquisition information at a different frame rate. When the frame rates are different, the sensor information may be interpolated and used in accordance with the frame rate of the video, or information on a nearby time may be used. In the following, for simplicity, it is assumed that the frame rate of the video and the sensor information is the same, and the sensor information is a set of information at the same time as each frame of the video.

  In the following, the process of estimating the relative position and orientation between cameras for a certain frame in the video will be described. By performing this process for all frames (loop of steps S102 to S107), the relative position between the cameras for the entire video is described. The posture can be determined. Alternatively, it may be performed only for a specific frame, and the relative position and orientation may be determined based on interpolation processing for the remaining frames.

  In the processing for each frame, first, a reference frame and a reference frame, which are processing target frames stored in the input image memory 102, are determined between frames based on corresponding sensor information stored in the sensor information memory 104. The camera position / posture displacement is estimated (step S103). The recording time of the reference frame may be the previous time or the later time as long as it is different from the reference frame. The sensor information to be used may be any information that can estimate the displacement of the camera position and orientation between frames, such as a geomagnetic sensor and a GPS (Global Positioning System) sensor, in addition to the acceleration sensor and the gyro sensor described above. Good. As the position / orientation variation, a combination of a rotation matrix and a translation vector may be used, or only a translation vector may be used by a three-dimensional information estimation method.

Below, the case where an acceleration sensor is used is demonstrated. When the time of the base frame of the base camera O is t ₀ , the time of the reference frame is t ₁ , and the value of the acceleration sensor at time t is a (t), the relative position / posture displacement ΔF ₀ ^t1, between the frames ^t0 is represented by the following equation. Here, τ is a variable related to time and is an integral variable.

  Once the position / orientation displacement is estimated, the reference camera three-dimensional information estimation unit 106 uses the estimated position / orientation displacement and the reference frame and reference frame stored in the input image memory 102 to calculate the three-dimensional information of the reference frame. Estimate (step S104). Any 3D information may be used. It may be depth information that is a distance from a camera at a certain point, three-dimensional coordinates in the coordinate system of the reference camera, or coordinate values in other coordinate systems. Below, the case where depth information is used is demonstrated.

  Any method may be used to obtain the depth information. Assuming that there is a corresponding point between the two frames in the general method, and that the point does not move between the two frames and is stationary or has a negligibly small movement The depth information of the point can be obtained by a method such as triangulation.

  Here, an example of obtaining depth information of points on an image by triangulation will be described with reference to FIG. FIG. 3 is a diagram illustrating a method for obtaining depth information of points on an image by triangulation. For simplicity, the horizontal direction of the reference frame is the X axis, the vertical direction of the camera is the Y axis, and the depth direction of the camera is the Z axis, and the position and orientation displacement of the reference frame relative to the reference frame is a parallel movement in the X direction. Is assumed to be 1 meter.

で表される。 First, a set of corresponding points (x ₀ , x _r ) between the base frame and the reference frame is estimated. (X ₀ , x _r ) is observation in each frame of the point X in the real space, and is depth information obtained by the distance D from the camera in the reference frame of this point X. Any method may be used for estimating the corresponding points. Here, assuming that the angle of view of the camera is φ and the resolution in the horizontal direction is W, the light receiving element of the camera records the angle of view of φ at equal intervals with W pixels. As shown in FIG. 4, the angle γ formed by the light beam entering the i-th pixel and the surface orthogonal to the light receiving element is

It is represented by

で表される。このとき、ｘ_０が基準フレームにおいて第ｉ番目画素に位置し、ｘ_ｒが参照フレームにおいて第ｉ＋ｄ番目画素に位置するなら、参照フレームにおいて受光素子に直行する面とこの画素に入る光線がなす角δは

となる。 The angle α between the same light beam and the plane parallel to the light receiving element is

It is represented by At this time, if _x0 is located at the i-th pixel in the reference frame and _xr is located at the i + d-th pixel in the reference frame, the angle formed between the surface perpendicular to the light receiving element and the light ray entering this pixel in the reference frame δ is

It becomes.

で表される。これにより、点ＸからＰ_０までの距離ｒについて

が成立する。 Similarly, the angle β is

It is represented by Thus, for the distance r from point X to P ₀

Is established.

となる。位置姿勢変位が６自由度である場合にも同様の方法で深度情報を求めることができる。また、この点がフレーム間で動いているものと仮定して、その動きと深度情報を推定するなどしてもよい。深度情報はどのように表現されていてもよい。カメラからの距離を表す実数値でもよいし、離散値でもよいし、カメラ間の視差を表す値などでもよい。 Therefore, the required D is

It becomes. The depth information can be obtained by the same method even when the position and orientation displacement is 6 degrees of freedom. Also, assuming that this point is moving between frames, the motion and depth information may be estimated. The depth information may be expressed in any way. It may be a real value representing the distance from the camera, a discrete value, or a value representing the parallax between the cameras.

ここで、ｆは任意の定数であり、（ｆ_０，ｆ_ｔ）はカメラの焦点距離である。このとき、対応点の組（ｘ_０，ｙ_０）、（ｘ_ｒ，ｙ_ｒ）の三次元座標値は以下の連立方程式の解として与えられる。

ここで使用される併進ベクトルはセンサの値から求められたものであるため、画像から三次元情報を復元する場合と異なり三次元座標値は実世界における値と同じスケールを持つ。 Further, as an example, a case where the three-dimensional coordinates of a certain point in the reference camera coordinate system are used as the three-dimensional information will be described. First, the projection matrix of the base frame and the reference frame is defined based on the position and orientation displacement. When the position and orientation displacement is expressed by the rotation matrix R and the translation vector t, the projection matrices P ₀ and P _r of the two frames can be defined as follows.

Here, f is an arbitrary constant, and (f ₀ , f _t ) is the focal length of the camera. At this time, the three-dimensional coordinate values of the pair of corresponding points (x ₀ , y ₀ ) and (x _r , y _r ) are given as solutions of the following simultaneous equations.

Since the translation vector used here is obtained from the value of the sensor, the three-dimensional coordinate value has the same scale as the value in the real world, unlike the case where the three-dimensional information is restored from the image.

  Also, the movement distance obtained by an acceleration sensor is defined as the position and orientation displacement, the position and orientation are estimated based on the image between the base frame and the reference frame, and the indeterminate scale is determined based on the position and orientation displacement. Also good. Even when the position and orientation between the cameras are estimated based on the image, the three-dimensional coordinate value can be determined by the above simultaneous equations. However, the size of the translation vector is indefinite, and generally a solution with ‖t‖ = 1 is obtained. At this time, when the real distance 1 between the cameras is given by the position and orientation displacement, the three-dimensional coordinates on the real space scale can be determined as a value obtained by multiplying the value in the case of ‖t‖ = 1 by d.

  In addition, any three-dimensional information may be defined and estimated. The three-dimensional information may be estimated for all points on the image, or only feature points may be extracted and estimated only for those feature points. In addition, it may be estimated only for points used for inter-camera position / posture estimation, which will be described later, or after estimating three-dimensional information about points different from the points used for inter-camera position / posture estimation, The three-dimensional information may be determined by a method such as interpolation.

  The corresponding points between the two frames may be determined by any method. Each feature point may be extracted, and points with high feature correlation may be used as corresponding points. The position of the corresponding region on the other image is determined so as to minimize the square error sum of pixels for each small region. You may do it. In addition, a point used for estimating the position and orientation between cameras described later on the base frame is set as one of the corresponding points, and a point on the reference frame having the highest correlation with the point is set as the other corresponding point. Also good.

  When the three-dimensional information is estimated, the inter-camera position / orientation estimation unit 107 estimates the relative position / orientation between indeterminate cameras using the reference frame and the target frame stored in the input image memory 102 (step S105). Any method may be used for estimating the position and orientation. Further, the estimated position and orientation may be expressed in any way. It may be a conversion matrix between cameras, a projection matrix for each camera, or a combination of an internal parameter, a rotation matrix, and a translation vector.

As a general method of position and orientation estimation, there is a method called bundle adjustment (Non-Patent Document 1). In this method, a basic matrix representing a correspondence relationship between different cameras is determined from a set of a plurality of corresponding points existing on two images. There is the following relationship between a set of corresponding points (x ₀ , x _t ) on the image of the reference frame and the target frame and the basic matrix F.
(X ₀ , Fx _t ) = 0

対象行列ＥＥ^Ｔの最小固有値に対する単位固有ベクトルをｔとすることで求められる。ｆは任意の定数であり、基礎行列導出から焦点距離導出・回転並進導出の過程において計算を安定化させるために存在する。このとき回転行列Ｒは−ｔ×Ｅの特異値分解を経て計算されるが、ｔのスケールには影響を受けない。 The relative translation vector t and rotation matrix of the two cameras can be estimated from the basic matrix and the focal lengths (f ₀ , f _t ) of the camera. The focal length can be estimated from the basic matrix. However, at this time, the scale of the translation vector t is indefinite, and in general, a solution in which ‖t‖ = 1 is obtained. The translation vector t calculates the following basic matrix E

The unit eigenvector for the smallest eigenvalue of the target matrix EE ^T determined by a t. f is an arbitrary constant and exists to stabilize the calculation in the process of deriving the focal length and the rotational translation from the basic matrix. At this time, the rotation matrix R is calculated through singular value decomposition of −t × E, but is not affected by the scale of t.

  Finally, the scale estimation unit 108 estimates the scale of the relative position and orientation between cameras with indefinite scale using the reference frame 3D information, and determines the relative position and orientation between cameras (step S106). Any method may be used for the scale estimation. For example, if a corresponding point on the target frame is estimated for a point whose depth has already been estimated on the reference frame, and the corresponding point on the target frame is determined based on the depth information and the projection matrix for the same point, the error between the two It is also possible to determine the scale of the projection matrix so as to reduce.

  Alternatively, the coordinate scale may be determined so that an error with respect to the three-dimensional coordinate value based on the position and orientation with respect to the target frame is reduced for a point for which the three-dimensional coordinate value has been estimated in the reference frame. In addition, the three-dimensional coordinate value based on the determined scale does not need to be a real space value. When the present application is used to fix a scale between a large number of cameras, an arbitrary value is further added to the real space value. A scale multiplied by a coefficient may be used.

  Alternatively, the relative position and orientation between cameras of a fixed scale may be determined directly using the three-dimensional information without estimating the relative position and orientation between cameras with indefinite scale. For example, by using a point for which three-dimensional information has been estimated as a corresponding point used when determining the position and orientation between cameras, it is possible to fix the scale and determine the position and orientation between cameras. As described above, in the camera position and orientation estimation problem in the multi-viewpoint video, the 3D information of the reference camera is estimated using the sensor information for the reference camera, and another 3D information is used using the estimated 3D information. Estimate the position and orientation between two cameras on a real-world scale by determining the position and orientation scale with the camera, or estimate the position and orientation of each camera in a multi-view video while maintaining consistency be able to.

  Note that the order of some processes shown in FIG. 2 may be changed.

  As described above, in obtaining the relative position and orientation between cameras using images acquired by a plurality of cameras, an acceleration sensor, a gyro sensor, or the like associated with the camera is used to determine the orientation displacement between the previous and next frames in a certain reference camera. The 3D information is used to estimate the 3D information for the points on the image of the reference camera based on the estimated displacement of the real space scale. The scale of indefinite relative position and posture between the camera and another camera can be adjusted to an absolute value.

  You may make it implement | achieve the video data processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  When estimating the position and orientation between two cameras on a real-world scale, or when estimating the position and orientation of each camera in a multi-view video, if there is not enough common area between all cameras, It can be applied to applications where it is indispensable to maintain consistency of relative positions and postures when there is an area but sufficient corresponding points cannot be taken.

  DESCRIPTION OF SYMBOLS 101 ... Video input part, 102 ... Input image memory, 103 ... Sensor information input part, 104 ... Sensor information memory, 105 ... Reference | standard camera position and orientation displacement estimation part, 106 ... Reference | standard Camera three-dimensional information estimation unit, 107 ... inter-camera position and orientation estimation unit, 108 ... scale estimation unit

Claims (6)

Estimating the position and orientation of a plurality of imaging means using a multi-view video composed of a plurality of videos acquired by a plurality of imaging means and sensor information output by a sensor capable of estimating the position and orientation of the reference imaging means. A video data processing method,
Using the sensor information, the imaging means between the reference frame that is an image at one time of the video acquired by the imaging means serving as the reference and the reference frame that is an image at a time different from the reference frame. A position and orientation displacement estimation step for estimating a position and orientation displacement;
A three-dimensional information estimation step for estimating three-dimensional information of a point on the reference frame based on the position of the corresponding point on the reference frame and the position and orientation displacement;
Between the reference frame and a target frame that is an image at the same time as the reference frame in the video acquired by another imaging unit, the position and orientation of the other imaging unit are estimated using the three-dimensional information. And a position / orientation estimation step. The position and orientation estimation step includes:
Estimating an indefinite scale relative position and orientation between the reference imaging unit and the other imaging unit between the reference frame and the target frame, and based on the three-dimensional information, the scale indefinite relative 2. The video data processing method according to claim 1 , wherein a scale of the position and orientation is determined. The position and orientation estimation step includes:
A relative position and orientation at which a projection error from the reference imaging unit to the other imaging unit based on the three-dimensional information is minimized is estimated between the reference frame and the target frame. The video data processing method according to claim 1 . The three-dimensional information is depth information,
The three-dimensional information estimation step includes:
Video data processing method according to any one of claims 1 to 3, characterized in that to estimate the depth information of the corresponding point on the reference frame and the reference frame by triangulation based on the position and orientation displacement. Estimating the position and orientation of a plurality of imaging means using a multi-view video composed of a plurality of videos acquired by a plurality of imaging means and sensor information output by a sensor capable of estimating the position and orientation of the reference imaging means. A video data processing device,
Using the sensor information, the imaging means between the reference frame that is an image at one time of the video acquired by the imaging means serving as the reference and the reference frame that is an image at a time different from the reference frame. Position and orientation displacement estimating means for estimating position and orientation displacement;
Three-dimensional information estimation means for estimating three-dimensional information of a point on the reference frame based on the position of the corresponding point on the reference frame and the position and orientation displacement;
Between the reference frame and a target frame that is an image at the same time as the reference frame in the video acquired by another imaging unit, the position and orientation of the other imaging unit are estimated using the three-dimensional information. A video data processing apparatus comprising: a position and orientation estimation unit. A video data processing program for causing a computer to execute the video data processing method according to any one of claims 1 to 4 .

Images