JP4809316B2

JP4809316B2 - Image generating apparatus, program, and recording medium

Info

Publication number: JP4809316B2
Application number: JP2007283638A
Authority: JP
Inventors: 雅浩上野; 豊國田; 愛磯貝
Original assignee: Nippon Telegraph and Telephone Corp; NTT Inc USA
Current assignee: NTT Inc; NTT Inc USA
Priority date: 2007-10-31
Filing date: 2007-10-31
Publication date: 2011-11-09
Anticipated expiration: 2027-10-31
Also published as: JP2009110397A

Description

本発明は、複数台のカメラで撮像した画像を用いて、任意の視点の２次元画像を生成する技術に関する。 The present invention relates to a technique for generating a two-dimensional image at an arbitrary viewpoint using images captured by a plurality of cameras.

従来の画像生成方法について、以下に説明する（たとえば、非特許文献１）。従来技術の代表的な全体処理の流れを図３を用いて説明する。以下の第１のステップ〜第１４のステップは図３のＳ１〜Ｓ１４に対応する。 A conventional image generation method will be described below (for example, Non-Patent Document 1). A typical overall processing flow of the prior art will be described with reference to FIG. The following first to fourteenth steps correspond to S1 to S14 in FIG.

まず、視点の異なる複数のカメラを用いて同一の物体や風景を撮影する（第１のステップ）。
次に、ユーザが見たい視点を空間中に設置する（第２のステップ）。
次に、ユーザが見たい空間的な範囲内に複数の仮想的な面で構成した層を設置する（第３のステップ）。
次に、ユーザが見たい視野を表す仮想的な面を設置する（第４のステップ）。
次に、ユーザが見たい精細度となるように、ユーザが見たい視野を表す仮想的な面上に、複数の画素を配置する（第５のステップ）。
次に、画素を１つ選び、また、仮想的な面を１つ選び、仮想視点とその画素を通る直線（視線）と、その仮想的な面との交点に対して、カメラで撮像した画像上におけるそれぞれの対応点を計算する（第６のステップ）。
次に、複数のカメラで撮像したそれぞれの画像上の対応点およびその付近の画像同士の類似性から、１つの仮想的な面との交点における、物体の表面が存在する程度を計算する（第７のステップ）。
次に、複数のカメラで撮像したそれぞれの画像上の対応点における画素の色と、複数のカメラとユーザが見たい視点との幾何学的な関係から、１つの仮想的な面との交点における、物体の表面の色を計算する（第８のステップ）。
次に、選んだ１つの仮想的な面において、ユーザが見たい視野を表す仮想的な面上の全画素に対応する全交点について、第６〜８のステップを行うことで、物体の表面が存在する程度と色（第１の色）を計算する（第９のステップ）。
次に、層の中の全ての仮想的な面において、第６〜９のステップを行うことで、物体の表面が存在する程度と色（第１の色）を計算する（第１０のステップ）。
次に、仮想視線上の全て交点における物体の表面が存在する程度を足し合わせる（第１１のステップ）。
次に、仮想視線上の各交点における物体の表面が存在する程度を、それら全てを足し合わせたもので割り、それらの値を各交点における物体の表面が存在する確率とする（第１２のステップ）。
次に、各仮想的な面の交点における、物体の存在する確率と、物体の表面の色（第１の色）との積を、それらの交点における色（第４の色）とする（第１３のステップ）。
最後に、仮想視線上の全ての交点における色（第４の色）を足し合わせ、その色を、ユーザが見たい視野を表す仮想的な面上の画素の色（第３の色）とする（第１４のステップ）。 First, the same object or landscape is photographed using a plurality of cameras with different viewpoints (first step).
Next, the viewpoint that the user wants to see is placed in the space (second step).
Next, a layer composed of a plurality of virtual planes is installed within a spatial range that the user wants to see (third step).
Next, a virtual surface representing the visual field that the user wants to see is installed (fourth step).
Next, a plurality of pixels are arranged on a virtual surface representing the visual field that the user wants to see so as to obtain the definition that the user wants to see (fifth step).
Next, one pixel is selected, and one virtual plane is selected. An image captured by the camera at the intersection of the virtual viewpoint and a straight line (line of sight) passing through the pixel and the virtual plane Each corresponding point in the above is calculated (sixth step).
Next, the degree to which the surface of the object exists at the intersection with one virtual plane is calculated from the similarity between the corresponding points on the images captured by a plurality of cameras and the images in the vicinity thereof (first). Step 7).
Next, from the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see, at the intersection of one virtual plane The color of the surface of the object is calculated (eighth step).
Next, in the selected one virtual surface, the sixth to eighth steps are performed on all the intersection points corresponding to all the pixels on the virtual surface representing the visual field that the user wants to see, so that the surface of the object is The degree of existence and the color (first color) are calculated (9th step).
Next, by performing the sixth to ninth steps on all the virtual surfaces in the layer, the degree and color (first color) at which the surface of the object exists are calculated (tenth step). .
Next, the degree to which the surface of the object exists at all the intersections on the virtual line of sight is added (11th step).
Next, the degree to which the surface of the object exists at each intersection on the virtual line of sight is divided by the sum of all of them, and these values are used as the probability that the surface of the object at each intersection exists (Twelfth Step). ).
Next, the product of the probability that an object exists at the intersection of each virtual plane and the color of the surface of the object (first color) is used as the color (fourth color) at the intersection (first color). 13 steps).
Finally, the colors (fourth color) at all the intersections on the virtual line of sight are added together, and the color is set as the color (third color) of the pixel on the virtual plane representing the visual field that the user wants to see. (14th step).

図１は画像生成法を説明するための構成図である。１００１はユーザが見たい画像を得るために便宜的に記した仮想カメラ、１００２は１００１の仮想カメラのレンズ、１００３はユーザが見たい視点（仮想視点）であり、１００２の仮想カメラのレンズの中心、１００４はユーザが見たい視野を表す仮想平面であり、１００１の仮想カメラの撮像素子、１００５はユーザが見たい視野を表す仮想平面の原点であり、１００４の仮想視点の撮像素子の原点、１００６はユーザが見たい視野を表す仮想平面上の画素であり、１００１の仮想カメラにおける後述の１０１２の仮想平面と視線との交点の結像点、１０１０はユーザが見たい視線（仮想視線）、１０１１はユーザが見たい空間的な範囲内の仮想平面群、１０１２は１０１１の層を形成している仮想平面と仮想視線の交点群、１１０１はカメラ、１１０２はカメラ１１０１のレンズ、１１０３はレンズ１１０２のレンズ中心、１１０４はカメラ１１０１の撮像素子、１１０５は撮像素子１１０４の原点、１１０６は仮想平面と仮想視線の交点の結像点、１１１０はカメラ視線、である。１１０１はカメラＣ_１であり、１１０２〜１１０６はカメラＣ_１に関するものである。カメラはＮ台あるとして、それぞれＣ_１〜Ｃ_Ｎと名付ける。Ｃ_２〜Ｃ_Ｎのそれぞれの構成はカメラＣ_１と同じである。 FIG. 1 is a configuration diagram for explaining an image generation method. 1001 is a virtual camera described for convenience in order to obtain an image that the user wants to see, 1002 is a lens of the virtual camera 1001, 1003 is a viewpoint (virtual viewpoint) that the user wants to see, and the center of the lens of the virtual camera 1002 , 1004 is a virtual plane that represents the visual field that the user wants to see, 1001 is the virtual camera image sensor, 1005 is the virtual plane origin that represents the user's visual field, and 1004 is the virtual viewpoint image sensor origin, 1006 Is a pixel on a virtual plane that represents a visual field that the user wants to see. An imaging point of an intersection of a virtual plane 1012 (described later) and a line of sight in the virtual camera 1001, 1010 is a line of sight (virtual line of sight) 1011 that the user wants to see Is a group of virtual planes within a spatial range that the user wants to see, 1012 is a group of intersections of virtual planes forming a layer of 1011 and virtual lines of sight, 1101 is Mera, 1102 is the lens of the camera 1101, 1103 is the lens center of the lens 1102, 1104 is the image sensor of the camera 1101, 1105 is the origin of the image sensor 1104, 1106 is the imaging point of the intersection of the virtual plane and the virtual line of sight, 1110 is the camera Line of sight. 1101 is a camera _{C 1,} 1102-1106 relates to camera _{C 1.} The camera as being N units, named respectively _C 1 -C _N. Each configuration of the C ₂ -C _N are the same as the camera _{C 1.}

以下、ステップに分けて動作を説明する。 Hereinafter, the operation will be described in steps.

ステップ１：［複数のカメラを用いて同一物体や風景を撮影］
Ｎ台のカメラで、同一物体や風景を撮影する。このとき、各カメラＣ_ｉ（ｉ＝１，．．．，Ｎ）において、コンピュータビジョンの分野で知られているカメラの内部に関するパラメタ（カメラ内部パラメタ）と、カメラの位置、姿勢に関するパラメタ（カメラ外部パラメタ）は既知とする。カメラ内部パラメタは、行列表記すると、たとえば、以下のようなものである。 Step 1: [Shoot the same object or landscape using multiple cameras]
Photograph the same object or landscape with N cameras. At this time, in each camera C _i (i = 1,..., N), parameters relating to the inside of the camera (camera internal parameters) and parameters relating to the position and orientation of the camera (cameras) known in the field of computer vision External parameters are assumed to be known. The camera internal parameters are, for example, as follows in matrix notation.

Ａ_Ｃｉをカメラ内部行列といい、３行３列の行列である。ここで、α_ｕ，Ｃｉは焦点距離を撮像素子の横軸の画素ピッチで割ったもの、α_ｖ，Ｃｉは焦点距離を撮像素子の縦軸の画素ピッチで割ったもの、θ_Ｃｉは撮像素子の縦軸と横軸の成す角度、ｕ_Ｃｉ０、ｖ_Ｃｉ０は撮像素子とレンズ光軸の交点の座標であり、画素単位で表記し、撮像した画像の左上を原点とする座標、である。 A _Ci is called a camera internal matrix and is a matrix of 3 rows and 3 columns. Here, α _{u, C i} is the focal length divided by the horizontal pixel pitch of the image sensor, α _{v, C} _i is the focal length divided by the vertical pixel pitch of the image sensor, and θ _Ci is the image sensor. The angle between the vertical axis and the horizontal axis, u _Ci0 and v _Ci0 are the coordinates of the intersection of the imaging element and the lens optical axis, and are expressed in pixel units and are the coordinates with the upper left corner of the captured image as the origin.

なお、ｕ_Ｃｉ0、ｖ_Ｃｉ0で使用される座標系をカメラＣ_ｉにおけるディジタル画像座標といい、カメラの撮像素子上の２次元の斜交座標であり、撮像素子の画素間隔を単位とし、撮像素子の横軸、縦軸をその軸方向とし、撮像素子の左上を原点とするものであり、カメラごとに固有のものとなる。 Incidentally, u _Ci0, v refers to the coordinate system and a digital image coordinates of the camera C _i to be used in _Ci0, a two-dimensional oblique coordinates on the image pickup device of the camera, a unit pixel interval of the imaging device, the imaging device The horizontal axis and the vertical axis are the axial directions, and the upper left of the image sensor is the origin, which is unique to each camera.

また、カメラの外部パラメタを行列表記すると、たとえば、以下のようなものとなる。 Further, when the external parameters of the camera are expressed in a matrix, for example, the following is obtained.

Ｒ_Ｃｉはカメラ回転行列といい、３行３列の行列であり、ｔ_Ｃｉはカメラ平行並進ベクトルといい、３次元のベクトルである。これらのパラメタを説明する前に、世界座標系について説明する。世界座標系とは、空間全体を表す３次元の直交座標系であり、原点、単位は適宜ユーザが設定する。この世界座標系を用いると、それぞれのパラメタは以下のように説明できる。 R _Ci is a camera rotation matrix, which is a 3 × 3 matrix, and t _Ci is a camera parallel translation vector, which is a three-dimensional vector. Before describing these parameters, the world coordinate system will be described. The world coordinate system is a three-dimensional orthogonal coordinate system that represents the entire space, and the origin and unit are appropriately set by the user. Using this world coordinate system, each parameter can be explained as follows.

ｒ_Ｃｉｘ、ｒ_Ｃｉｙ、ｒ_Ｃｉｚは、カメラＣ_ｉのレンズ中心を原点とした３次元の直交座標系（これをカメラ座標系という）の各軸の単位ベクトルを、世界座標系で表したベクトル、ｔ_Ｃｉｘ、ｔ_Ｃｉｙ、ｔ_Ｃｉｚは、カメラ座標系における、世界座標系の原点の座標を表すベクトルの要素、である。 r _Cix , r _Ciy , and r _Ciz are vectors in which a unit vector of each axis of a three-dimensional orthogonal coordinate system (referred to as a camera coordinate system) with the lens center of the camera C _i as an origin is represented in the world coordinate system, t _Cix , t _Ciy , and t _Ciz are vector elements representing the coordinates of the origin of the world coordinate system in the camera coordinate system.

ステップ２：［ユーザが見たい視点を設置］
ユーザが自分の目を持っていきたい場所を決め、そこに仮想的に視点（以下、仮想視点と表記）１００３を設置する。 Step 2: [Set the viewpoint that the user wants to see]
A user decides where he / she wants to take his / her eyes, and virtually installs a viewpoint (hereinafter referred to as a virtual viewpoint) 1003 there.

ステップ３：［層状の仮想的な面を設置]
ユーザが見たい３次元的な範囲に、複数の仮想的な面で層状に構成される平面群１０１１を設置する。 Step 3: [Set up a virtual layered surface]
A plane group 1011 configured in layers in a plurality of virtual planes is installed in a three-dimensional range that the user wants to see.

ステップ４：［ユーザが見たい視野を表す仮想的な面を設置]
仮想視点１００３から見た視野内であって、ユーザが生成したい画像の範囲を表す仮想的な面１００４を設置する。これは、仮想視点１００３をレンズ中心としたレンズ１００２を持つ仮想カメラ１００１の撮像素子と見なすことができるので、以下、仮想カメラの撮像素子と言う。 Step 4: [Install a virtual surface that represents the field of view that the user wants to see]
A virtual plane 1004 that is within the field of view as viewed from the virtual viewpoint 1003 and that represents the range of the image that the user wants to generate is installed. This can be regarded as an image sensor of a virtual camera 1001 having a lens 1002 with the virtual viewpoint 1003 as the center of the lens, and is hereinafter referred to as an image sensor of a virtual camera.

ステップ５：［ユーザが見たい精細度となるように、仮想カメラの撮像素子面上に、画素を配置]
最終的にユーザが見たい画像の精細度を決め、それを満足するように、仮想カメラの撮像素子１００４の面上の画素を配置する。 Step 5: [Place pixels on the image sensor surface of the virtual camera so that the user wants to see the fineness]
Finally, the resolution of the image that the user wants to see is determined, and the pixels on the surface of the imaging device 1004 of the virtual camera are arranged so as to satisfy the definition.

上記ステップ２〜５まで決めると、ユーザが見たい視点に設置した仮想カメラのパラメタが決まる。この仮想カメラの内部／外部パラメタを、たとえば、以下のように表す。 When the above steps 2 to 5 are determined, the parameters of the virtual camera installed at the viewpoint that the user wants to see are determined. The internal / external parameters of this virtual camera are expressed as follows, for example.

ここで、Ｃ_Ｖは仮想視点カメラを表す。上記式の内容は、（３．１）〜（３．３）式と同様であるので、説明は省略する。 Here, _CV represents a virtual viewpoint camera. Since the contents of the above formula are the same as the formulas (3.1) to (3.3), the description thereof is omitted.

ステップ６：［仮想視点と画素の１つを通る直線（視線）と１つの仮想的な面の交点に対して、各カメラ画像上の対応点を計算］
仮想カメラの撮像素子１００４の面上の画素１００６を１つ選び、また、ユーザが見たい空間的な範囲内の仮想平面群１０１１の中の層を１つ選び、仮想視点とその選んだ画素を通る直線（仮想視線）と、その仮想的な面との交点（仮想平面と仮想視線との交点群の中の１つ）に対して、複数のカメラで撮像した複数の画像上におけるそれぞれの対応点（図１では、カメラＣ_１については仮想平面と視線の交点の結像点１１０６で表記）を計算する。 Step 6: [Calculate corresponding points on each camera image for the intersection of a virtual viewpoint and a straight line (line of sight) passing through one of the pixels and one virtual plane]
One pixel 1006 on the surface of the image sensor 1004 of the virtual camera is selected, and one layer in the virtual plane group 1011 within the spatial range desired by the user is selected, and the virtual viewpoint and the selected pixel are selected. Correspondence on a plurality of images taken by a plurality of cameras with respect to a crossing line (virtual line of sight) and an intersection of the virtual plane (one of a group of intersections of the virtual plane and the virtual line of sight) (in Figure 1, the camera C ₁ are denoted by the imaging point 1106 of the intersection of the imaginary plane and the line of sight) the point is calculated.

図１では、カメラＣ_１にのみ説明のための番号が付いているが、全カメラＣ_ｉ（ｉ＝１，．．．，Ｎ）ともに構成は同様であるので、以下では、カメラを区別せずにＣ_ｉと表記する。 In FIG. 1, only the camera C ₁ has a number for explanation, but the configuration is the same for all the cameras C _i (i = 1,..., N). referred to as _{C i} without.

たとえば、仮想カメラＣ_ｖの撮像素子１００４上の画素１００６の、仮想カメラＣ_ｖのディジタル画像座標系での座標をｍ_Ｃｖ［ｕ_Ｃｖｖ_Ｃｖ］^Ｔ、カメラＣ_１の撮像素子１１０４上の対応点１１０６の、カメラＣ_ｉのディジタル画像座標系での座標をｍ_{Ｃｖ,Ｌｋ,Ｃｉ}＝［ｕ_{Ｃｖ,Ｌｋ,Ｃｉ} ｖ_{Ｃｖ,Ｌｋ,Ｃｉ}］とすると、 For example, the virtual camera _{C v} pixel 1006 on the imaging device 1004, coordinates _m Cv in the digital image coordinate system of the virtual camera _{_{_{^{C v [u Cv v Cv]}}}} T, corresponding points on the image pickup device 1104 of the camera _{C 1} When the coordinates of the digital image coordinate system of the camera C _{i in} 1106 is m _{Cv, Lk,} C _i = [u _{Cv, Lk,} C _i v _{Cv, Lk,} C _i ],

となる。 It becomes.

ここで、^~ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}、^~ｍ_Ｃｖはそれぞれｍ_{Ｃｖ，Ｌｋ，Ｃｉ}、ｍ_Ｃｖの同次座標であり、それぞれ、^~ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}＝［ｕ_{Ｃｖ，Ｌｋ，Ｃｉ} ｖ_{Ｃｖ，Ｌｋ，Ｃｉ} １］^Ｔ、^~ｍ_Ｃｖ＝［ｕ_Ｃｖｖ_Ｃｖ１］^Ｔ、である。なお、（３．７）式のイメージに記載されているような「〜」が上についた「ｍ」等を文章中では「^~ｍ」等と記載する。Ｈ_{Ｌｋ，Ｃｉ，Ｃｖ}はホモグラフィ行列といい、 Here, ^~ m _{Cv, Lk, Ci} and ^~ m _Cv are the homogeneous coordinates of m _{Cv, Lk, Ci} and m _Cv , respectively, and ^~ m _{Cv, Lk, Ci} = [u _{Cv, Lk, Ci} v, respectively. _{Cv, Lk, Ci} 1] ^T , ^~ m _Cv = [u _Cv v _Cv 1] ^T. It should be noted, is described as ^"~ m", and the like in a sentence (3.7) formula is "~" such as those described with above in image "m", and the like. _{HLk, Ci, and Cv} are called homography matrices,

である。ｓはスカラ値であり、ｓが（３．７）式右辺のＨ_{Ｌｋ，Ｃｉ，Ｃｖ}の三行目を示すベクトルと^~ｍ_Ｃｖとの内積値、である。ここで、Ｒ_{Ｃｉ，Ｃｖ}とｔ_{Ｃｉ，Ｃｖ}は仮想カメラＣ_ｖのカメラ座標系からカメラＣ_１のカメラ座標系へ変換する際の回転行列と平行並進ベクトルであり、 It is. s is a scalar value, and s is an inner product value of a vector indicating the third row of _{HLk, Ci, and Cv on} the right side of Expression (3.7) and ^~ _mCv . _{Here, R Ci, Cv} and _{t Ci, Cv} is parallel translation vector and rotation matrix when converting from the camera coordinate system of the virtual camera _{C v} of the camera _{C 1} to the camera coordinate system,

である。ｎは仮想カメラＣ_ｖのカメラ座標系におけるｋ層の仮想平面の法線ベクトル（仮想カメラの撮像素子１００４の面とｋ層の仮想平面が平行ならｎ＝［００１］^Ｔ）、ｄは仮想視点１００３からｋ層の仮想平面までの距離、である。 It is. n is the normal vector of the virtual plane in k layer in the camera coordinate system of the virtual camera C _v (surface and k layer virtual plane if parallel n = [001] T of the image pickup device 1004 of the virtual camera), ^d is the virtual viewpoint The distance from 1003 to the virtual plane of the k layer.

ステップ７：［対応点上の画像の類似点から、１つの仮想的な面の交点における、物体の表面が存在する程度を計算]
仮想カメラの撮像素子の画素に関して、ユーザが見たい空間的な範囲内の仮想平面群の中の１層の平面と、今注目している仮想カメラの撮像素子の画素の中心から引かれた仮想視線との交点における、物体の表面が存在する程度を計算する。ここでは、その交点に対応する各カメラの撮像素子上の点とその近傍の点のテクスチャが類似しているほど物体の表面が存在する程度が高いとする計算を行う。 Step 7: [Calculate the degree to which the surface of the object exists at the intersection of one virtual plane from the similarities of the images on the corresponding points]
As for the pixels of the image sensor of the virtual camera, a virtual plane drawn from the center of the pixel of the image sensor of the virtual camera that is now focused on the virtual plane group in the virtual plane group within the spatial range that the user wants to see The degree to which the surface of the object exists at the intersection with the line of sight is calculated. Here, the calculation is performed such that the surface of the object is higher as the texture of the point on the image sensor of each camera corresponding to the intersection and the point near the point are similar.

仮想平面上の交点の近傍の点とは、たとえば、今注目している仮想カメラの撮像素子上の画素の近傍の画素の中心から引かれる仮想カメラの視線と仮想平面との交点としても良い。 The point in the vicinity of the intersection point on the virtual plane may be, for example, the intersection point of the virtual camera's line of sight drawn from the center of the pixel in the vicinity of the pixel on the imaging element of the virtual camera of interest.

この様子を図２（ａ）に示す。５００３は仮想カメラのレンズ中心、５００６は仮想カメラの撮像素子の注目画素の中心、５００６−１は仮想カメラの撮像素子の注目画素の近傍の中心、５０１０は仮想カメラの注目画素を通る仮想視線、５０１０−１は仮想カメラの注目画素の近傍画素を通る仮想視線、５０１２はｋ層の仮想平面と注目画素を通る仮想視線の交点、５０１２−１はｋ層の仮想平面と注目画素の近傍画素を通る仮想視線の交点、５０１３はユーザが見たい空間的な範囲内の仮想平面群のｋ層、である。注目画素５００６と近傍画素５００６−１は、仮想カメラの撮像素子上の画素であり、レンズ中心５００３を通る視線５０１０、５０１０−１によって、それぞれ、ｋ層の仮想平面５０１３上の点に対応点を作る。図では、近傍画素が１つしか示していないが、注目画素５００６の周りの画素群や、さらにその周りの画素群を近傍の画素としても良い。 This is shown in FIG. 5003 is the center of the lens of the virtual camera, 5006 is the center of the target pixel of the image sensor of the virtual camera, 5006-1 is the center near the target pixel of the image sensor of the virtual camera, 5010 is a virtual line of sight passing through the target pixel of the virtual camera, Reference numeral 5010-1 denotes a virtual line of sight passing through the neighboring pixel of the target pixel of the virtual camera, 5012 denotes an intersection of the virtual plane of the k layer and the virtual sight line passing through the target pixel, and 5012-1 denotes a pixel near the target layer of the k layer. The intersection of the virtual lines of sight through, 5013, is the k layer of the virtual plane group within the spatial range that the user wants to see. The pixel of interest 5006 and the neighboring pixel 5006-1 are pixels on the image sensor of the virtual camera, and corresponding points are points corresponding to points on the virtual plane 5013 of the k layer, respectively, by the lines of sight 5010 and 5010-1 passing through the lens center 5003. create. Although only one neighboring pixel is shown in the figure, a pixel group around the pixel of interest 5006 and a pixel group around it may be used as neighboring pixels.

これらｋ層の仮想平面上の交点とその近傍の点に対応する、各カメラの撮像素子上の点も、仮想平面上の点と各カメラのレンズの中心を通るカメラ視線によって、決まる。 The points on the image sensor of each camera corresponding to the intersections of the k layers on the virtual plane and the points in the vicinity thereof are also determined by the point of view on the virtual plane and the camera line of sight passing through the center of each camera lens.

この様子を図２（ｂ）に示す。５００６は仮想カメラの撮像素子の注目画素の中心、５０１０は仮想カメラの注目画素を通る仮想視線、５０１１は仮想カメラの撮像素子の画素、５０１２はｋ層の仮想平面と注目画素を通る仮想視線の交点、５０１３はユーザが見たい空間的な範囲内の仮想平面群のｋ層、５１０６は仮想平面と仮想視線の交点の結像点、５１１０はカメラ視線、５１１１は仮想平面と仮想視線の交点の結像点を中心とする、撮像素子の画素と同じ大きさの四角形、５１１２は仮想平面と仮想視線の交点の結像点の近傍の撮像素子上の画素群、である。 This is shown in FIG. 5006 is the center of the target pixel of the image sensor of the virtual camera, 5010 is the virtual line of sight passing through the target pixel of the virtual camera, 5011 is the pixel of the image sensor of the virtual camera, and 5012 is the virtual line of sight passing through the virtual plane of the k layer and the target pixel. Intersection, 5013 is the k layer of the virtual plane group within the spatial range that the user wants to see, 5106 is the imaging point of the intersection of the virtual plane and the virtual line of sight, 5110 is the camera line of sight, 5111 is the intersection of the virtual plane and the virtual line of sight A quadrangle 5112 having the same size as the pixel of the image sensor centered on the image point is a pixel group on the image sensor in the vicinity of the image point at the intersection of the virtual plane and the virtual line of sight.

仮想カメラの撮像素子上の画素５０１１は、その画素中心５００６と仮想カメラ視線５０１０によって、ｋ層の仮想平面の上に交点５０１２で対応づけられている。そのｋ層の仮想平面の上に交点５０１２に対して、その交点５０１２とカメラのレンズ中心を通るカメラ視線５１１０によって、カメラの撮像素子上に交点５０１２が結像する点５１０６に対応付けられる。 A pixel 5011 on the imaging element of the virtual camera is associated with an intersection 5012 on the virtual plane of the k layer by the pixel center 5006 and the virtual camera line of sight 5010. With respect to the intersection 5012 on the virtual plane of the k layer, the intersection 5012 and the camera line of sight 5110 passing through the center of the lens of the camera are associated with the point 5106 where the intersection 5012 forms an image on the image sensor of the camera.

このように、仮想カメラの撮像素子上の画素５０１１とｋ層の仮想平面上の交点５０１２とカメラの撮像素子上の結像点５１０６は対応付けられるが、結像点５１０６はカメラ撮像素子の画素の中心とは一般には一致しない。その結像点５１０６の色を得るには、たとえば、結像点の近傍の画素値を使う。 As described above, the pixel 5011 on the imaging device of the virtual camera, the intersection 5012 on the virtual plane of the k layer, and the imaging point 5106 on the imaging device of the camera are associated, but the imaging point 5106 is a pixel of the camera imaging device. In general, it does not coincide with the center of. In order to obtain the color of the image point 5106, for example, a pixel value near the image point is used.

この様子を図２（ｃ）に示す。５１０６は仮想平面と仮想視線の交点の結像点、５１１１は仮想平面と仮想視線の交点の結像点を中心とする、撮像素子の画素と同じ大きさの四角形、５１１２は仮想平面と仮想視線の交点の結像点の近傍の撮像素子上の画素群、である。 This is shown in FIG. 5106 is an imaging point at the intersection of the virtual plane and the virtual line of sight, 5111 is a quadrangle having the same size as the pixel of the image sensor centering on the imaging point at the intersection of the virtual plane and the virtual line of sight, and 5112 is the virtual plane and the virtual line of sight Is a pixel group on the image sensor in the vicinity of the image forming point of the intersection.

結像点５１０６の色は、たとえば、結像点を中心とする四角形５１１１が重なっているカメラの撮像素子の画素群５１１２の各画素の面積比で、それらの画素の色を足し合わせるという、バイリニア法で得ることができる。この方法は、たとえば、図２（ｃ）に示すように、四角形５１１１が撮像素子の画素５１１２の境界で、上下の辺がｓ：１-ｓに、左右の辺がｔ：１-ｔにそれぞれ内分されていた場合、近傍の画素の座標をｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ１}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ２}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ３}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ４}、として、それぞれの画素の色をＴ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ１}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ２}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ３}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ４}］とすると、結像点の色Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}］は、以下のようになる。 The color of the image formation point 5106 is, for example, the bilinearity in which the color of these pixels is added by the area ratio of each pixel of the pixel group 5112 of the image sensor of the camera on which the quadrangle 5111 centering on the image formation point overlaps. Can be obtained by law. In this method, for example, as shown in FIG. 2 (c), a quadrangle 5111 is the boundary of the pixel 5112 of the image sensor, the upper and lower sides are s: 1-s, and the left and right sides are t: 1-t. In the case of internal division, the coordinates of neighboring pixels are set as m _{Cv, Lk, Ci, q1} , m _{Cv, Lk, Ci, q2} , m _{Cv, Lk, Ci, q3} , m _{Cv, Lk, Ci, q4} , , T [m _{Cv, Lk, Ci, q1} ], T [m _{Cv, Lk, Ci, q2} ], T [m _{Cv, Lk, Ci, q3} ], T [m _{Cv, Lk, Assuming that Ci, q4} ], the color T [m _{Cv, Lk, Ci} ] at the image point is as follows.

ここでは、バイリニア法について述べたが、ニアレストネイバー法、バイキュービック法等を用いても良い。 Although the bilinear method has been described here, the nearest neighbor method, the bicubic method, or the like may be used.

ｋ層の仮想視点の交点５０１２に対応する各カメラの撮像素子上の結像点５１０６の色を用いて、その交点５０１２における相違度ｄｉｓＲ_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を求める。 The dissimilarity disR _{Lk, Cv} [m _Cv ] at the intersection 5012 is obtained using the color of the imaging point 5106 on the image sensor of each camera corresponding to the intersection 5012 of the virtual viewpoint of the k layer.

ここで、ｐは注目画素の座標ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}の近傍の画素の座標を指すためのベクトル、ａは注目画素を原点とする近傍の画素の範囲、ｌは関数の形を変えるパラメタ、である。ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｐ}は、 Here, p is a vector for _indicating the coordinates of pixels in the vicinity of the coordinates m _{Cv, Lk, Ci} of the pixel of interest, a is a range of pixels near the pixel of interest as the origin, l is a parameter that changes the shape of the function, It is. m _{Cv, Lk, Ci, p} are

となるときの２次元座標、を表す。 Represents the two-dimensional coordinates.

また、⁻Ｔ_Ｌｋ［ｍ_Ｃｖ＋ｐ］は、注目画素、および、各近傍画素における全カメラの平均の色であり、 ^{_{_{Also, - T Lk [m Cv +}}} p] is the pixel of interest, and a color average of all camera at each neighboring pixel,

である。なお、（３．１４）式のイメージに記載されているような「−」が上についた「Ｔ」等を文章中では「^-Ｔ」等と記載する。 It is. It should be noted, (3.14) equation image such as that described in the referred to as a - ^- "T" and the like in the sentence is attached to the top "T" and the like "."

類似度Ｒ_{Ｌｋ，Ｃｖ}［ｍ，_Ｃｖ］は、相違度が大きければ小さく、小さければ大きいという関係があるとして、たとえば、以下のような関数で表す。 The similarity R _{Lk, Cv} [m, _Cv ] is expressed by the following function, for example, assuming that there is a relationship that the difference is small if the dissimilarity is large and large if the dissimilarity is small.

ここで、σは関数の形を変えるパラメタ、を表す。 Here, σ represents a parameter that changes the shape of the function.

また、類似度Ｒ_{Ｌｋ，Ｃｖ}は、以下のような関数でも良い。 Further, the similarity R _{Lk, Cv} may be a function as follows.

ここで、ｌ’は関数の形を変えるパラメタ、εは分母が０にならないようにするための微小値、である。 Here, l ′ is a parameter for changing the shape of the function, and ε is a minute value for preventing the denominator from becoming zero.

この類似度Ｒ_{Ｌｋ，Ｃｖ}［ｍ，_Ｃｖ］を、物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］とする。つまり、 The degree of similarity R _{Lk, Cv} [m, _Cv ] is _{defined as 程度} β _{Lk, Cv} [m _Cv ] to the extent that the surface of the object exists. In other words,

とする。なお、（３．１７）式のイメージに記載されているような「＾」が上についた「β」等を文章中では「＾β」等と記載する。 And Note that “β” or the like with “^” on it as described in the image of (3.17) is described as “^ β” or the like in the text.

算出した物体の表面の存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］は、現在対象となっている仮想平面とカメラ視線との交点にマッピングする。 The calculated degree of existence of the surface of the object ^ β _{Lk, Cv} [m _Cv ] is mapped to the intersection of the virtual plane that is the current object and the camera line of sight.

ステップ８：［複数のカメラで撮像したそれぞれの画像上の対応点における画素の色と、複数のカメラとユーザが見たい視点との幾何学的な関係から、１つの仮想的な面との交点における、物体の表面の色を計算する]
仮想カメラの撮像素子の画素に関して、ユーザが見たい空間的な範囲内の仮想平面群の中の１層の平面と、今注目している仮想カメラの撮像素子の画素の中心から引かれた仮想視線との交点における、物体の色を計算する。 Step 8: [Intersection of one virtual plane from the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see) Calculate the surface color of the object in]
As for the pixels of the image sensor of the virtual camera, a virtual plane drawn from the center of the pixel of the image sensor of the virtual camera that is now focused on the virtual plane group in the virtual plane group within the spatial range that the user wants to see Calculate the color of the object at the intersection with the line of sight.

この計算方法としては、たとえば、仮想カメラＣ_ｖの撮像素子上の点ｍ_Ｃｖに対応する、仮想平面ｋ層上の仮想視線との交点の色Ｔ_Ｌｋ［ｍ_Ｃｖ］は、仮想カメラＣ_ｖの仮想視線とカメラＣ_ｉのカメラ視線との成す角をθ_{Ｃｖ，Ｌｋ，Ｃｉ}［ｍ_Ｃｖ］とすると、角度が小さいときの色の寄与率を高くするようにすると、カメラの色Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}］を使って、次のようにする。これを、視点依存テクスチャマッピングという。 As a calculation method, for example, corresponds to a point _{m Cv} on the image pickup device of the virtual camera _{C v,} the color _T Lk at the intersection of the virtual line of sight on the virtual plane k layer _{[m Cv]} is the virtual camera _{C v} virtual sight and camera C _i to the angle formed theta _Cv of the camera line of _{sight, Lk, Ci} when _[m _Cv], when so as to increase the color contribution of when the angle is small, the camera color T [m _{Cv , Lk, Ci} ] is used as follows. This is called viewpoint dependent texture mapping.

算出したカメラの色は、現在対象となっている仮想平面とカメラ視線との交点にマッピングする。 The calculated camera color is mapped to the intersection of the current virtual plane and the camera line of sight.

ステップ９：［選んだ１つの仮想的な面において、ユーザが見たい視野を表す仮想的な面上の全画素に対応する全交点について、ステップ６〜８を繰り返し行うことで、物体の表面が存在する程度と色（第１の色）を計算する]
ユーザが見たい空間的な範囲の仮想面を１つ選び、ユーザが見たい視野を表す仮想的な面上の全画素に対応する全交点について、ステップ６〜８を繰り返し行い、それにより、ユーザが見たい空間的な範囲内にある１つの仮想平面上における物体の表面が存在する程度と色（第１の色）の両方を計算し、その値を仮想面上の交点にマッピングする。 Step 9: [By repeating Steps 6 to 8 for all intersection points corresponding to all pixels on the virtual surface representing the visual field that the user wants to see in the selected virtual surface, the surface of the object is Calculate the degree and color (first color) present]
The user selects one virtual plane in the spatial range that the user wants to see, and repeats Steps 6 to 8 for all intersections corresponding to all pixels on the virtual plane that represents the field of view that the user wants to see. Both the degree and color (first color) of the surface of the object on one virtual plane within the spatial range desired to be viewed are calculated, and the value is mapped to the intersection on the virtual plane.

ステップ１０：[層の中の全ての仮想的な面において、ステップ６〜９を繰り返し行うことで、物体の表面が存在する程度と色（第１の色）を計算する]
ユーザが見たい空間的な範囲の全部の仮想平面について、ステップ６〜９を繰り返し行い、それにより、ユーザが見たい空間的な範囲内の全ての仮想平面上の、カメラ視線との交点において、物体の表面が存在する程度と色（第１の色）をそれぞれ計算し、それらの２つの値をマッピングする。 Step 10: [Calculate the degree to which the surface of the object exists and the color (first color) by repeating Steps 6 to 9 on all virtual surfaces in the layer]
Steps 6-9 are repeated for all virtual planes in the spatial range that the user wants to see, so that at the intersections with the camera line of sight on all virtual planes in the spatial range that the user wants to see, The degree to which the surface of the object exists and the color (first color) are calculated, and the two values are mapped.

ステップ１１：［仮想視線上の全て交点における物体の表面が存在する程度を足し合わせる]
仮想視線と、ユーザが見たい空間的な範囲の全部の仮想平面との交点に関して、それら仮想視線ごとに、交点にマッピングされている物体の表面が存在する程度の総和ｓｕｍ＾β_Ｃｖ［ｍ_Ｃｖ］をとる。この式は次式となる。 Step 11: [Add the degree to which the surface of the object exists at all intersections on the virtual line of sight]
With respect to the intersections between the virtual line of sight and all the virtual planes in the spatial range that the user wants to see, the sum sum _{ββ Cv} [m _Cv to the extent that the surface of the object mapped to the intersection exists for each virtual line of sight. ]. This formula becomes the following formula.

Ｍは層数である。 M is the number of layers.

ステップ１２：［仮想視線上の各交点における物体の表面が存在する程度を、それら全てを足し合わせたもので割り、それらの値を各交点における物体の表面が存在する確率とする]
仮想視線上の各交点について、それぞれにマッピングされている物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を、上記の仮想視線上の各交点の総和ｓｕｍ＾β_Ｃｖ［ｍ_Ｃｖ］で割り、それらの値を各交点における物体の表面が存在する確率β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ]とする。この式は次式となる。 Step 12: [The degree to which the surface of the object exists at each intersection on the virtual line of sight is divided by the sum of all of them, and these values are used as the probability that the surface of the object at each intersection exists.]
For each intersection point on the virtual line of sight, the extent to which the surface of the object mapped to each exists ^ β _{Lk, Cv} [m _Cv ] is the sum of the intersection points on the virtual line of sight sum ^ β _Cv [m _Cv ] The values are divided by the probability β _{Lk, Cv} [m _Cv ] that the surface of the object exists at each intersection. This formula becomes the following formula.

ステップ１３［各仮想的な面の交点における、物体の表面が存在する確率と、物体の表面の色（第１の色）との積を、それらの交点における色（第４の色）とする］
仮想視線上の各交点について、それぞれにマッピングされている物体の表面が存在する確率β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］とＴ_Ｌｋ［ｍ_Ｃｖ］（第１の色）の積をとり、その積を各交点にマッピングする。この積を第４の色＾Ｔ［ｍ_Ｃｖ］と呼ぶこととする。この式は次式となる。 Step 13 [The product of the probability that the surface of the object is present at the intersection of each virtual plane and the color of the surface of the object (first color) is the color (fourth color) at those intersections. ]
For each intersection on the virtual line of sight, take the product of the probability β _{Lk, Cv} [m _Cv ] and T _Lk [m _Cv ] (first color) that the surface of the object mapped to each is present, and calculate the product Map to each intersection. This product is referred to as a fourth color ^ T [m _Cv ]. This formula becomes the following formula.

ステップ１４：［仮想視線上の全ての交点における色（第４の色）＾Ｔ_Ｌｋ［ｍ_Ｃｖ］を足し合わせ、その色を、ユーザが見たい視野を表す仮想的な面上の画素の色（第３の色）とする］
最後に、各仮想視線上の全ての交点にマッピングされた色（第４の色）＾Ｔ_Ｌｋ［ｍ_Ｃｖ］の和をとり、その色を、ユーザが見たい視野を表す仮想的な面上の画素の色（第３の色）Ｔ［ｍ_Ｃｖ］として、その画素にマッピングする。この式は次式となる。 Step 14: [Color (fourth color) ^ T _Lk [m _Cv ] at all intersections on the virtual line of sight) is added, and the color is the color of the pixel on the virtual plane representing the visual field that the user wants to see (Third color)]
Finally, the sum of the colors (fourth color) ^ T _Lk [m _Cv ] mapped to all the intersections on each virtual line of sight is taken, and the color is displayed on the virtual plane representing the visual field that the user wants to see. The pixel color (third color) T [m _Cv ] is mapped to that pixel. This formula becomes the following formula.

このようにして仮想カメラの撮像素子の全ての画素に色（第３の色）をマッピングし、それによって画像が生成される。 In this way, the color (third color) is mapped to all the pixels of the image pickup device of the virtual camera, thereby generating an image.

國田豊，上野雅浩，田中敬二，多層信頼度マップを用いた３次元映像実時間生成システム，映像情報メディア学会誌，Ｖｏｌ．６０，Ｎｏ．７，ｐｐ．１１０２-１１１０，２００６Yutaka Kunida, Masahiro Ueno, Keiji Tanaka, 3D video real-time generation system using multilayer reliability map, Journal of the Institute of Image Information and Television Engineers, Vol. 60, no. 7, pp. 1102-1110, 2006

しかしながら、上記従来方法では、ユーザが見たい空間的な範囲内の仮想平面群と仮想視線との交点群において、
（１）各交点上で物体の表面が存在する程度と色を求め（ステップ７、８、９、１０）、
（２）全ての交点上の物体の表面が存在する程度の総和をとり（ステップ１１）、
（３）各交点において物体の存在する程度をその総和で除して確率を求め（ステップ１２）、
（４）各交点において物体の存在する程度と色の積を取って、それぞれの交点にマッピングした色とする（ステップ１３）、
（５）各視線ごとに、交点にマッピングした色を積算していき、全ての交点が積算されると画像が生成される（ステップ１４）、
操作をしているので、（１）から（４）までの操作においては、全ての仮想平面の上の交点の情報を持つ必要があり、そのため、仮想平面の枚数のメモリ量（仮想平面の枚数と交点の積に比例）が必要である。 However, in the above conventional method, at the intersection group of the virtual plane group and the virtual line of sight within the spatial range that the user wants to see,
(1) Obtain the degree and color of the surface of the object on each intersection (steps 7, 8, 9, 10),
(2) Take the sum of the extent to which the surfaces of the objects on all the intersections exist (step 11),
(3) A probability is obtained by dividing the degree of existence of an object at each intersection by its sum (step 12),
(4) The product of the degree of existence of the object and the color at each intersection and the color mapped to each intersection (step 13).
(5) For each line of sight, the colors mapped to the intersections are accumulated, and when all the intersections are accumulated, an image is generated (step 14).
Since the operations are performed, the operations from (1) to (4) need to have information on the intersections on all the virtual planes. Therefore, the memory amount of the number of virtual planes (the number of virtual planes) And proportional to the product of the intersection points).

以上のことから、計算機のメモリが膨大に必要であるという欠点があった。また、メモリが膨大となるため、装置が高価になるという欠点があった。 From the above, there was a drawback that a huge amount of computer memory was required. Further, since the memory becomes enormous, there is a disadvantage that the apparatus becomes expensive.

また、メモリが少ない場合は、ユーザが見たい空間的な範囲内の仮想平面群の数を減らさざるを得ず、そのため、ユーザが見たい空間的な範囲を狭めざるを得なくなり、画質が低下するという欠点があった。 In addition, when the memory is small, the number of virtual plane groups in the spatial range that the user wants to see must be reduced, and thus the spatial range that the user wants to see must be narrowed, resulting in poor image quality. There was a drawback of doing.

また、メモリが少ない場合は、ユーザが見たい空間的な範囲内の仮想平面群の数を減らさざるを得ず、そのため、ユーザが見たい空間的な範囲内に設置する仮想平面群の数が減るため、表現できる奥行きの精度が低下し、生成する画像の画質が低下するという欠点があった。 In addition, when the memory is small, the number of virtual plane groups within the spatial range that the user wants to see must be reduced. Therefore, the number of virtual plane groups installed within the spatial range that the user wants to see is reduced. Therefore, the accuracy of the depth that can be expressed is lowered, and the image quality of the generated image is lowered.

本発明は、上記に鑑みてなされたものであり、その目的とするところは、上記従来方式と同等な画質で画像生成でき、画像生成の計算に必要なメモリ量を削減した画像生成方法を提供することにある。 The present invention has been made in view of the above, and an object of the present invention is to provide an image generation method capable of generating an image with an image quality equivalent to that of the conventional method and reducing the amount of memory necessary for image generation calculation. There is to do.

本願において開示される発明のうち、代表的なものの概要を簡単に説明すれば、以下の通りである。 Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

画像生成装置における画像生成方法であって、視点の異なる複数のカメラを用いて同一の物体や風景を撮影した画像を入力する第１のステップと、ユーザが見たい視点（仮想視点）を空間中に設置する第２のステップと、次に、ユーザが見たい空間的な範囲内に複数の仮想的な面を層状に設置する第３のステップと、ユーザが見たい視野を表す仮想的な面を設置する第４のステップと、ユーザが見たい精細度となるように、ユーザが見たい視野を表す仮想的な面上に、複数の画素を配置する第５のステップと、画素を１つ選び、また、仮想的な面を１つ選び、仮想視点と画素を通る直線（仮想視線）と、仮想的な面との交点に対して、複数のカメラで撮像した複数の画像上におけるそれぞれの対応点を計算する第６のステップと、複数のカメラで撮像したそれぞれの画像上の対応点およびその付近の画像同士の類似性から、１つの仮想的な面との交点における、物体の表面が存在する程度を計算する第７のステップと、複数のカメラで撮像したそれぞれの画像上の対応点における画素の色と、複数のカメラとユーザが見たい視点との幾何学的な関係から、１つの仮想的な面との交点における、物体の表面の色（第１の色）を計算する第８のステップと、選んだ１つの仮想的な面において、ユーザが見たい視野を表す仮想的な面上の全画素に対応する全交点について、第６〜８のステップを行うことで、物体の表面が存在する程度と色（第１の色）を計算する第９のステップと、選んだ１つの仮想的な面と仮想視線との交点における、物体の存在する程度と、物体の表面の色（第１の色）との積を、その交点における色（第２の色）とする第１０のステップと、もし以前に別の仮想的な面において色（第２の色）を計算しており、その色（第２の色）を積算していれば、選んだ１つの仮想的な面における色（第２の色）を、同じ仮想視線ごとにその積算値に積算し、もし、積算値がない場合（つまり、最初の色（第２の色）を算出する場合）は、その色（第２の色）を積算値の初期値とする第１１のステップと、もし以前に別の仮想的な面において物体の表面が存在する程度を計算しており、その物体の表面が存在する程度を積算していれば、選んだ１つの仮想的な面における物体の表面が存在する程度を、同じ仮想視線ごとにその積算値に積算し、もし、積算値がない場合（つまり、最初の物体の表面が存在する程度を算出する場合）は、その物体の表面が存在する程度を積算値の初期値とする第１２にステップと、色（第２の色）と物体の表面が存在する程度について、第６〜１２のステップを行い、それぞれの値を仮想視線ごとに、層状の全ての仮想的な面において積算する第１３のステップと、仮想視線ごとに、色（第２の色）の積算値を、物体の表面が存在する程度の積算値で割って、色（第３の色）を求め、その色（第３の色）を、ユーザが見たい視野を表す仮想的な面上の画素の色とする第１４のステップと、を有することを特徴とする。 An image generation method in an image generation apparatus, wherein a first step of inputting images obtained by photographing the same object or landscape using a plurality of cameras with different viewpoints, and a viewpoint (virtual viewpoint) that the user wants to see in space A second step of installing a plurality of virtual planes in a spatial range desired to be viewed by the user, and a virtual plane representing a visual field desired to be viewed by the user A fourth step of disposing a plurality of pixels on a virtual plane representing a visual field desired by the user so as to obtain a definition desired by the user, and one pixel. Select one virtual plane, and the intersection of the virtual viewpoint and the straight line (virtual line of sight) passing through the pixel and the virtual plane is displayed on each of a plurality of images captured by a plurality of cameras. Sixth step of calculating corresponding points and multiple turtles A seventh step of calculating the degree to which the surface of the object exists at the intersection with one virtual plane from the corresponding points on the respective images captured in step 1 and the similarity between the adjacent images; From the geometric relationship between the color of the pixel at the corresponding point on each image captured by the camera and the viewpoint that the user wants to see, the surface of the object at the intersection of one virtual plane The eighth step of calculating the color (first color) and the sixth intersection for all the intersections corresponding to all the pixels on the virtual surface representing the visual field that the user wants to see in one selected virtual surface. The ninth step of calculating the degree to which the surface of the object exists and the color (first color) by performing steps ˜8, and the object at the intersection of the selected one virtual plane and virtual line of sight And the color of the surface of the object (first )) As a color at the intersection (second color), and a color (second color) is calculated in another virtual plane before that color (second color) If the second color) is integrated, the color (second color) on the selected virtual plane is integrated with the integrated value for each virtual line of sight, and if there is no integrated value ( That is, when calculating the first color (second color), the eleventh step in which that color (second color) is the initial value of the integrated value, If the degree to which the surface of the object exists is calculated, and the degree to which the surface of the object exists is integrated, the degree to which the surface of the object exists in the selected one virtual plane is calculated for each virtual line of sight. If there is no integrated value (that is, calculate the degree to which the surface of the first object exists) In the case where the surface of the object is present, the twelfth step having the initial value of the integrated value as the initial value of the integrated value, and the sixth to twelfth steps regarding the degree of existence of the color (second color) and the surface of the object. And the 13th step of integrating the respective values for each virtual line of sight on all the virtual surfaces in the layer, and the integrated value of the color (second color) for each virtual line of sight, A color (third color) is obtained by dividing by an existing integrated value, and the color (third color) is used as the color of a pixel on a virtual plane representing the visual field desired by the user. These steps are included.

本発明は、ユーザが見たい空間的な範囲内の仮想平面群と仮想視線との交点群に関して、ある１つの仮想平面上の全交点において、
（１）各交点上で物体の表面が存在する程度と色を求め（ステップ７、８、９）、
（２）各交点において物体の存在する程度と色の積を取って、それぞれの交点における色とし（ステップ１０）、
（３）上記（１）（２）を仮想平面１枚を計算するごとに、同じ仮想視線ごとに物体の存在する程度と色の積を積算し（ステップ１１、１３）、
（４）上記（１）を仮想平面１枚を計算するごとに、同じ仮想視線ごとに物体の表面が存在する程度を積算し（ステップ１２、１３）、
（５）全てに仮想平面についての計算が終了後、同じ仮想視線ごとに、物体の存在する程度と色の積を積算値を物体の表面が存在する程度を積算値で割って最終的な画像を生成する（ステップ１４）、
操作をしているので、（１）から（５）までの操作においては、物体の表面が存在する程度の計算と、色の計算と、それらの積の計算と、物体の表面が存在する程度の積算と、積の積算をするのに、最大でもそれぞれ仮想平面１枚ずつ、つまり、計５枚のメモリ量（５枚と交点の積に比例）のみ必要であり、従来方法で必要であった仮想平面の枚数のメモリ量（仮想平面の枚数と交点の積に比例）と異なるが、通常、良好な生成画像を得るには、仮想平面は最低３０枚程度必要なことが経験的に分かっており、そのことから、メモリ量が削減できる利点がある。 The present invention relates to a group of intersections between a virtual plane group and a virtual line of sight within a spatial range that the user wants to see, at all intersections on a certain virtual plane,
(1) Obtain the degree and color of the surface of the object on each intersection (steps 7, 8, 9),
(2) The product of the degree and color of the object at each intersection is taken as the color at each intersection (step 10),
(3) Every time one virtual plane is calculated in (1) and (2) above, the product of the degree of existence of the object and the color is integrated for each same virtual line of sight (steps 11 and 13).
(4) Every time one virtual plane is calculated in (1) above, the degree to which the surface of the object exists for each same virtual line of sight is integrated (steps 12 and 13).
(5) After the calculation for the virtual plane is completed for all, the final image is obtained by dividing the product of the degree of existence and color by the integrated value and the degree of existence of the surface of the object by the integrated value for each virtual line of sight. (Step 14),
Since the operation is performed, in the operations from (1) to (5), the calculation to the extent that the surface of the object exists, the calculation of the color, the calculation of their product, and the extent to which the surface of the object exists In order to integrate the product and product, only one virtual plane is required at most, that is, only a total of 5 memory amounts (proportional to the product of 5 images and the intersection) are required. The number of virtual planes is different from the amount of memory (proportional to the product of the number of virtual planes and the intersection point), but it is empirically found that usually about 30 virtual planes are required to obtain a good generated image. Therefore, there is an advantage that the amount of memory can be reduced.

本発明の実施形態の画像生成装置における画像生成方法の処理の流れを図４を用いて説明する。以下の第１のステップ〜第１４のステップは図４のＳ１〜Ｓ１４に対応する。 A processing flow of the image generation method in the image generation apparatus according to the embodiment of the present invention will be described with reference to FIG. The following first to fourteenth steps correspond to S1 to S14 in FIG.

まず、視点の異なる複数のカメラを用いて同一の物体や風景を撮影した画像を画像生成装置（図示していない。）に入力する（第１のステップ）。
次に、ユーザが見たい視点を空間中に設置する（第２のステップ）。
次に、ユーザが見たい空間的な範囲内に複数の仮想的な面で構成した層を設置する（第３のステップ）。
次に、ユーザが見たい視野を表す仮想的な面を設置する（第４のステップ）。
次に、ユーザが見たい精細度となるように、ユーザが見たい視野を表す仮想的な面上に、複数の画素を配置する（第５のステップ）。
次に、画素を１つ選び、また、仮想的な面を１つ選び、仮想視点とその画素を通る直線（視線）と、その仮想的な面との交点に対して、カメラで撮像した画像上におけるそれぞれの対応点を計算する（第６のステップ）。
次に、複数のカメラで撮像したそれぞれの画像上の対応点およびその付近の画像同士の類似性から、１つの仮想的な面との交点における、物体の表面が存在する程度を計算する（第７のステップ）。
次に、複数のカメラで撮像したそれぞれの画像上の対応点における画素の色と、複数のカメラとユーザが見たい視点との幾何学的な関係から、１つの仮想的な面との交点における、物体の表面の色を計算する（第８のステップ）。
次に、選んだ１つの仮想的な面において、ユーザが見たい視野を表す仮想的な面上の全画素に対応する全交点について、第６〜８のステップを行うことで、物体の表面が存在する程度と色（第１の色）を計算する（第９のステップ）。
次に、１つの仮想的な面の交点における、物体の存在する程度と、物体の表面の色（第１の色）との積を、その交点における色（第２の色）とする（第１０のステップ）、
次に、もし以前に別の仮想的な面において色（第２の色）を計算しており、その色（第２の色）を積算していれば、今回選んだ１つの仮想的な面における色（第２の色）を、同じ視線ごとに積算する。以前の積算値がない場合（つまり、最初の色（第２の色）を計算する場合）は、その色（第２の色）を積算値の初期値とする（第１１のステップ）、
次に、もし以前に別の仮想的な面において物体の表面が存在する程度を計算しており、その物体の表面が存在する程度を積算していれば、今回選んだ１つの仮想的な面における物体の表面が存在する程度を、同じ視線ごとに積算する。
以前の積算値がない場合（つまり、最初の物体の表面が存在する程度を計算する場合は、その物体の表面が存在する程度を積算値の初期値とする（第１２のステップ）、
次に、層状の全ての仮想的な面において、第６〜１３のステップを行うことで、色（第２の色）と物体の表面が存在する程度を積算する（第１３のステップ）、
最後に、仮想視線ごとに、色（第２の色）の積算値を、物体の表面が存在する程度の積算値で割って、色（第３の色）を求め、その色（第３の色）を、ユーザが見たい視野を表す仮想的な面上の画素の色とする（第１４のステップ）。 First, the image which image | photographed the same object and landscape using the several camera from which a viewpoint differs is input into an image generation apparatus (not shown) (1st step).
Next, the viewpoint that the user wants to see is placed in the space (second step).
Next, a layer composed of a plurality of virtual planes is installed within a spatial range that the user wants to see (third step).
Next, a virtual surface representing the visual field that the user wants to see is installed (fourth step).
Next, a plurality of pixels are arranged on a virtual surface representing the visual field that the user wants to see so as to obtain the definition that the user wants to see (fifth step).
Next, one pixel is selected, and one virtual plane is selected. An image captured by the camera at the intersection of the virtual viewpoint and a straight line (line of sight) passing through the pixel and the virtual plane Each corresponding point in the above is calculated (sixth step).
Next, the degree to which the surface of the object exists at the intersection with one virtual plane is calculated from the similarity between the corresponding points on the images captured by a plurality of cameras and the images in the vicinity thereof (first). Step 7).
Next, from the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see, at the intersection of one virtual plane The color of the surface of the object is calculated (eighth step).
Next, in the selected one virtual surface, the sixth to eighth steps are performed on all the intersection points corresponding to all the pixels on the virtual surface representing the visual field that the user wants to see, so that the surface of the object is The degree of existence and the color (first color) are calculated (9th step).
Next, the product of the degree of existence of the object and the color of the surface of the object (first color) at the intersection of one virtual plane is defined as the color (second color) at the intersection (first color). 10 steps),
Next, if the color (second color) has been previously calculated in another virtual surface, and that color (second color) has been integrated, then one virtual surface selected this time Are integrated for each line of sight. When there is no previous integrated value (that is, when calculating the first color (second color)), that color (second color) is set as the initial value of the integrated value (eleventh step).
Next, if the degree to which the surface of the object exists in another virtual surface has been calculated before, and the degree to which the surface of the object exists has been integrated, then one virtual surface selected this time The degree to which the surface of the object exists is integrated for each line of sight.
When there is no previous integrated value (that is, when calculating the extent to which the surface of the first object exists, the extent to which the surface of the object exists is set as the initial value of the integrated value (12th step),
Next, the sixth to thirteenth steps are performed on all the layered virtual surfaces to integrate the color (second color) and the degree to which the surface of the object exists (the thirteenth step).
Finally, for each virtual line of sight, the integrated value of the color (second color) is divided by the integrated value to the extent that the surface of the object exists to obtain the color (third color), and the color (third color) Let (color) be the color of a pixel on a virtual surface representing the visual field that the user wants to see (fourteenth step).

以下、本実施形態の画像生成方法をステップに分けて動作を説明する。 Hereinafter, the operation of the image generation method of this embodiment will be described by dividing it into steps.

ステップ１：［複数のカメラを用いて同一物体や風景を撮影]
Ｎ台のカメラで、同一物体や風景を撮影し、撮影した画像を画像生成装置（図示していない。）に入力する。このとき、各カメラＣ_ｉ（ｉ＝１，．．．，Ｎ）において、コンピュータビジョンの分野で知られているカメラの内部に関するパラメタ（カメラ内部パラメタ）と、カメラの位置、姿勢に関するパラメタ（カメラ外部パラメタ）は既知とする。カメラ内部パラメタは、行列表記すると、たとえば、以下のようなものである。 Step 1: [Shooting the same object or landscape using multiple cameras]
The same object or landscape is photographed by N cameras, and the photographed image is input to an image generation device (not shown). At this time, in each camera C _i (i = 1,..., N), parameters relating to the inside of the camera (camera internal parameters) and parameters relating to the position and orientation of the camera (cameras) known in the field of computer vision External parameters are assumed to be known. The camera internal parameters are, for example, as follows in matrix notation.

Ａ_Ｃｉをカメラ内部行列といい、３行３列の行列である。ここで、α_ｕ，Ｃｉは焦点距離を撮像素子の横軸の画素ピッチで割ったもの、α_ｖ，Ｃｉは焦点距離を撮像素子の縦軸の画素ピッチで割ったもの、θ_Ｃｉは撮像素子の縦軸と横軸の成す角度、ｕ_{Ｃｉ０，ｖＣｉ０}は撮像素子とレンズ光軸の交点の座標であり、画素単位で表記し、撮像した画像の左上を原点とする座標、である。 A _Ci is called a camera internal matrix and is a matrix of 3 rows and 3 columns. Here, α _{u, C i} is the focal length divided by the horizontal pixel pitch of the image sensor, α _{v, C} _i is the focal length divided by the vertical pixel pitch of the image sensor, and θ _Ci is the image sensor. The angles formed by the vertical axis and the horizontal axis, u _{Ci0 and vCi0} are the coordinates of the intersection of the image sensor and the lens optical axis, and are expressed in pixel units and are the coordinates with the upper left corner of the captured image as the origin.

なお、ｕ_{Ｃｉ0，ｖＣｉ0}で使用される座標系をカメラＣ_ｉにおけるディジタル画像座標といい、カメラの撮像素子上の２次元の斜交座標であり、撮像素子の画素間隔を単位とし、撮像素子の横軸、縦軸をその軸方向とし、撮像素子の左上を原点とするものであり、カメラごとに固有のものとなる。 Incidentally, referred to as digital image coordinate a coordinate system used by _{u Ci0,} vCi0 in the camera C _i, a two-dimensional oblique coordinates on the image pickup device of the camera, the pixel interval of the imaging device as a unit, of the image sensor The horizontal axis and the vertical axis are the axial directions, and the upper left of the image sensor is the origin, which is unique to each camera.

ｒ_Ｃｉｘ、ｒ_Ｃｉｙ、ｒ_Ｃｉｚ、は、カメラＣ_ｉのレンズ中心を原点とした３次元の直交座標系（これをカメラ座標系という）の各軸の単位ベクトルを、世界座標系で表したベクトル、ｔ_Ｃｉｘ、ｔ_Ｃｉｙ、ｔ_Ｃｉｚは、カメラ座標系における、世界座標系の原点の座標を表すベクトルの要素、である。 r _Cix , r _Ciy , and r _Ciz are vectors representing the unit vector of each axis in a three-dimensional orthogonal coordinate system (this is called the camera coordinate system) with the lens center of the camera C _i as the origin in the world coordinate system. , T _Cix , t _Ciy , t _Ciz are vector elements representing the coordinates of the origin of the world coordinate system in the camera coordinate system.

ステップ２：［ユーザが見たい視点を設置]
ユーザが自分の目を持っていきたい場所を決め、そこに仮想的に視点（以下、仮想視点と表記）１００３を設置する。 Step 2: [Set the viewpoint that the user wants to see]
A user decides where he / she wants to take his / her eyes, and virtually installs a viewpoint (hereinafter referred to as a virtual viewpoint) 1003 there.

ステップ４：[ユーザが見たい視野を表す仮想的な面を設置]
仮想視点１００３から見た視野内であって、ユーザが生成したい画像の範囲を表す仮想的な面１００４を設置する。これは、仮想視点１００３をレンズ中心としたレンズ１００２を持つ仮想カメラ１００１の撮像素子と見なすことができるので、以下、仮想カメラの撮像素子と言う。 Step 4: [Install a virtual surface that represents the field of view that the user wants to see]
A virtual plane 1004 that is within the field of view as viewed from the virtual viewpoint 1003 and that represents the range of the image that the user wants to generate is installed. This can be regarded as an image sensor of a virtual camera 1001 having a lens 1002 with the virtual viewpoint 1003 as the center of the lens, and is hereinafter referred to as an image sensor of a virtual camera.

ここで、Ｃ_ｖは仮想視点カメラを表す。上記式の内容は、（８．１）〜（８．３）式と同様であるので、説明は省略する。 Here, _Cv represents a virtual viewpoint camera. Since the contents of the above formula are the same as the formulas (8.1) to (8.3), the description thereof will be omitted.

ステップ６：［仮想視点と画素の１つを通る直線（視線）と１つの仮想的な面の交点に対して、各カメラ画像上の対応点を計算]
仮想カメラの撮像素子１００４の面上の画素１００６を１つ選び、また、ユーザが見たい空間的な範囲内の仮想平面群１０１１の中の層を１つ選び、仮想視点とその選んだ画素を通る直線（仮想視線）と、その仮想的な面との交点（仮想平面と仮想視線との交点群の中の１つ）に対して、複数のカメラで撮像した複数の画像上におけるそれぞれの対応点（図１では、カメラＣ_１については仮想平面と視線の交点の結像点１１０６で表記）を計算する。 Step 6: [Calculate corresponding points on each camera image for the intersection of a virtual viewpoint and a straight line (line of sight) passing through one of the pixels and one virtual plane]
One pixel 1006 on the surface of the image sensor 1004 of the virtual camera is selected, and one layer in the virtual plane group 1011 within the spatial range desired by the user is selected, and the virtual viewpoint and the selected pixel are selected. Correspondence on a plurality of images taken by a plurality of cameras with respect to a crossing line (virtual line of sight) and an intersection of the virtual plane (one of a group of intersections of the virtual plane and the virtual line of sight) (in Figure 1, the camera C ₁ are denoted by the imaging point 1106 of the intersection of the imaginary plane and the line of sight) the point is calculated.

たとえば、仮想カメラＣ_ｖの撮像素子１００４上の画素１００６の、仮想カメラＣ_ｖのディジタル画像座標系での座標をｍ_Ｃｖ＝［ｕ_Ｃｖｖ_Ｃｖ］^Ｔ、カメラＣ_１の撮像素子１１０４上の対応点１１０６の、カメラＣ_ｉのディジタル画像座標系での座標をｍ_{Ｃｖ,Ｌｋ,Ｃｉ}＝[ｕ_{Ｃｖ,Ｌｋ,Ｃｉ} ｖ_{Ｃｖ,Ｌｋ,Ｃｉ}]とすると、 For example, the virtual camera _{C v} pixel 1006 on the imaging device 1004, the coordinates of the digital image coordinate system of the virtual camera _{_{_{_{C v m Cv = [u Cv}}}} v Cv] T, corresponding on the image pickup device 1104 of the camera _{C 1} If the coordinates of the point 1106 in the digital image coordinate system of the camera _Ci are m _{Cv, Lk,} C _i = [u _{Cv, Lk,} C _i v _{Cv, Lk,} C _i ],

となる。 It becomes.

ここで、^~ｍ_{Ｃｖ,Ｌｋ,Ｃｉ}、^~ｍ_Ｃｖはそれぞれｍ_{Ｃｖ,Ｌｋ,Ｃｉ}、ｍ_Ｃｖの同次座標であり、それぞれ、^~ｍ_{Ｃｖ,Ｌｋ,Ｃｉ}［ｕ_{Ｃｖ，Ｌｋ，Ｃｉ} ｖ_{Ｃｖ，Ｌｋ，Ｃｉ} １］^Ｔ、^~ｍ_Ｃｖ［ｕ_Ｃｖｖ_Ｃｖ１］^Ｔ、である。 ^{_{Here, ~ m Cv, Lk, Ci}} , ~ m Cv are homogeneous coordinates of _{_{m Cv, Lk, Ci, m}} Cv, ^{_{respectively, ~ m Cv, Lk, Ci}} [u Cv, Lk, Ci v Cv _{, Lk, Ci} 1] ^T , ^~ m _Cv [u _Cv v _Cv 1] ^T.

Ｈ_{Ｌｋ,Ｃ１,Ｃｖ}はホモグラフィ行列といい、 _{HLk, C1, and Cv} are called homography matrices,

である。ｓはスカラ値であり、ｓが（８.７）式右辺のＨ_{Ｌｋ,Ｃｉ,Ｃｖ}の三行目を示すベクトルと^~ｍ_Ｃｖとの内積値、である。ここで、Ｒ_{Ｃｉ,Ｃｖ}とｔ_{Ｃｉ,Ｃｖ}は仮想カメラＣ_ｖのカメラ座標系からカメラＣ_１のカメラ座標系へ変換する際の回転行列と平行並進ベクトルであり、 It is. s is a scalar value, and s is an inner product value of a vector indicating the third row of _{HLk, Ci, Cv on} the right side of the equation (8.7) and ^~ _mCv . Here, R _{Ci, Cv} and t _{Ci, Cv} are a rotation matrix and a parallel translation vector when converting from the camera coordinate system of the virtual camera C _{v to} the camera coordinate system of the camera C ₁ ,

ステップ７：［対応点上の画像の類似点から、１つの仮想的な面の交点における、物体の表面が存在する程度を計算］
仮想カメラの撮像素子の画素に関して、ユーザが見たい空間的な範囲内の仮想平面群の中の１層の平面と、今注目している仮想カメラの撮像素子の画素の中心から引かれた仮想視線との交点における、物体の表面が存在する程度を計算する。ここでは、その交点に対応する各カメラの撮像素子上の点とその近傍の点のテクスチャが類似しているほど物体の表面が存在する程度が高いとする計算を行う。仮想平面上の交点の近傍の点とは、たとえば、今注目している仮想カメラの撮像素子上の画素の近傍の画素の中心から引かれる仮想カメラの視線と仮想平面との交点としても良い。 Step 7: [Calculate the degree to which the surface of the object exists at the intersection of one virtual plane from the similarities of the images on the corresponding points]
As for the pixels of the image sensor of the virtual camera, a virtual plane drawn from the center of the pixel of the image sensor of the virtual camera that is now focused on the virtual plane group in the virtual plane group within the spatial range that the user wants to see The degree to which the surface of the object exists at the intersection with the line of sight is calculated. Here, the calculation is performed such that the surface of the object is higher as the texture of the point on the image sensor of each camera corresponding to the intersection and the point near the point are similar. The point in the vicinity of the intersection point on the virtual plane may be, for example, the intersection point of the virtual camera's line of sight drawn from the center of the pixel in the vicinity of the pixel on the imaging element of the virtual camera of interest.

結像点５１０６の色は、たとえば、結像点を中心とする四角形５１１１が重なっているカメラの撮像素子の画素群５１１２の各画素の面積比で、それらの画素の色を足し合わせるという、バイリニア法で得ることができる。この方法は、たとえば、図２（ｃ）に示すように、四角形５１１１が撮像素子の画素５１１２の境界で、上下の辺がｓ：１−ｓに、左右の辺がｔ：１−ｔにそれぞれ内分されていた場合、近傍の画素の座標をｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ１}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ２}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ３}、ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ４}、として、それぞれの画素の色をＴ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ１}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ２}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ３}］、Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｑ４}］とすると、結像点の色Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}］は、以下のようになる。 The color of the image formation point 5106 is, for example, the bilinearity in which the color of these pixels is added by the area ratio of each pixel of the pixel group 5112 of the image sensor of the camera on which the quadrangle 5111 centering on the image formation point overlaps. Can be obtained by law. In this method, for example, as shown in FIG. 2C, a quadrangle 5111 is the boundary of the pixel 5112 of the image sensor, the upper and lower sides are s: 1-s, and the left and right sides are t: 1-t. In the case of internal division, the coordinates of neighboring pixels are set as m _{Cv, Lk, Ci, q1} , m _{Cv, Lk, Ci, q2} , m _{Cv, Lk, Ci, q3} , m _{Cv, Lk, Ci, q4} , , T [m _{Cv, Lk, Ci, q1} ], T [m _{Cv, Lk, Ci, q2} ], T [m _{Cv, Lk, Ci, q3} ], T [m _{Cv, Lk, Assuming that Ci, q4} ], the color T [m _{Cv, Lk, Ci} ] at the image point is as follows.

ｋ層の仮想視点の交点５０１２に対応する各カメラの撮像素子上の結像点５１０６の色を用いて、その交点５０１２における相違度ｄｉｓＲ_{Ｌｋ,Ｃｖ}［ｍ_Ｃｖ］を求める。 The dissimilarity disR _{Lk, Cv} [m _Cv ] at the intersection point 5012 is obtained using the color of the imaging point 5106 on the image sensor of each camera corresponding to the intersection point 5012 of the virtual viewpoint of the k layer.

ここで、ｐは注目画素の座標ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}の近傍の画素の座標を指すためのベクトル、ａは注目画素を原点とする近傍の画素の範囲、ｌは関数の形を変えるパラメタである。ｍ_{Ｃｖ，Ｌｋ，Ｃｉ，ｐ}は、 Here, p is a vector for _indicating the coordinates of pixels in the vicinity of the coordinates m _{Cv, Lk, Ci} of the pixel of interest, a is a range of pixels near the pixel of interest as the origin, and l is a parameter that changes the shape of the function. is there. m _{Cv, Lk, Ci, p} are

また、^-Ｔ_Ｌｋ［ｍ_Ｃｖ＋ｐ］は、注目画素、および、各近傍画素における全カメラの平均の色であり、 ^{_{_{Also, - T Lk [m Cv +}}} p] is the pixel of interest, and a color average of all camera at each neighboring pixel,

である。 It is.

類似度Ｒ_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］は、相違度が大きければ小さく、小さければ大きいという関係があるとして、たとえば、以下のような関数で表す。 The similarity R _{Lk, Cv} [m _Cv ] is represented by the following function, for example, assuming that there is a relationship that the difference is small if the dissimilarity is large and is large if the dissimilarity is small.

また、類似度Ｒ_{Ｌｋ,Ｃｖ}は、以下のような関数でも良い。 Further, the similarity R _{Lk, Cv} may be a function as follows.

この類似度Ｒ_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を、物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］とする。つまり、 The degree of similarity R _{Lk, Cv} [m _Cv ] is _{defined as 程度} β _{Lk, Cv} [m _Cv ] to the extent that the surface of the object exists. In other words,

とする。 And

ステップ８：［複数のカメラで撮像したそれぞれの画像上の対応点における画素の色と、複数のカメラとユーザが見たい視点との幾何学的な関係から、１つの仮想的な面との交点における、物体の表面の色（第１の色）を計算する]
仮想カメラの撮像素子の画素に関して、ユーザが見たい空間的な範囲内の仮想平面群の中の１層の平面と、今注目している仮想カメラの撮像素子の画素の中心から引かれた仮想視線との交点における、物体の色を計算する。 Step 8: [Intersection of one virtual plane from the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see) Calculate the surface color of the object (first color)]
As for the pixels of the image sensor of the virtual camera, a virtual plane drawn from the center of the pixel of the image sensor of the virtual camera that is now focused on the virtual plane group in the virtual plane group within the spatial range that the user wants to see Calculate the color of the object at the intersection with the line of sight.

この計算方法としては、たとえば、仮想カメラＣ_ｖの撮像素子上の点ｍ_Ｃｖに対応する、仮想平面ｋ層上の仮想視線との交点の色（第１の色）Ｔ_Ｌｋ［ｍ_Ｃｖ］は、仮想カメラＣ_ｖの仮想視線とカメラＣ_ｉのカメラ視線との成す角をθ_{Ｃｖ，Ｌｋ，Ｃｉ}［ｍ_Ｃｖ］とすると、角度が小さいときの色の寄与率を高くするようにすると、カメラの色Ｔ［ｍ_{Ｃｖ，Ｌｋ，Ｃｉ}］を使って、次のようにする。これを、視点依存テクスチャマッピングという。 As this calculation method, for example, the color (first color) T _Lk [m _Cv ] of the intersection with the virtual line of sight on the virtual plane k layer corresponding to the point m _Cv on the image sensor of the virtual camera C _v is , if the virtual line of sight and camera C _i to the angle formed between the camera line of sight _{θ Cv, Lk, Ci [m} Cv] of the virtual camera C _v, when so as to increase the color contribution of when the angle is small, the camera The color T [m _{Cv, Lk, Ci} ] is used as follows. This is called viewpoint dependent texture mapping.

ステップ１０：［１つの仮想的な面の交点における、物体の表面の存在する程度と、物体の表面の色（第１の色）との積を、その交点における色（第２の色）とする。]
１つのユーザが見たい空間的な範囲の仮想面と仮想視点との交点における、物体の表面の存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］と、物体の表面の色（第１の色）Ｔ_Ｌｋ［ｍ_Ｃｖ］との積をとり、その積をとった色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を新たにその交点における色としてマッピングする。色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］の式は次式となる。 Step 10: [The product of the degree of existence of the surface of the object at the intersection of one virtual plane and the color of the surface of the object (first color) is the color at the intersection (second color). To do. ]
At the intersection between the virtual viewpoint and a single user spatial extent of the virtual plane to be viewed, the degree ^ beta _Lk present of the surface of the _object, and _{Cv [m} _Cv], of the surface of the object color (first color) The product of T _Lk [m _Cv ] is taken, and the color (second color) ^ T _Lk '[m _Cv ] obtained by the product is newly mapped as the color at the intersection. The expression of the color (second color) ^ T _Lk '[m _Cv ] is as follows.

ステップ１１：［もし以前に別の仮想的な面において色（第２の色）を計算しており、その色（第２の色）を積算していれば、今回選んだ１つの仮想的な面における色（第２の色）を、同じ視線ごとに積算する。以前の積算値が無い場合（つまり、最初の色（第２の色）を計算する場合）は、その色（第２の色）を積算値の初期値とする。］
もし、以前に、上記の計算を、ユーザが見たい空間的な範囲の別の仮想面において、色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を計算しており、その色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を、同じ仮想視線上の交点の色に対して積算していれば、今回選んだ１つの仮想的な面における色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を、前と同様に同じ仮想視線ごとに積算する。 Step 11: [If a color (second color) has been previously calculated in another virtual plane, and that color (second color) has been integrated, then one selected virtual The color on the surface (second color) is integrated for each line of sight. When there is no previous integrated value (that is, when calculating the first color (second color)), that color (second color) is set as the initial value of the integrated value. ]
If, in the previous calculation, the color (second color) ^ T _Lk '[m _Cv ] is calculated in another virtual plane in the spatial range that the user wants to see, the color (second 2 color) ^ T _Lk '[m _Cv ] is integrated with the color of the intersection point on the same virtual line of sight, the color (second color) ^ T in the one virtual plane selected this time _Lk ′ [m _Cv ] is integrated for each virtual line of sight as before.

もし、以前の積算値が無い場合、つまり、今回が最初の色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を計算する場合は、その色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を積算値の初期値として、以降、この値に積算していく。 If there is no previous integrated value, that is, if this time the first color (second color) ^ T _Lk '[m _Cv ] is calculated, that color (second color) ^ T _Lk ' [ m _Cv ] is set as an initial value of the integrated value, and thereafter, this value is integrated.

処理を１層から始めてＫ層まで進んだとし、この色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］の１〜Ｋ層までの積算値をｓｕｍ＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］とすると、この積算値は次式となる。 Assume that the processing starts from the first layer and proceeds to the K layer, and the integrated value of this color (second color) ^ T _Lk '[m _Cv ] to the 1st to K layers is sum ^ T _Lk ' [m _Cv ]. The integrated value is given by the following equation.

ここで、Ｋは１≦Ｋ≦Ｍである。なお、Ｍは層数である。 Here, K is 1 ≦ K ≦ M. M is the number of layers.

ステップ１２：［もし以前に別の仮想的な面において物体の表面が存在する程度を計算しており、その物体の表面の存在する程度を積算していれば、今回選んだ１つの仮想的な面における物体の表面が存在する程度を、同じ視線ごとに積算する。以前の積算値がない場合（つまり、最初の物体の表面が存在する程度を計算する場合）は、その物体の表面が存在する程度を積算値の初期値とする。]
もし、以前に、上記の計算を、ユーザが見たい空問的な範囲の別の仮想面において、物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を計算しており、その物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を、同じ仮想視線上の交点における物体の表面の存在する程度に対して積算していれば、今回選んだ１つの仮想的な面における物体の表面の存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を、前と同様に同じ仮想視線ごとに積算する。 Step 12: [If the degree to which the surface of the object exists in another virtual plane has been calculated before, and the degree to which the surface of the object exists has been integrated, then one virtual The degree to which the surface of the object exists on the surface is integrated for each line of sight. When there is no previous integrated value (that is, when calculating the degree to which the surface of the first object exists), the degree to which the surface of the object exists is set as the initial value of the integrated value. ]
If, in the above calculation, the degree to which the surface of the object exists in another virtual plane in the sky range that the user wants to see is calculated _{, ββ Lk, Cv} [m _Cv ] If β β _{Lk, Cv} [m _Cv ] is integrated with the degree of existence of the surface of the object at the intersection point on the same virtual line of sight, the one virtual plane selected this time The degree to which the surface of the object exists ^ β _{Lk, Cv} [m _Cv ] is integrated for each virtual visual line as before.

もし、以前の積算値が無い場合、つまり、今回が最初の物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を計算する場合は、その物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を積算値の初期値として、以降、この値に積算していく。 If there is no previous integrated value, that is, if this time is calculated to the extent that the surface of the first object exists ^ β _{Lk, Cv} [m _Cv ], the extent to which the surface of the object exists ^ β _{Lk, Cv} [m _Cv ] is set as an initial value of the integrated value, and thereafter, this value is integrated.

処理を１層から始めてＫ層まで進んだとし、この物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］の１〜Ｋ層までの積算値をｓｕｍ＾β_Ｌｋ，Ｋ’［ｍ_Ｃｖ］とすると、この積算値は次式となる。 Assume that the processing is started from the first layer and proceeds to the K layer, and the integrated value from the 1st to the Kth layers of the amount ^ β _{Lk, Cv} [m _Cv ] to which the surface of the object exists is represented by sum ^ β _{Lk, K} ′ [m _Cv ], The integrated value is given by the following equation.

ステップ１３：［ステップ６〜１２を繰り返し行うことにより、層状の全ての仮想的な面において色（第２の色）と物体の表面が存在する程度を積算する。]
ステップ６〜１２を繰り返し行うことにより、ユーザが見たい空間的な範囲で、層状になっている全ての仮想平面について、ステップ６〜１２を行い、それにより、色（第２の色）＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］と、物体の表面が存在する程度＾β_{Ｌｋ，Ｃｖ}［ｍ_Ｃｖ］を、仮想視線ごとに積算する。 Step 13: [By repeating Steps 6 to 12, the color (second color) and the extent to which the surface of the object exists in all layered virtual surfaces are integrated. ]
By repeating Steps 6 to 12, Steps 6 to 12 are performed for all the virtual planes that are layered in the spatial range that the user wants to see, so that the color (second color) ^ T _Lk ′ [m _Cv ] and the extent to which the surface of the object exists ^ β _{Lk, Cv} [m _Cv ] are integrated for each virtual line of sight.

それぞれの積算値は、（８．２０）、（８．２１）式のＫをＭにしたものであり、下記のようになる。 Each integrated value is obtained by changing K in the equations (8.20) and (8.21) to M, and is as follows.

ステップ１４：［仮想視線ごとに、色（第２の色）の積算値を、物体の表面が存在する程度の積算値で割って、色（第３の色）を求め、その色（第３の色）を、ユーザが見たい視野を表す仮想的な面上の画素の色とする。]
最後に、仮想視線ごと、つまり、仮想カメラの撮像素子の画素ごとに、色（第２の色）の積算値ｓｕｍ＾Ｔ_Ｌｋ’［ｍ_Ｃｖ］を、物体の表面が存在する程度の積算値＾β_Ｌｋ’［ｍ_Ｃｖ］で割って、色（第３の色）Ｔ［ｍ_Ｃｖ］を求め、その色（第３の色）Ｔ［ｍ_Ｃｖ］を、ユーザが見たい視野を表す仮想的な面上、つまり、仮想カメラの撮像素子上の画素の色とする。最終的な画素の色（第３の色）Ｔ［ｍ_Ｃｖ］は以下の式で表される。 Step 14: [For each virtual line of sight, the integrated value of the color (second color) is divided by the integrated value to the extent that the surface of the object exists to obtain the color (third color), and the color (third color) Is a color of a pixel on a virtual surface representing a visual field that the user wants to see. ]
Finally, for each virtual line of sight, that is, for each pixel of the image sensor of the virtual camera, the integrated value sum ^ T _Lk '[m _Cv ] of the color (second color) is integrated to the extent that the surface of the object exists. ^ Β _Lk ′ [m _Cv ] is divided to obtain a color (third color) T [m _Cv ], and the color (third color) T [m _Cv ] represents a virtual field of view desired by the user. The color of the pixel on the image plane, that is, on the image sensor of the virtual camera. The final pixel color (third color) T [m _Cv ] is expressed by the following equation.

本実施の形態の効果としては、
ユーザが見たい空間的な範囲内の仮想平面群と仮想視線との交点群に関して、ある１つの仮想平面上の全交点において、
（１）各交点上で物体の表面が存在する程度と色を求め（ステップ７、８、９）、
（２）各交点において物体の存在する程度と色の積を取って、それぞれの交点における色とし（ステップ１０）、
（３）上記（１）（２）を仮想平面１枚を計算するごとに、同じ仮想視線ごとに物体の存在する程度と色の積を積算し（ステップ１１、１３）、
（４）上記（１）を仮想平面１枚を計算するごとに、同じ仮想視線ごとに物体の表面が存在する程度を積算し（ステップ１２、１３）、
（５）全てに仮想平面についての計算が終了後、同じ仮想視線ごとに、物体の存在する程度と色の積を積算値を物体の表面が存在する程度を積算値で割って最終的な画像を生成する（ステップ１４）、
操作をしているので、（１）から（５）までの操作においては、物体の表面が存在する程度の計算と、色の計算と、それらの積の計算と、物体の表面が存在する程度の積算と、積の積算をするのに、最大でもそれぞれ仮想平面１枚ずつ、つまり、計５枚のメモリ量（５枚と交点の積に比例）のみ必要であり、従来方法で必要であった仮想平面の枚数のメモリ量（仮想平面の枚数と交点の積に比例）と異なるが、通常、良好な生成画像を得るには、仮想平面は最低３０枚程度必要なことが経験的に分かっており、そのことから、メモリ量が削減できる利点がある。 As an effect of this embodiment,
Regarding the intersection point group of the virtual plane group and the virtual line of sight within the spatial range that the user wants to see, at all the intersection points on a certain virtual plane,
(1) Obtain the degree and color of the surface of the object on each intersection (steps 7, 8, 9),
(2) The product of the degree and color of the object at each intersection is taken as the color at each intersection (step 10),
(3) Every time one virtual plane is calculated in (1) and (2) above, the product of the degree of existence of the object and the color is integrated for each same virtual line of sight (steps 11 and 13).
(4) Every time one virtual plane is calculated in (1) above, the degree to which the surface of the object exists for each same virtual line of sight is integrated (steps 12 and 13).
(5) After the calculation for the virtual plane is completed for all, the final image is obtained by dividing the product of the degree of existence and color by the integrated value and the degree of existence of the surface of the object by the integrated value for each virtual line of sight. (Step 14),
Since the operation is performed, in the operations from (1) to (5), the calculation to the extent that the surface of the object exists, the calculation of the color, the calculation of their product, and the extent to which the surface of the object exists In order to integrate the product and product, only one virtual plane is required at most, that is, only a total of 5 memory amounts (proportional to the product of 5 images and the intersection) are required. The number of virtual planes is different from the amount of memory (proportional to the product of the number of virtual planes and the intersection point), but it is empirically found that usually about 30 virtual planes are required to obtain a good generated image. Therefore, there is an advantage that the amount of memory can be reduced.

本発明の実施形態の画像生成装置は、以上に説明した各ステップを実行する手段を有しており、１または複数のコンピュータとプログラムを用いて構成することができる。そのプログラムは記録媒体に記憶することができる。また、そのプログラムの一部または全部の代わりにハードウェアを用いて構成してもよい。 The image generation apparatus according to the embodiment of the present invention includes means for executing the steps described above, and can be configured using one or a plurality of computers and a program. The program can be stored in a recording medium. Moreover, you may comprise using a hardware instead of a part or all of the program.

以上、本発明者によってなされた発明を、前記実施形態に基づき具体的に説明したが、本発明は、前記実施形態に限定されるものではなく、その要旨を逸脱しない範囲において種々変更可能であることは勿論である。 As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Of course.

画像生成を行うための構成図である。It is a block diagram for performing image generation. 仮想平面と仮想視線の交点と結像点の近傍の撮像素子上の画素の図（仮想平面と仮想視線の交点と２つのカメラ内の撮像素子の関係図）である。It is a figure of the pixel on the image sensor in the vicinity of the intersection of the virtual plane and the virtual line of sight and the imaging point (relationship diagram between the intersection of the virtual plane and the virtual line of sight and the image sensor in the two cameras). 仮想平面と仮想視線の交点と結像点の近傍の撮像素子上の画素の図（仮想カメラの撮像素子上の画素と、仮想平面と仮想視線の交点と、カメラ内の撮像素子の関係図）である。Diagram of the pixel on the image sensor in the vicinity of the intersection of the virtual plane and the virtual line of sight and the imaging point (the relationship between the pixel on the image sensor of the virtual camera, the intersection of the virtual plane and the virtual line of sight, and the image sensor in the camera) It is. 仮想平面と仮想視線の交点と結像点の近傍の撮像素子上の画素の図（注目点を中心とする撮像素子画素と同じ大きさの四角形と撮像素子の画素の関係図）である。It is a figure of the pixel on the image sensor in the vicinity of the intersection of the virtual plane and the virtual line of sight and the image formation point (relationship diagram of a rectangle having the same size as the image sensor pixel around the point of interest and the pixel of the image sensor). 従来技術の代表的な全体処理のフローを示す図である。It is a figure which shows the flow of the typical whole process of a prior art. 本発明の実施形態の全体処理のフローを示す図である。It is a figure which shows the flow of the whole process of embodiment of this invention. 本発明の実施形態の全体処理のフローを示す図（続き）である。It is a figure (continuation) which shows the flow of the whole process of embodiment of this invention.

Explanation of symbols

１００１はユーザが見たい画像を得るために便宜的に記した仮想カメラ、１００２は１００１の仮想カメラのレンズ、１００３はユーザが見たい視点（仮想視点）であり、１００２の仮想カメラのレンズの中心、１００４はユーザが見たい視野を表す仮想平面であり、１００１の仮想カメラの撮像素子、１００５はユーザが見たい視野を表す仮想平面の原点であり、１００４の仮想視点の撮像素子の原点、１００６はユーザが見たい視野を表す仮想平面上の画素であり、１００１の仮想カメラにおける後述の１０１２の仮想平面と視線との交点の結像点、１０１０はユーザが見たい視線（仮想視線）、１０１１はユーザが見たい空間的な範囲内の仮想平面群、１０１２は１０１１の層を形成している仮想平面と仮想視線の交点群、１１０１はカメラ、１１０２はカメラ１１０１のレンズ、１１０３はレンズ１１０２のレンズ中心、１１０４はカメラ１１０１の撮像素子、１１０５は撮像素子１１０４の原点、１１０６は仮想平面と仮想視線の交点の結像点、１１１０はカメラ視線、５００３は仮想カメラのレンズ中心、５００６は仮想カメラの撮像素子の注目画素の中心、５００６−１は仮想カメラの撮像素子の注目画素の近傍の中心、５０１０は仮想カメラの注目画素を通る仮想視線、５０１０−１は仮想カメラの注目画素の近傍画素を通る仮想視線、５０１１は仮想カメラの撮像素子の画素、５０１２はｋ層の仮想平面と注目画素を通る仮想視線の交点、５０１２−１はｋ層の仮想平面と注目画素の近傍画素を通る仮想視線の交点、５０１３はユーザが見たい空間的な範囲内の仮想平面群のｋ層、５１０６は仮想平面と仮想視線の交点の結像点、５１１０はカメラ視線、５１１１は仮想平面と仮想視線の交点の結像点を中心とする、撮像素子の画素と同じ大きさの四角形、５１１２は仮想平面と仮想視線の交点の結像点の近傍の撮像素子上の画素群、である。 1001 is a virtual camera described for convenience in order to obtain an image that the user wants to see, 1002 is a lens of the virtual camera 1001, 1003 is a viewpoint (virtual viewpoint) that the user wants to see, and the center of the lens of the virtual camera 1002 , 1004 is a virtual plane that represents the visual field that the user wants to see, 1001 is the virtual camera image sensor, 1005 is the virtual plane origin that represents the user's visual field, and 1004 is the virtual viewpoint image sensor origin, 1006 Is a pixel on a virtual plane that represents a visual field that the user wants to see. An imaging point of an intersection of a virtual plane 1012 (described later) and a line of sight in the virtual camera 1001, 1010 is a line of sight (virtual line of sight) 1011 that the user wants to see Is a group of virtual planes within a spatial range that the user wants to see, 1012 is a group of intersections of virtual planes forming a layer of 1011 and virtual lines of sight, Mera, 1102 is the lens of the camera 1101, 1103 is the lens center of the lens 1102, 1104 is the image sensor of the camera 1101, 1105 is the origin of the image sensor 1104, 1106 is the imaging point of the intersection of the virtual plane and the virtual line of sight, 1110 is the camera Line of sight, 5003 is the center of the lens of the virtual camera, 5006 is the center of the target pixel of the image sensor of the virtual camera, 5006-1 is the center near the target pixel of the image sensor of the virtual camera, and 5010 is a virtual passing through the target pixel of the virtual camera Line of sight, 5010-1 is a virtual line of sight passing through the neighboring pixels of the target pixel of the virtual camera, 5011 is a pixel of the imaging element of the virtual camera, 5012 is an intersection of the virtual plane of the k layer and the virtual line of sight passing through the target pixel, and 5012-1 is The intersection of the virtual line of sight passing through the virtual plane of the k layer and the neighboring pixel of the target pixel, 5013 is within the spatial range that the user wants to see The k layer of the virtual plane group, 5106 is the imaging point of the intersection of the virtual plane and the virtual line of sight, 5110 is the camera line of sight, and 5111 is the same as the pixel of the image sensor centering on the imaging point of the intersection of the virtual plane and the virtual line of sight A square of a size 5112 is a pixel group on the image sensor in the vicinity of the imaging point at the intersection of the virtual plane and the virtual line of sight.

Claims

A first means for inputting images obtained by photographing the same object or landscape using a plurality of cameras having different viewpoints;
A second means for setting a viewpoint (virtual viewpoint) that the user wants to see in the space;
Next, a third means for layering a plurality of virtual surfaces within a spatial range desired by the user,
A fourth means for setting a virtual surface representing a visual field that the user wants to see;
A fifth means for arranging a plurality of pixels on a virtual surface representing a visual field desired by the user so as to obtain a definition desired by the user;
Select one pixel, select one virtual plane, multiple images captured by multiple cameras at the intersection of the virtual viewpoint and a straight line (virtual line of sight) passing through the pixel and the virtual plane A sixth means for calculating each corresponding point in the above,
A seventh means for calculating the degree to which the surface of the object exists at the intersection with one virtual plane from the similarity between corresponding points on each image captured by a plurality of cameras and the images in the vicinity thereof; ,
From the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see, the object at the intersection of one virtual plane An eighth means for calculating the color of the surface (first color);
In the selected virtual surface, the object surface exists by performing the processing by means of the sixth to eighth means for all the intersections corresponding to all the pixels on the virtual surface representing the visual field that the user wants to see. A ninth means for calculating a degree and a color (first color);
The product of the degree of existence of the object and the color of the surface of the object (first color) at the intersection of the selected virtual plane and virtual line of sight is the color at the intersection (second color). Tenth means to:
If the color (second color) has been calculated in another virtual surface before and the color (second color) has been integrated, the color (first color) in the selected one virtual surface 2) is integrated into the integrated value for each virtual line of sight, and if there is no integrated value (that is, when the first color (second color) is calculated), that color (the second color) Eleventh means for setting the color) to an initial value of the integrated value;
If the degree to which the surface of the object exists in another virtual plane has been calculated before, and the degree to which the surface of the object exists has been integrated, the surface of the object in one selected virtual plane Is added to the integrated value for each virtual line of sight, and if there is no integrated value (that is, when calculating the degree to which the first object surface exists), the surface of the object exists. A twelfth means that sets an initial value of the integrated value as a degree to
For the extent to which the color (second color) and the surface of the object exist, processing by means of the sixth to twelfth means is performed, and each value is integrated for all virtual sight lines in all layered virtual surfaces. Means of
For each virtual line of sight, the integrated value of the color (second color) is divided by the integrated value to the extent that the surface of the object exists to obtain the color (third color), and the color (third color) is obtained. , A fourteenth means for setting the color of a pixel on a virtual surface representing a visual field desired by the user;
An image generation apparatus comprising:

A first means for inputting images obtained by photographing the same object or landscape using a plurality of cameras having different viewpoints;
A second means for setting a viewpoint (virtual viewpoint) that the user wants to see in the space;
Next, a third means for laying a plurality of virtual surfaces in layers within a spatial range that the user wants to see,
A fourth means for setting up a virtual surface representing the field of view the user wants to see;
A fifth means for arranging a plurality of pixels on a virtual surface representing a visual field desired by the user so as to obtain a definition desired by the user;
Select one pixel, select one virtual plane, multiple images captured by multiple cameras at the intersection of the virtual viewpoint and a straight line (virtual line of sight) passing through the pixel and the virtual plane A sixth means for calculating each corresponding point in the above,
A seventh means for calculating the degree to which the surface of the object exists at the intersection with one virtual plane from the similarity between corresponding points on each image captured by a plurality of cameras and the images in the vicinity thereof;
From the geometrical relationship between the color of the pixel at the corresponding point on each image captured by a plurality of cameras and the viewpoint that the user wants to see, the object at the intersection of one virtual plane An eighth means for calculating the color of the surface (first color);
In the selected virtual surface, the object surface exists by performing the processing by means of the sixth to eighth means for all the intersections corresponding to all the pixels on the virtual surface representing the visual field that the user wants to see. A ninth means for calculating the degree and color (first color)
The product of the degree of existence of the object and the color of the surface of the object (first color) at the intersection of the selected virtual plane and virtual line of sight is the color at the intersection (second color). Tenth means to
If the color (second color) has been calculated in another virtual surface before and the color (second color) has been integrated, the color (first color) in the selected one virtual surface 2) is integrated into the integrated value for each virtual line of sight, and if there is no integrated value (that is, when the first color (second color) is calculated), that color (the second color) An eleventh means having the color) as an initial value of the integrated value;
If the degree to which the surface of the object exists in another virtual plane has been calculated before, and the degree to which the surface of the object exists has been integrated, the surface of the object in one selected virtual plane Is added to the integrated value for each virtual line of sight, and if there is no integrated value (that is, when calculating the degree to which the first object surface exists), the surface of the object exists. A twelfth means that sets an initial value of the integrated value as a degree to
For the extent to which the color (second color) and the surface of the object exist, processing by means of the sixth to twelfth means is performed, and each value is integrated for all virtual sight lines in all layered virtual surfaces. The integrated value of the color (second color) is divided by the integrated value to the extent that the surface of the object exists for each virtual visual line, and the color (third color) is obtained. A color of a pixel on a virtual surface representing a visual field that the user wants to see,
As an image generation program for causing a computer to function.

A computer-readable recording medium on which the image generation program according to claim 2 is recorded.