JP3746380B2 - Spatial representation device using video images - Google Patents

Description

【００４０】
となる。×はベクトルの外積、・はベクトルの内積を示す。なお、Ｍ_x は画像フレーム２４の水平方向の画素数である。ｆは焦点距離でありｆ＝Ｍ_x ／｛２ｔａｎ（φ_i ／２）｝となる。
これより、
【００４１】
【数２】

【００４２】
を得る。
また、ｚ方向については、視点位置２８から注視点２６に向かう視線が画像面２９を交差する点について、その視点位置２８からのｘｙ平面上での距離は焦点距離ｆと異なる。これをｆ_v とおくと、
【００４３】
【数３】

【００４４】
である。
これより、
【００４５】
【数４】

【００４６】
であるので、
【００４７】
【数５】

【００４８】
を得る。
ここで、Ｍ_y は画像フレーム２４の水平方向の画素数である。
次に、ｒ_z は、視点位置から注視点２６までの距離の比とにより、
【００４９】
【数６】

【００５０】
とする。
これにより図８に示すように、画像フレーム２４にｒ_h とｒ_v の平行移動を行い、倍率ｒ_z の拡大あるいは縮小を施すことにより、中心２６ｂが画像中心にきて、注視点２６にある対象被写体の大きさが正しく描かれる変換画像２５を生成する。
【００５１】
また、特に、図９のように撮影経路が直線で、光軸が直線前方に向かう場合、の直線経路上、進行方向にｘ軸をとれば、ａ_x ＝ｑ_x ＋Ｄ、ａ_y ＝ｐ_iy＝０であるから、倍率ｒ_z は、
【００５２】
【数７】

【００５３】
となり、図１０のように拡大または縮小のみで扱うことができる。
【００５４】
さらに、図１１のように撮影経路６が直線で、光軸２７がその直線に直交する場合、やはり、この直線上で進行方向にｘ軸をとれば、ａ_x ＝ｑ_x 、ｑ_y ＝ｐ_iy＝０であるから、画像の変換は、図１２に示すように、
【００５５】
【数８】

【００５６】
の水平平行移動となる。このような場合、視点位置間の距離の差によりどのくらいの倍率、あるいは平行移動量が適当かをあらかじめビデオ映像データ７を用いて確認しておけば、より画質のよい映像を得ることができる。もちろん、被写体の詳細な形状や位置が既知の場合は、より精密に合致するよう変換することもできる。
【００５７】
なお、画像フレーム２４の平行移動や縮小においては、変換画像２５の周辺部に対応のつかない領域が発生することになる。これに対処するため、図１３に示すように、提示画像１３においてはその中心部のみを表示する。その適切なサイズはビデオ映像データ７によって異なってくるが、例えば、全体の９０％を表示する。
【００５８】
以上、説明したように、この実施の形態１によれば、視点位置の移動速度を変更した場合も、用いる画像フレームの順序およびフレーム速度を変えずに生成映像を合成するので、ビデオ映像データ７に示された対象物の微妙な動き、例えば木の葉の揺れ具合などが不自然でない状態で生成映像に再現できるビデオ映像による空間表現装置が得られる効果がある。
【００５９】
なお、生成映像１４の一例を図１４に示すが、これは図３５に示す例と対応させたものである。図３５に示す生成映像１４と比べ図１４に示す生成映像１４は対象物の微妙な動き、例えば木の葉の揺れ具合などが不自然でない状態で再現されるのがわかる。
この結果、例えば、木々に囲まれた環境内を歩調を変えて散策する場合などの感覚を不自然さなく与えることができるビデオ映像による空間表現装置が得られる効果がある。
【００６０】
実施の形態２．
図１６は、この発明の実施の形態２のビデオ映像による空間表現装置の構成を示すブロック図である。図１６において図１と同一または相当の部分については同一の符号を付し説明を省略する。図１６において、３０はビデオ映像データの次に提示するフレームを１つ選択するフレーム選択装置（フレーム選択手段）であるが、視点位置間の距離の差が所定の値を越えると、指定地点位置に最も近い視点位置をもつ画像フレームを選択する。
【００６１】
図１７は、この発明の実施の形態２のビデオ映像による空間表現装置の動作を示す説明図であり、図４と同一または相当の部分については同一の符号を付し説明を省略する。図において、３１は新たに選択された視点位置、３２は視点位置３１で撮影された画像フレームである。
【００６２】
この実施の形態２では、前記実施の形態１と同様にビデオ映像データ７をＩ_n （ｎ＝０，１，２，・・・Ｎ）、提示画像１３をＪとする。また、指定視点位置８は撮影経路６上にあり、この撮影経路上を移動するものとして考える。このとき、映像生成の過程で、指定視点位置ｑが現在選択されている画像フレームの撮影視点位置ｐ_i と大きく離れた場合、変換後においても歪みが大きくなり、情景が正しく表現されなくなる可能性が出てくる。そこで、ビデオ映像データ撮影時の視点移動速度と大きく異なる速度で指定視点を移動させても対処できるように適切な画像フレームに選択し直す機構を付加して構成する。
【００６３】
この結果、指定視点位置と選択しているビデオ映像データ７の画像フレームの撮影視点位置の差が大きくなった場合、新たに適切な画像フレームを選択し直すことにより、生成映像の画質を維持することが可能になる。
【００６４】
次に動作について説明する。
図１８は、この発明の実施の形態２のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。このフローチャートでは、前記実施の形態１で説明した図１５に示したフローチャートにステップＳＴ２０１、ステップＳＴ２０２が追加されており、図１５と同一または相当のステップについては同一の符号を付し説明を省略する。
【００６５】
ステップＳＴ２０１では、視点位置間の距離の差｜ｑ−ｐ_i ｜が所定の閾値Ｌ以下かどうかを判定する。この閾値Ｌは適宜設定してもよいが、例えばビデオ映像データ撮影時の撮影経路上１秒間の移動距離の平均値、というように設定する。Ｌ以下であればステップＳＴ２に戻り、そうでなければステップＳＴ２０２に進む。
【００６６】
ステップＳＴ２０２では、フレーム選択装置３０において新たな画像フレームを設定する。これはｐ_i とｑとの距離｜ｑ−ｐ_i ｜を最小とするような画像フレームである。変数ｉをこの新しい画像フレームの番号に置き換え、ステップＳＴ２に進む。
【００６７】
すなわち、図１７に示す画像フレーム２３が表示に用いられた後、ステップＳＴ２０１で「ＮＯ」となりステップＳＴ２０２に進んだ場合、フレーム選択装置３０は指定視点位置８に最も近い視点位置３１をもつ画像フレーム３２を選択する。
従って、次のステップＳＴ２では、画像フレーム２３に連続する画像フレーム２４ではなく、画像フレーム３２が読み出され、以下、この画像フレーム３２を起点に連続的に画像フレームが選択されていく。
【００６８】
以上、説明したように、この実施の形態２においては、指定視点位置と画像フレームの撮影視点位置との距離が大きく離れた場合、再度適切な画像フレームを選択し、その画像フレームからの映像に移行するように構成したので、大きく歩調を変えても情景の歪みが大きくならず、情景が正しく表現されなくなるのを回避できる空間表現装置が得られる効果がある。
【００６９】
実施の形態３．
図１９は、この発明の実施の形態３のビデオ映像による空間表現装置の構成を示すブロック図である。図１９において図１と同一または相当の部分については同一の符号を付し説明を省略する。
図１９において、３３はビデオ映像データについて使用画像フレームを切り替える際の移行点をあらかじめ定める移行点設定装置（移行点設定手段）、３４は移行点において次回用いる画像フレームを選択する機能を付加したフレーム選択装置（フレーム選択手段）である。
【００７０】
図２０、図２１、図２２はこの発明の実施の形態３のビデオ映像による空間表現装置の動作を示す説明図であり、図４と同一または相当の部分については同一の符号を付し説明を省略する。図において、３５は移行点での視点位置、３６は視点位置３５での画像フレーム、３７は移行先の視点位置、３８は視点位置３７での画像フレーム、３９は移行点に関連する画像フレーム、４０はビデオ映像データ７の前方への移行を判定するための距離、４１はビデオ映像データ７の後方への移行を判定するための距離である。
【００７１】
この実施の形態３でも前記実施の形態１と同様に、ビデオ映像データ７をＩ_n （ｎ＝０，１，２，・・・Ｎ）、提示画像１３をＪとする。また、指定視点位置８は撮影経路６上にあり、この撮影経路６上を移動するものとして考える。前記実施の形態２でも説明したように、映像生成の過程で、指定視点位置ｑが現在選択されている画像フレームの撮影視点位置ｐ_i と大きく離れた場合、変換後においても歪みが大きくなり、情景が正しく表現されなくなる可能性が出てくる。
そこで、この実施の形態３においては、ビデオ映像データ撮影時の視点移動速度と大きく異なる速度で指定視点を移動させても対処できるよう、チェックポイントである移行点を設け、適宜適切な画像フレームに移行するように構成する。
【００７２】
この結果、指定視点位置と選択しているビデオ映像データ７の画像フレームの撮影視点位置の差が大きくなった場合、画像フレームをあらかじめ設定してある画像フレーム間で選択し直すことにより、生成映像の画質を維持することが可能になる。
【００７３】
次に動作について説明する。
先ず、移行点設定装置３３による移行点の設定方法について説明する。図２３は、この移行点の設定方法を示すフローチャートである。
初期状態として前記実施の形態１に示した各数値が求まっている他、各ｐ_i について、その撮影経路上の始点２１からの距離が配列Ｘ［ｉ］により求まっているとする。これは例えば、隣接する画像フレームについての視点位置の差｜ｐ_i −ｐ_(i-1) ｜を積算して求めることができる。
【００７４】
ステップＳＴ３０１では、ビデオ映像データの画像フレームを示す変数ｉを“１”に初期化する。また、仮の移行点の画像フレームを示す変数ｈをＨｍｉｎに初期化する。
このＨｍｉｎは、移行点間の最小のフレーム差を表し、あらかじめ例えば１秒当たりのフレーム数３０というように設定する。また、移行点フレームを示す変数ＨもＨｍｉｎに初期化する。また、移行点の数を示す変数ｎを“１”に初期化し、それを格納する配列Ｔの第１の撮影経路上の始点２１からの距離として格納する。
さらに、誤差を示す変数Ｅを十分大きな値（例えば、画素値の取り得る値の二乗に画素数を乗じた値）に初期化する。
【００７５】
ステップＳＴ３０２では、ｉ番目の画像フレームとｈ番目の画像フレームとの不一致度を計算する。これは、ｉ番目の画像フレームを撮影視点ｐ_h からの情景に変換し、その変換画像ＫとＩ_h との画素値の誤差の二乗和をとることで行う。
この値をｅとする。なお、このとき用いる画像の変換は、前記実施の形態１において説明した変換である。
【００７６】
ステップＳＴ３０３では、Ｅ＞ｅかどうかを判定する。「ＹＥＳ」であるならステップＳＴ３０４へ進み、「ＮＯ」であればステップＳＴ３０５に進む。
【００７７】
ステップＳＴ３０４では、移行点を示す変数Ｈをｈに、誤差Ｅをｅに更新する。ステップＳＴ３０５では、ｈに“１”加算する。
【００７８】
ステップＳＴ３０６では、ｈがＮ以下であるか否かを判定し、「ＮＯ」であればステップＳＴ３０７でｎを“１”加算し、最後の移行点ＨをＴ［ｎ］に設定し、移行点設定処理を終了する。
この結果、ビデオ映像データ内でｎ個の移行点が設定され、それらのフレーム番号が配列Ｔに格納されることになる。
【００７９】
ステップＳＴ３０８では、ｈ−ｉがＨｍａｘ以下であるか否かを判定する。Ｈｍａｘは移行点間の最大のフレーム差を表し、あらかじめ、例えば５秒当たりのフレーム数５×３０＝１５０というように設定する。ステップＳＴ３０８で「ＹＥＳ」であればステップＳＴ３０２に戻り、「ＮＯ」であればステップＳＴ３０９に進む。
【００８０】
ステップＳＴ３０９では、移行点Ｈについて、その数を示す変数ｎを“１”増加させ配列Ｔに格納する。この移行点Ｈは、１つ前の移行点からフレーム幅Ｈｍｉｎ以上、Ｈｍａｘ以下の範囲の中で、その１つ前の移行点の画像フレームとの不一致度が最も小さくなる画像フレームである。
【００８１】
ステップＳＴ３１０では、移行点Ｈを新たなｉに、そのｉにＨｍｉｎを加算したものを新たなｈに設定し、またＥとＨとの再初期化を行う。
【００８２】
ステップＳＴ３１１ではｈがフレーム数Ｎ以下であるか否かを判定し、「ＹＥＳ」であればステップＳＴ３０２に戻り、「ＮＯ」であれば移行点設定処理を終了する。
【００８３】
以上の処理により、最小Ｈｍｉｎ、最大Ｈｍａｘの間隔をもつｎ個の移行点が設定される。この例を図２０に示す。ビデオ映像データ７上に移行点となる視点位置２１，３５ｂ，３５ｃ，３５ｄと、その画像フレーム３９ａ、３９ｂ，３９ｃ，３９ｄが設定される。これにより、隣接の移行点間で画像フレームを切り替えた場合、誤差が小さくなるため顕著な不連続が生じないように生成映像を構成することが可能になる。
【００８４】
次に、この実施の形態３による映像生成の動作について説明する。
図２４は、この映像生成の動作を示すフローチャートである。図２４において図１５と同一または相当のステップについては同一の符号を付し説明を省略する。
ステップＳＴ３５１では、移行点の番号を示すｍを１に設定し、その移行点をｈとする。
ステップＳＴ３５２では、ｈがｉより大きいか否かを判断し、「ＮＯ」であれば「ＹＥＳ」になるまで、ステップＳＴ３５３で移行点を順次変更していく。
ステップＳＴ３５４では、ｉが移行点ｈかどうかを判定する。この結果「ＹＥＳ」であればステップＳＴ３５５へ進み、「ＮＯ」であればステップＳＴ２へ戻る。
【００８５】
ステップＳＴ３５５では、視点位置間についてｔ−Ｘ［Ｔ［ｍ＋１］］≧０であるか否かを判定する。この結果、「ＹＥＳ」であればステップＳＴ３５６に進み、「ＮＯ」であればステップＳＴ３５７へ進む。
ただし、ｍ＋１がｎより大きくなるときは、Ｔ［ｍ＋１］に対する移行点は存在しないのでステップＳＴ３５７へ進む。
【００８６】
ステップＳＴ３５６では、画像フレームｉが移行点でありｐ_i からみて、指定視点ｑは撮影経路上、次の移行点Ｔ［ｍ＋１］以遠にあることになる。そこで、Ｔ［ｍ＋１］で示される次の移行点におけるフレームを用いて画像を生成した方が歪みが小さくなると期待できる。したがって、Ｔ［ｍ＋１］をｉと置き直す。さらに、Ｔ［ｍ＋２］をｈとし、ｍ＋２を新たにｍとする。ただし、ｍ＋２がｎより大きくなるときは、ｈとｍの更新は行わない。
これを図２１に示す。判定に用いた距離４０が正または“０”であれば、前回表示に用いた画像フレーム２３に対して、それに続く画像フレーム３６（視点位置３５）を取りやめ、指定視点位置８に近い画像フレーム３８（視点位置３７）を表示に用いる。画像フレーム３６、画像フレーム３８は移行点の画像フレームであるため滑らかな置き換え、移行が可能になる。
【００８７】
ステップＳＴ３５７では、視点位置間について、Ｘ［Ｔ［ｍ−１］］−ｔ≧０であるか否かを判定する。この結果、「ＹＥＳ」であればステップＳＴ３５８に進む。ただし、ｍが“１”のときは、Ｔ［ｍ−１］に対する移行点は存在しないので「ＮＯ」の場合と同様にステップＳＴ３５９へ進む。
ステップＳＴ３５９では移行点を更新する。ただし、ｍ＋１がｎより大きくなるときは、更新を行わない。
【００８８】
ステップＳＴ３５８では、画像フレームｉが移行点であり、ｐ_i からみて指定視点ｑは撮影経路上、前の移行点Ｔ［ｍ−１］以遠にあることになる。そこで、Ｔ［ｍ−１］で示される前の移行点におけるフレームを用いて画像を生成した方が歪みが小さくなると期待できる。したがって、Ｔ［ｍ−１］をｉと置き直す。
これを図２２に示す。判定に用いた距離４１が正または“０”であれば前回表示に用いた画像フレーム２３に対して、それに続く画像フレーム３６（視点位置３５）を取りやめ、指定視点位置８に近い画像フレーム３８（視点位置３７）を表示に用いる。画像フレーム３６、画像フレーム３８は移行点の画像フレームであるため滑らかな置き換え、移行が可能になる。
【００８９】
以上、説明したように、この実施の形態３によれば、映像を切り替える移行点を、映像に不連続を生じないようにあらかじめ設定し、設定した移行点間のみで映像を切り替えるように構成したので、例えば対象環境内を人や車などが横切っていくような場面が発生しても、その間には移行点が設定されず移行時にいきなり車が現れたり消えたりはしない。従って、移動する事物を含むビデオ映像データを用いる場合でも時間的に不連続性のない空間を表現できる空間表現装置が得られる効果がある。
【００９０】
実施の形態４．
図２５は、この発明の実施の形態４のビデオ映像による空間表現装置の構成を示すブロック図である。図２５において図１６と同一または相当の部分については同一の符号を付し説明を省略する。図２５において、４２は２つの画像フレームから提示画像を合成するとともに補間し、補間における割合を徐々に再選択された画像フレームに関して大きく設定し、再選択された画像フレームへ徐々に変化させ提示画像の不連続を顕著にさせないための画像合成装置（画像合成手段）である。
【００９１】
図２６は、このビデオ映像による空間表現装置の動作を示す説明図であり、図において、４３は画像フレームを変換して得られる第２の変換画像である。
【００９２】
この空間表現装置では、生成映像における移行点において所定のフレーム幅で徐々に映像を変化させるように移行させる。しかしながらビデオ映像データの移行点においては必ずしも情景を一致させられるとは限らない。そこで、この実施の形態では不連続を顕著にさせないため一方から他方の映像に徐々に変化させて映像生成を行う。
【００９３】
次に動作について説明する。
図２７は、この発明の実施の形態４のビデオ映像による空間表現装置による映像生成動作を示すフローチャートである。なお、図２７において図１８と同一または相当のステップについては同一の符号を付し説明を省略する。
ステップＳＴ４０１では、移行領域か否かを示す変数Ｆを“０”に初期化する。この変数Ｆが“０”のときは通常の画像生成を行い、正のときは移行領域とみて２つの画像フレームを合成して提示画像を得る。
【００９４】
ステップＳＴ２からステップＳＴ７の処理の後、ステップＳＴ４０２では、変数Ｆが“０”であるか否かを判定する。「ＹＥＳ」であればステップＳＴ２０１に進み、映像を移行するかどうかを判定する。また、変数Ｆが“０”でなければ、すでに移行領域であるのでステップＳＴ４０４以下の移行領域の処理を行う。
【００９５】
ステップＳＴ２０１へ進んで映像を移行すると判定した場合、すなわち「ＮＯ」の場合でありステップＳＴ４０３へ進む。ステップＳＴ４０３では画像フレーム番号ｉをｊに格納し、変数ＦをＷとする。Ｗは正の整数であり、２つの画像フレームを合成して生成する提示画像のフレーム数、すなわち、移行領域の幅である。この値はあらかじめ定め、例えば０．５秒分の１５フレームというようにする。
ステップＳＴ２０２終了後、移行先の画像フレーム３２が新たにｉで表される。ステップＳＴ２０２からステップＳＴ４０４へ進み、ステップＳＴ４０４以下では、前記新たな画像フレーム３２と、前回表示した画像フレーム２３に連続する画像フレーム２４（ｊ）とを混合し、徐々に移行先の映像に変化させていくように生成映像を合成していく。移行領域の処理であるステップＳＴ４０４では、通常のｉ番目の画像フレーム３２および変数に加え、ｊ番目の画像フレーム２４および変数を記憶装置から読み出す。
【００９６】
ステップＳＴ４０５では、ｉ番目の画像フレーム３２およびｊ番目の画像フレーム２４をそれぞれ変換し、図２６に示す変換画像２５（Ｋ）および第２の変換画像４３（Ｋ’）を生成する。この変換は上述のものと同一である。
【００９７】
ステップＳＴ４０６では、画像合成装置４２において、変換されたそれぞれの映像フレームを式（９）に示すように、
【数９】

のように合成して図２６に示す提示画像１３（Ｊ）を得る。これは、前記変数Ｆに従って徐々に移行後の映像フレームの比率を高めていくもので、Ｗフレーム目以後で完全に移行後の映像に切り替わる。なお、ここでＫ，Ｋ’に対する係数は、画像の画素値に係数を乗算することを表す。
【００９８】
ステップＳＴ４０７では、Ｊを画像表示装置２０により表示する。この場合もフレーム速度を撮影時の値に一致させる。ステップＳＴ４０８では、変数Ｆを“１”減じるとともに、ｊに“１”加算する。ここまでが移行領域の処理で、この後、ステップＳＴ５に進む。
【００９９】
以上、説明したように、この実施の形態４によれば、映像を切り替える移行点において幅Ｗフレームの移行領域を設定し、その領域で移行前後の映像を滑らかに徐々に変化させるようにしたため、移行部が顕著に表れない生成映像を合成できる空間表現装置が得られる効果がある。
【０１００】
実施の形態５．
図２８は、この発明の実施の形態５のビデオ映像による空間表現装置の構成を示すブロック図である。図２８において図１と同一または相当の部分については同一の符号を付し説明を省略する。図２８において、６１はスタート指定視点位置ｑに対し最も適切な画像フレームを選択するフレーム選択装置（フレーム選択手段）、６２はスタート指定視点位置ｑと光軸方向、画角を初期設定する視点位置設定装置（視点位置設定手段）である。
【０１０１】
図２９および図３０は、この発明の実施の形態５のビデオ映像による空間表現装置の動作を示す説明図であり、図４および図７と同一または相当の部分については同一の符号を付し説明を省略する。図において、４７はフレーム選択装置６１により初期段階で選択された画像フレーム、４８は画像フレーム４７の視点位置である。
【０１０２】
この実施の形態５では、指定視点位置ｑは撮影経路上に限らず自由に指定できる。
【０１０３】
次に動作について説明する。
図３１は、この発明の実施の形態５のビデオ映像による空間表現装置による映像生成動作を示すフローチャートである。なお、図３１において図１５と同一または相当のステップについては同一の符号を付し説明を省略する。
先ず、ステップＳＴ５０１では、視点位置設定装置６２により撮影経路上のスタート指定視点位置ｑと光軸方向Θ、画角Φを初期設定する。この場合、スタート指定視点位置ｑは撮影経路上には限定しない。続くステップＳＴ５０２では、フレーム選択装置６１によりスタート指定視点位置ｑに対し最も適切な画像フレーム４７（Ｉ_i ）を選択する。この場合、例えば、光軸方向のなす角が所定の閾値、例えば１５度以下であって、視点位置４８について視点間の差｜ｑ−ｐ_i ｜が最小となる画像フレームＩ_i というように選択する。この画像フレーム４７の番号を変数ｉに格納する。
【０１０４】
なお、ステップＳＴ２からステップＳＴ４までの動作は、前記実施の形態１で説明した動作と同一である。
ステップＳＴ５０３では、視点位置設定装置６２により視点の移動ｖと光軸方向の移動ｕ、画角の変化ｓを指定する。
続くステップＳＴ６，ステップＳＴ７は前記実施の形態１で説明した動作と同一である。
【０１０５】
ここで、この実施の形態のステップＳＴ３における画像変換について説明する。
前記実施の形態１と同様に、この実施の形態でも提示画像１３の注視点２６が正しく中心に描かれるようにする。このためには、図３０に示すように符号２６で示す注視点ａ＝Ｄ（ｃｏｓΘ，ｓｉｎΘ，０）＋ｑ＝（ａ_x ，ａ_y ，ａ_z ）として、
【０１０６】
【数１０】

【０１０７】
となるから、
【０１０８】
【数１１】

【０１０９】
を得る。ｒ_v に関しては前記実施の形態１と同様である。倍率ｒ_z は、視点位置から注視点２６までの距離の比と画角の差により、
【０１１０】
【数１２】

【０１１１】
となる。
【０１１２】
以上、説明したように、この実施の形態５によれば、ビデオ映像データの撮影時の撮影経路、および撮影経路上の指定視点位置によらず、動きのある環境内で歩調を変えて散策する場合などの感覚を与えることができる空間表現装置が得られる効果がある。
特に、指定視点位置８を撮影経路６に沿って移動させ、提示画像１３の光軸方向を画像フレーム４７の光軸方向の近傍で変化させたならば、ビデオ映像データ７に撮影された動きのある空間内を、左右を見渡しつつ、自由な歩調で散策する映像を生成できる。
【０１１３】
なお、以上説明した各実施の形態における記憶装置１６は、計算機の主記憶装置あるいは磁気ディスク装置あるいはビデオテープ装置およびビデオテープである。
【０１１４】
実施の形態６．
以上説明した各実施の形態において、前記実施の形態１のビデオ映像による空間表現装置では速度調整装置１７、フレーム選択装置１８、画像変換装置１９を構成要素として備え、また前記実施の形態２では特徴的な構成要素としてフレーム選択装置３０を備え、また前記実施の形態３では特徴的な構成要素として移行点設定装置３３、フレーム選択装置３４を備え、また前記実施の形態４では特徴的な構成要素として画像合成装置４２を備え、また前記実施の形態５では特徴的な構成要素としてフレーム選択装置６１、視点位置設定装置６２を備える構成であったが、これら各構成要素の動作を計算機によりソフトウェアで実現するようにしてもよい。
【０１１５】
実施の形態７．
前記各実施の形態では、生成映像１４のフレーム速度をビデオ映像データ７と同じとしたが、これを異なるように構成してもよい。この場合、仮に生成映像１４のフレーム速度がビデオ映像データ７の１／ｎ（単位時間に提示するフレームが１／ｎ）とすれば、ステップＳＴ６の動作はｉ←ｉ＋ｎとなる。
逆に、生成映像１４のフレーム速度がビデオ映像データ７のｎ倍（単位時間に提示するフレームがｎ倍）とすれば、ステップＳＴ６の動作は、ステップＳＴ６がｎ回実行される毎にｉが“１”加算されるような動作となる。
【０１１６】
実施の形態８．
前記実施の形態５では、連続したビデオ映像データ７を用いたが、複数に分けて撮影された複数のビデオ映像データ７を用いて構成してもよい。
【０１１７】
実施の形態９．
前記各実施の形態では、水平面をｘｙ平面内として説明したが、鉛直面など他の平面に設定してもよく、同一の動作となり、同一の効果が期待できる。
【０１１８】
実施の形態１０．
前記各実施の形態では、ビデオ画像データと提示画像の光軸をｘｙ平面に平行として説明したが、これに限るものではなく、ビデオ画像データと提示画像の光軸がｘｙ平面に平行でない場合でも、簡単な画像の変換によって対処できる。
【０１１９】
実施の形態１１．
前記各実施の形態では、ビデオ画像データと提示画像の水平軸をｘｙ平面に平行として説明したが、これに限るものではなく、ビデオ画像データと提示画像の水平軸がｘｙ平面に平行でない場合でも、簡単な画像の変換によって対処できる。
【０１２０】
実施の形態１２．
前記実施の形態４では、画像合成装置４２の行う演算を式（９）に示す画素値の補間混合として説明したが、合成する２つの変換画像間で被写体中の同一点を表す特徴点の対応付けを行い、それらが同一画素に置かれるように、互いを移動、変形させてから画素値を混合する、いわゆるモーフィングにより行うように構成してもよい。
【０１２１】
実施の形態１３．
前記各実施の形態では、画像フレームを順に１フレームずつ選択して提示したが、対象物の動く速度が定まっていない場合、指定視点位置の移動が速いときは、例えば、最大１フレームまでは画像フレームを間引いて選択する。また、指定視点位置の移動が遅いときは、例えば、同一画像フレームを最大２回までは続けて提示するというように構成してもよい。
【０１２２】
【発明の効果】
以上のように、この発明によれば、ビデオ映像データから選択された画像フレームを、撮影経路に対し指定された指定視点位置での情景に合致するよう画像変換する構成を備えるようにしたので、視点位置が異なるが被写体の動きを正しく示す画像フレームを選択することができ、被写体の動きを正しく再現する映像を生成できる効果がある。
【０１２３】
この発明によれば、ビデオ映像データから連続する画像フレームを順に１フレームずつフレーム選択手段が選択する構成を備えるようにしたので、視点位置は異なるが被写体の正しい動きを示す画像フレームを順に１フレームずつ選択し、被写体の動きを正しく再現する映像を生成できる効果がある。
【０１２４】
この発明によれば、速度調整手段により与えられる指定視点の移動量によって更新される指定視点位置と、選択されている画像フレームの撮影視点位置との間の距離が所定の距離以上離れていると、前記指定視点位置に近い撮影視点位置の画像フレームを再選択するフレーム選択手段を備えるように構成したので、再現する情景が実際のものと大きく異なることなく、映像空間を再現できる効果がある。
【０１２５】
この発明によれば、フレーム選択手段が選択する画像フレームに対応する移行点をあらかじめ設定する移行点設定手段を備え、指定視点位置に近い撮影視点位置の画像フレームを、前記移行点設定手段によりあらかじめ設定した移行点の画像フレーム間でフレーム選択手段が選択するように構成したので、移行点での再現される映像空間の不連続が顕著に現われない映像を生成できる効果がある。
【０１２６】
この発明によれば、フレーム選択手段がすでに選択した画像フレームと再選択した画像フレームを共に画像変換手段が画像変換し、前記画像変換手段が画像変換した変換画像を合成し、提示画像を補間し、補間における割合を徐々に前記再選択された画像フレームに関して大きくする画像合成手段を備えるように構成したので、映像を滑らかに移行することができ、不連続が顕著にならない映像を生成できる効果がある。
【０１２７】
この発明によれば、ビデオ映像データが撮影された空間内にスタート視点と光軸方向、画角を初期設定する視点位置設定手段を備え、前記視点位置設定手段により設定された前記スタート視点に最適な画像フレームをフレーム選択手段が選択し、前記フレーム選択手段により選択された画像フレームを、前記視点位置設定手段により設定されたスタート視点と光軸方向、画角をもとに画像変換手段が提示画像へ画像変換するように構成したので、ビデオ映像データの撮影時の撮影経路、および撮影経路上の指定視点位置によらず、ビデオ映像データに撮影された動きのある空間内で、前記初期設定されたスタート視点と光軸方向、画角に応じた映像を生成できる効果がある。
【０１２８】
この発明によれば、提示画像の光軸上一定距離にある被写体が提示画像上で正しく描画されるように、選択された画像フレームを画像変換手段が変換するように構成したので、注視する中心付近が正しく描かれる提示画像を生成できる効果がある。
【０１２９】
この発明によれば、選択された画像フレームの平行移動と拡大または縮小により画像変換手段が画像変換を行うように構成したので、簡単な演算で被写体の動きを正しく再現する提示画像を生成できる効果がある。
【０１３０】
この発明によれば、ビデオ映像データの撮影視点位置は直線上に並び、光軸はその直線上、撮影視点位置の進行方向に向かっており、指定視点位置と選択された画像フレームの前記撮影視点位置との距離によって決まる、画像フレームの拡大または縮小により画像変換手段が画像変換を行うように構成したので、簡単な演算で前進速度を任意に変えた映像を生成できる効果がある。
【０１３１】
この発明によれば、直線上の撮影経路上でその直線に直交する方向に向けて撮影されたビデオ映像データを用いる処理にあたり、画像フレームの変換を画像の平行移動で行うようにしたので、簡単な演算で並進速度を任意に変えた映像を生成できる効果がある。
【０１３２】
この発明によれば、ビデオ映像データの画像フレームより小さい変換画像を画像変換手段が生成するように構成したので、画像フレームの変換において発生する変換画像の周辺部の対応付けることができない領域が影響しない映像を生成できる効果がある。
【図面の簡単な説明】
【図１】 この発明の実施の形態１のビデオ映像による空間表現装置の構成を示すブロック図である。
【図２】 この発明の実施の形態１のビデオ映像による空間表現装置の動作を示す説明図である。
【図３】 この発明の実施の形態１のビデオ映像による空間表現装置の動作におけるカメラ配置を示す説明図である。
【図４】 この発明の実施の形態１のビデオ映像による空間表現装置の動作を示す説明図である。
【図５】 この発明の実施の形態１のビデオ映像による空間表現装置の記憶装置に記憶されるデータを示す説明図である。
【図６】 この発明の実施の形態１のビデオ映像による空間表現装置の動作を示す説明図である。
【図７】 この発明の実施の形態１のビデオ映像による空間表現装置の動における視点配置を示す説明図である。
【図８】 この発明の実施の形態１のビデオ映像による空間表現装置の動作における画像変換を示す説明図である。
【図９】 この発明の実施の形態１のビデオ映像による空間表現装置の動作を示す説明図である。
【図１０】 この発明の実施の形態１のビデオ映像による空間表現装置の動作における画像変換の一例を示す説明図である。
【図１１】 この発明の実施の形態１のビデオ映像による空間表現装置の動作を示す説明図である。
【図１２】 この発明の実施の形態１のビデオ映像による空間表現装置の動作における画像変換の一例を示す説明図である。
【図１３】 この発明の実施の形態１のビデオ映像による空間表現装置の動作における画像変換の一例を示す説明図である。
【図１４】 この発明の実施の形態１のビデオ映像による空間表現装置における生成映像の一例を示す説明図である。
【図１５】 この発明の実施の形態１のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。
【図１６】 この発明の実施の形態２のビデオ映像による空間表現装置の構成を示すブロック図である。
【図１７】 この発明の実施の形態２のビデオ映像による空間表現装置の動作における画像変換の一例を示す説明図である。
【図１８】 この発明の実施の形態２のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。
【図１９】 この発明の実施の形態３のビデオ映像による空間表現装置の構成を示すブロック図である。
【図２０】 この発明の実施の形態３のビデオ映像による空間表現装置の動作を示す説明図である。
【図２１】 この発明の実施の形態３のビデオ映像による空間表現装置の動作を示す説明図である。
【図２２】 この発明の実施の形態３のビデオ映像による空間表現装置の動作を示す説明図である。
【図２３】 この発明の実施の形態３のビデオ映像による空間表現装置の移行点の設定手順を示すフローチャートである。
【図２４】 この発明の実施の形態３のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。
【図２５】 この発明の実施の形態４のビデオ映像による空間表現装置の構成を示すブロック図である。
【図２６】 この発明の実施の形態４のビデオ映像による空間表現装置の動作を示す説明図である。
【図２７】 この発明の実施の形態４のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。
【図２８】 この発明の実施の形態５のビデオ映像による空間表現装置の構成を示すブロック図である。
【図２９】 この発明の実施の形態５のビデオ映像による空間表現装置の動作を示す説明図である。
【図３０】 この発明の実施の形態５のビデオ映像による空間表現装置の動作における視点配置を示す説明図である。
【図３１】 この発明の実施の形態５のビデオ映像による空間表現装置の映像生成動作を示すフローチャートである。
【図３２】 従来の空間表現装置の構成を示すブロック図である。
【図３３】 従来の空間表現装置の動作を示す説明図である。
【図３４】 従来の空間表現装置の動作を示す説明図である。
【図３５】 従来の空間表現装置による生成映像の一例を示す説明図である。
【図３６】 従来の空間表現装置による生成映像の一例を示す説明図である。
【符号の説明】
１６ 記憶装置（記憶手段）、１７ 速度調整装置（速度調整手段）、１８，３０，３４，６１ フレーム選択装置（フレーム選択手段）、１９ 画像変換装置（画像変換手段）、２０ 画像表示装置（画像表示手段）、３３ 移行点設定装置（移行点設定手段）、４２ 画像合成装置（画像合成手段）、６２ 視点位置設定装置（視点位置設定手段）。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a space expression apparatus using video images that presents a space for reproducing the motion as a virtual image using a video image obtained by photographing a space with movement.
[0002]
[Prior art]
As a conventional space expression device, there are techniques disclosed in Japanese Patent Application Laid-Open Nos. 9-16772 and 9-220308. FIG. 32 is a block diagram showing such a conventional space expression device. is there. 32, reference numeral 1 denotes a video image data and a storage device that stores the shooting viewpoint position and optical axis direction of each frame of the video image data, 2 a viewpoint input device that inputs a viewpoint position to be displayed, and 3 a viewpoint input device 2. A frame selection device 4 for selecting one image frame of video image data for a designated viewpoint, and 4 is an image display device for displaying the selected image frame.
[0003]
FIG. 33, FIG. 34, FIG. 35, and FIG. 36 are explanatory diagrams showing the operation of the conventional space expression device. In the figure, 5 is a video camera, 6 is a shooting path of the video camera 5, 7 is video image data shot by the video camera 5, 8 is a designated viewpoint position designated by the user, and 9 is shooting in the video image data 7. An image frame shot at the viewpoint position 10, 10 is a shooting viewpoint position at the time of shooting each frame on the shooting path 6, 11 is a close-up shooting viewpoint closest to the designated viewpoint position 8 in the shooting viewpoint position 10, and 12 is a video An image frame photographed at the close-up viewpoint 11 in the video data 7, 13 is a presentation image, 14 is a generated video expressed by updating the presentation image 13 as needed according to the change of the designated viewpoint position 8, and 15 is a space. It is trees which are an example of the moving object which exists in.
[0004]
Next, the operation will be described.
This conventional space expression apparatus generates a scene image at a designated viewpoint position 8 designated by a user from video image data 7 photographed on a photographing path 6 in a target space by a video camera 5 at any time. The video image data 7 is used to express the target space.
[0005]
Further, in this spatial expression device, as shown in FIG. 33, captured video image data 7 is stored in the storage device 1. As shown in FIG. 34, when the designated viewpoint position 8 is designated by the viewpoint input device 2, the frame selection device 3 refers to the position information of the photographing viewpoint position 10 stored in the storage device 1, and If there is a photographing viewpoint position that coincides with the designated viewpoint position 8, it is detected, and if there is no coincident photographing viewpoint position, the closest photographing viewpoint 11 that is closest to the designated viewpoint position 8 is detected. Then, for example, an image frame 12 that is one frame in the video image data 7 photographed at the close-up viewpoint 11 is read from the storage device 1. The read image frame 12 is presented to the user as a presentation image 13 by the image display device 4.
[0006]
[Problems to be solved by the invention]
Since the conventional spatial image representation apparatus using video images is configured as described above, the image frame 12 is selected only by the positional relationship between the designated viewpoint position 8 and the photographing viewpoint position 10 designated by the user, and the selected image frame is presented. Since it is the image 13, the frame order of the video image data 7, that is, the shooting time is not considered, and the image frame 12 is selected regardless of the shooting time and presented as the presentation image 13.
On the other hand, when the captured target space is a real environment such as a cityscape or natural scenery, there are moving objects in such a situation. However, the movement cannot be faithfully reproduced, and there is a problem that the space expressed by the generated video 14 is not accompanied by a sense of reality.
[0007]
More specifically, as shown in FIG. 35, when the designated viewpoint position 8 moves to 8a, 8b, and 8c in the figure, the close-up viewpoints 11a, 11b, and 11c are sequentially selected, and the image frames 12a, 12b, and 12c are selected. Constitutes the generated video 14 as the presentation images 13a, 13b, and 13c. For this reason, when the designated viewpoint position 8 is moved on the shooting path 6 at a higher speed than during shooting, frames are thinned out in the video image data 7 and the generated image 14 is generated. For example, the movement of the trees 15 swaying in the wind becomes faster than the actual movement.
[0008]
On the other hand, as shown in FIG. 36, when the designated viewpoint position 8 moves on the shooting path 6 at a lower speed than when shooting, as indicated by 8a, 8b, 8c in the figure, the close-up shooting viewpoint 11 is the same. Since the selected image frames 12 are successively presented as images 13a, 13b, and 13c, the trees 15 are stationary in the generated video 14. As described above, there is a problem that the real environment cannot be faithfully reproduced.
[0009]
The present invention has been made to solve the above-described problems, and is present in space by considering the shooting order of each image frame of video using video image data obtained by shooting the target space. An object of the present invention is to obtain a spatial expression device using video images that can generate an image that accurately reproduces the movement of an object as a virtual image.
[0010]
[Means for Solving the Problems]
According to the present invention, there is provided a video image spatial expression device that stores video image data and storage means for storing a shooting viewpoint position, an optical axis direction, and an angle of view of each image frame of the video image data, and specifies a video image data shooting path. Speed adjustment means for inputting the amount of movement of the designated viewpoint to be performed, and from the video image data stored in the storage means Sequentially every predetermined number of frames Frame selecting means for selecting an image frame to be presented, and an image frame selected by the frame selecting means Sequentially at that time Image conversion means for converting the image into a presentation image at the designated viewpoint, and the presentation image converted by the image conversion means Sequentially with the number of displays per unit time And an image display means for displaying a video space by a presentation image that displays a scene that matches the movement of the designated viewpoint.
[0011]
According to the present invention, there is provided a video image spatial expression device for selecting a sequence of image frames one by one from video image data in sequence. And image display means for sequentially displaying the presentation images sequentially obtained from the image conversion means at the number of display frames per unit time that matches the number of frames taken per unit time of the video image data. Is provided.
[0012]
According to the video image spatial expression device of the present invention, the distance between the designated viewpoint position updated by the movement amount of the designated viewpoint given by the speed adjusting unit and the photographing viewpoint position of the selected image frame is predetermined. A frame selection unit is provided that re-selects an image frame at a photographing viewpoint position close to the specified viewpoint position when the distance is more than a distance.
[0013]
The spatial representation apparatus using video images according to the present invention includes a transition point setting unit that presets a transition point corresponding to an image frame selected by a frame selection unit, and an image frame at a shooting viewpoint position close to the specified viewpoint position. Frame selection means for re-selecting between image frames at transition points preset by the transition point setting means is provided.
[0014]
According to the video image spatial representation apparatus of the present invention, an image conversion unit that converts both an image frame that has already been selected by the frame selection unit and a reselected image frame, and a converted image that has been converted by the image conversion unit are synthesized. The present invention further includes image synthesizing means for interpolating the presented image and gradually increasing the ratio of the interpolation with respect to the reselected image frame.
[0015]
According to the present invention, there is provided a video image spatial expression apparatus including a viewpoint position setting unit that initially sets a start viewpoint, an optical axis direction, and an angle of view in a space where video image data is captured, and is set by the viewpoint position setting unit. The frame selecting means selects an image frame optimal for the start viewpoint, and the image frame selected by the frame selecting means is selected based on the start viewpoint, the optical axis direction, and the angle of view set by the viewpoint position setting means. The image conversion means converts the image into a presentation image.
[0016]
According to the video image spatial expression device of the present invention, the image conversion means converts the selected image frame so that the subject at a fixed distance on the optical axis of the presentation image is correctly drawn on the presentation image. Is.
[0017]
In the video image spatial expression device according to the present invention, the image conversion means performs image conversion by translation and enlargement or reduction of the selected image frame.
[0018]
In the spatial image representation apparatus using video images according to the present invention, the shooting viewpoint positions of the video image data are arranged on a straight line, the optical axis is on the straight line toward the traveling direction of the shooting viewpoint position, and the selected viewpoint position is selected. The image conversion means performs image conversion by enlarging or reducing the image frame, which is determined by the distance of the image frame to the shooting viewpoint position.
[0019]
In the spatial image representation apparatus using video images according to the present invention, the shooting viewpoint positions of the video image data are arranged on a straight line, the optical axis is orthogonal to the straight line, and the specified viewpoint position and the shooting viewpoint position of the selected image frame The image conversion means performs image conversion by parallel movement of the image frame determined by the distance between
[0020]
A spatial expression device using video images according to the present invention, Image display means for displaying the center of the presentation image obtained from the image conversion means in a size smaller than the image frame of the video video data. It is what I did.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
An embodiment of the present invention will be described below.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a video image spatial expression apparatus according to Embodiment 1 of the present invention. In FIG. 1, reference numeral 16 denotes a frame number and a photographing viewpoint position of each frame of video image data and video image data. , A storage device (storage means) for storing the optical axis direction and the angle of view, 17 a speed adjustment device (speed adjustment means) for inputting the moving speed of the viewpoint, and 18 for sequentially selecting a frame to be presented next to the video image data. Frame selection device (frame selection means), 19 is an image conversion device (image conversion means) for converting the selected image frame, and 20 is an image display device (image display means) for displaying the converted image frame as a presentation image. It is.
[0022]
FIGS. 2 to 14 are explanatory diagrams showing the operation of the space representation apparatus using video images according to the first embodiment of the present invention. In the figure, 5 is a video camera, 6 is a shooting route of the video camera 5, 7 is video image data captured by the video camera 5, 8 is a designated viewpoint position designated by the user, and 9 is, for example, in the video image data 7 An image frame shot at the shooting viewpoint position 10 is a shooting viewpoint position at the time of shooting each frame on the shooting path 6, 11 is a close-up shooting viewpoint closest to the designated viewpoint position 8 in the shooting viewpoint position 10, and 12 is An image frame photographed from the close-up viewpoint 11 in the video image data 7, 13 is a presentation image, and 15 is a tree as an example of a moving object existing in the space.
[0023]
21 is the starting point of the shooting path 6, 22 is the previous position of the designated viewpoint position 8, 23 is the image frame used for the previous presentation, 24 is the image frame selected by the frame selection device 18, and 25 is the image frame 24 converted. The converted image 26, the gazing point in the presentation image 13, 27 the optical axis, 28 the viewpoint position of the selected image frame 24, and 29 the image plane.
[0024]
In the spatial representation apparatus using video images configured as described above, in order to faithfully reproduce the motion of the object, the image frame 9 of the video image data 7 is used as the presentation image 13 in accordance with the shooting order, thereby generating the image. For this reason, the movement of the object is reproduced at the same speed as that stored in the video image data 7, so that the reality of the space is not diluted in the generated image.
[0025]
Next, the operation will be described.
As shown in FIG. 2, video image data 7 composed of N image frames 9 is converted into I _n (N = 0, 1, 2,... N) and I _i Represents the i-th image frame. This video image data 7 is taken while moving the video camera 5 on the photographing path 6.
Here, the distance measured along the shooting path 6 from the start point 21 of the shooting path 6 is represented by t, and the point that is the distance t from the start point 21 is C (t). The horizontal plane is the xy plane, and as shown in FIG. 3, both the optical axis direction θ and the field angle φ are defined in the xy plane. Further, it is assumed that the video image data 7 is shot with the optical axis parallel to the xy plane and the horizontal axis parallel to the xy plane. The presentation image 13 is J.
[0026]
Furthermore, as shown in FIG. 35, the generated video 14 composed of the presentation image 13 is generated with the same video video data 7 and frame speed (the number of image frames shot or presented per unit time).
Then, the designated viewpoint position 8 is moved on the shooting path 6 at a speed different from that at the time of shooting, and a video at that time is generated from the video video data 7. Also, the designated viewpoint position 8 is q, and the shooting viewpoint position 10 of the image frame is p. _i Represented by
[0027]
After performing the above definition, first, the designated viewpoint position 8 indicated by q is set to the imaging viewpoint p of the image frame on the imaging path 6. _i Move from.
In this case, the presentation image J is p _i Image frame I with the shooting viewpoint _i Is selected and becomes the presentation image 13. From now on, image frame I _i The image frames that follow are sequentially selected and presented as the next presentation image J.
[0028]
However, since the viewpoint moving speed is variable, the designated viewpoint position 8 when the presentation image J is presented is not necessarily the image frame I selected at that time. _i Shooting point of view p _i Does not match. These are corrected and the image frame I _i Image frame I so as to match the scene at the designated viewpoint position 8 as much as possible. _i Is converted into a presentation image J.
As a result, the generated video 14 that maintains the speed of movement of the object and has a variable moving speed is synthesized.
[0029]
FIG. 15 is a flowchart showing the above-described video generation procedure, which will be described below according to this flowchart.
First, as an initial state, the video image data 7, the number N of frames shown in FIG. 5, and the position p of the shooting viewpoint of each image frame at the time of shooting. _i And the data of the photographing optical axis direction θ and the photographing field angle φ are stored in the storage device 16. The video image data 7 is captured by the video camera 5.
The photographing viewpoint position and the optical axis direction are measured and input by, for example, a GPS device or a gyro sensor, or are calculated from the image content of the previously input video image data 7 from the position in the image of a known point. Also good.
Further, when it can be considered that the robot has moved at a constant speed on a predetermined route, the shooting viewpoint position may be divided equally by the number of image frames taken. The angle of view is also obtained in advance by copying an existing pattern. If the angle of view is kept constant, subsequent processing becomes easy.
[0030]
First, in step ST1, a variable i indicating the number of an image frame used for presentation is initialized. This indicates the presentation video and viewpoint position at the start of processing, and is set to the first frame 1, for example. At the same time, the designated viewpoint position q on the shooting path 6 is set to p. _i Further, a variable t indicating the distance from the starting point 21 of the shooting route is initialized to t where C (t) = q, that is, a distance on the shooting route 6 from the starting point 21 to the designated viewpoint position q.
[0031]
In step ST2, the i-th image frame I from the video image data 7 stored in the storage device 16 by the frame selection device 18 is displayed. _i Is read. In addition, the viewpoint position p _i , Optical axis direction θ _i , Angle of view φ _i Is read.
[0032]
In step ST3, the image conversion device 19 uses the image frame I. _i Is converted to generate a converted image K that matches the scene from the designated viewpoint position q.
Details of this will be described later.
[0033]
In step ST4, the converted image K is displayed as a presentation image J by the image display device 20. However, in order to make the frame drawing speed of the video the same here, the drawing is switched from the image at the previous drawing to the current image at a predetermined timing. That is, when video image data having a frame drawing speed of 30 frames per second is used, a new image is drawn every 1/30 seconds. Here, it is assumed that the processing cycle from step ST2 to step ST7 can be sufficiently executed within this 1/30 second.
[0034]
In step ST5, the moving distance d of the designated viewpoint position 8 on the photographing path 6 is input by the speed adjustment device 17, and a new designated viewpoint is set. This is executed by an input device such as a mouse or a keyboard. When reaching step ST5, for example, a default value of d is determined, and if there is a specific key input, it is configured to increase or decrease d so that it does not wait for input of d.
[0035]
In step ST6, “1” is added to i. That is, an image frame that is continuous with the used image frame is used to generate the next presentation image.
[0036]
In step ST7, it is determined whether or not to continue the process. For this, it is checked whether or not a keyboard input or the like indicating an end designated in advance has been made. If it has been done, the process ends. If not, the process returns to step ST2. The process is also terminated when i becomes larger than N.
[0037]
In order to perform the processing as described above, as shown in FIG. 4, when the image frame 23 initially captured at the viewpoint position 22 is used, the “1” addition process of i in step ST6 performs the following step ST2. An image frame 24 photographed in succession to the image frame 23 is read out. This is different from the image frame 12 used in the conventional space expression device when the arrangement of the designated viewpoint position 8 is as shown in FIG. By presenting the image frame 24 following the image frame 23, it is possible to represent the correct movement of the object such as the trees 15.
[0038]
Here, the image conversion in step ST3 will be described.
Read image frame I _i Is the viewpoint position p _i In the optical axis direction θ _i , Angle of view φ _i Represents the scene at On the other hand, what is desired to be generated is a scene at the designated viewpoint position q at the same time in the same space. The optical axis and angle of view are _i , Φ _i Same as
Various methods are conceivable for the generation of the scene. Here, for the sake of simplicity, the subject position at a predetermined distance D on the optical axis 27 as shown in FIG. Convert to draw in size.
The predetermined distance D is set to 10 meters, for example. At this time, the gazing point 26 of the presented image is drawn with the correct size at the center of the presented image. Here, p _i (P _ix , P _iy , P _iz ), Q (q _x , Q _y , Q _z ), A = D (cos θ _i , Sinθ _i , 0) + q = (a _x , A _y , A _z ). In order to draw the gazing point 26 correctly, the movement r in the left-right direction shown below with respect to the image frame 24 _h (Positive rightward movement of image), vertical movement r _v (Positive upward movement of image) and magnification r _z Enlarge / reduce.
As shown in FIG. 7, in the xy plane, r _h And the ratio of f is equal to the ratio of the distance from the optical axis 27a of the image frame 24 of the gazing point 26 to the distance from the viewpoint position 28 of the perpendicular foot to the optical axis 27a of the gazing point 26,
[0039]
[Expression 1]

[0040]
It becomes. X indicates an outer product of vectors, and · indicates an inner product of vectors. M _x Is the number of pixels in the horizontal direction of the image frame 24. f is the focal length and f = M _x / {2 tan (φ _i / 2)}.
Than this,
[0041]
[Expression 2]

[0042]
Get.
In the z direction, the distance on the xy plane from the viewpoint position 28 is different from the focal distance f at the point where the line of sight from the viewpoint position 28 toward the gazing point 26 intersects the image plane 29. F _v After all,
[0043]
[Equation 3]

[0044]
It is.
Than this,
[0045]
[Expression 4]

[0046]
So
[0047]
[Equation 5]

[0048]
Get.
Where M _y Is the number of pixels in the horizontal direction of the image frame 24.
Next, r _z Is based on the ratio of the distance from the viewpoint position to the gazing point 26,
[0049]
[Formula 6]

[0050]
And
As a result, as shown in FIG. _h And r _v , And the magnification r _z Is applied to generate a converted image 25 in which the center 26b comes to the center of the image and the size of the target subject at the gazing point 26 is correctly drawn.
[0051]
In particular, as shown in FIG. 9, when the shooting path is a straight line and the optical axis is directed forward in the straight line, if the x axis is taken in the traveling direction on the straight path, a _x = Q _x + D, a _y = P _iy = 0, so magnification r _z Is
[0052]
[Expression 7]

[0053]
Thus, it can be handled only by enlargement or reduction as shown in FIG.
[0054]
Furthermore, when the imaging path 6 is a straight line and the optical axis 27 is orthogonal to the straight line as shown in FIG. _x = Q _x , Q _y = P _iy = 0, the image conversion is as shown in FIG.
[0055]
[Equation 8]

[0056]
Horizontal translation of In such a case, if the video image data 7 is used to check in advance how much magnification or the amount of parallel movement is appropriate depending on the difference in distance between the viewpoint positions, a video with better image quality can be obtained. Of course, if the detailed shape or position of the subject is known, it can be converted to match more precisely.
[0057]
Note that in the parallel movement or reduction of the image frame 24, an uncorresponding region occurs in the peripheral portion of the converted image 25. In order to cope with this, as shown in FIG. 13, only the central portion is displayed in the presentation image 13. The appropriate size varies depending on the video image data 7, but for example, 90% of the total size is displayed.
[0058]
As described above, according to the first embodiment, even when the moving speed of the viewpoint position is changed, the generated video is synthesized without changing the order and the frame speed of the image frames to be used. As a result, it is possible to obtain a space expression device using a video image that can be reproduced in a generated image in a state in which the subtle movements of the object shown in FIG.
[0059]
An example of the generated video 14 is shown in FIG. 14 and corresponds to the example shown in FIG. Compared with the generated image 14 shown in FIG. 35, it can be seen that the generated image 14 shown in FIG. 14 is reproduced in a state in which the subtle movements of the object, for example, the shaking of the leaves, etc. are not unnatural.
As a result, for example, there is an effect that it is possible to obtain a space expression device using video images that can give a sense of unnaturalness when walking in an environment surrounded by trees while changing the pace.
[0060]
Embodiment 2. FIG.
FIG. 16 is a block diagram showing the configuration of a spatial expression apparatus using video images according to Embodiment 2 of the present invention. In FIG. 16, the same or equivalent parts as in FIG. In FIG. 16, reference numeral 30 denotes a frame selection device (frame selection means) that selects one frame to be presented next to video image data. If the difference in distance between viewpoint positions exceeds a predetermined value, the designated point position The image frame having the viewpoint position closest to is selected.
[0061]
FIG. 17 is an explanatory diagram showing the operation of the video image spatial expression apparatus according to the second embodiment of the present invention. The same or corresponding parts as in FIG. In the figure, 31 is a newly selected viewpoint position, and 32 is an image frame taken at the viewpoint position 31.
[0062]
In the second embodiment, the video image data 7 is stored in the same manner as in the first embodiment. _n (N = 0, 1, 2,... N) and the presentation image 13 is J. Further, it is assumed that the designated viewpoint position 8 is on the shooting route 6 and moves on this shooting route. At this time, in the video generation process, the designated viewpoint position q is the shooting viewpoint position p of the currently selected image frame. _i If the distance is far away, the distortion will increase even after conversion, and the scene may not be represented correctly. Therefore, a mechanism for reselecting an appropriate image frame is added so as to cope with moving the designated viewpoint at a speed greatly different from the viewpoint moving speed at the time of video image data shooting.
[0063]
As a result, when the difference between the designated viewpoint position and the shooting viewpoint position of the image frame of the selected video image data 7 becomes large, the image quality of the generated image is maintained by newly selecting an appropriate image frame. It becomes possible.
[0064]
Next, the operation will be described.
FIG. 18 is a flowchart showing an image generation operation of the spatial expression apparatus using video images according to the second embodiment of the present invention. In this flowchart, step ST201 and step ST202 are added to the flowchart shown in FIG. 15 described in the first embodiment, and the same or equivalent steps as in FIG. .
[0065]
In step ST201, the difference in distance between the viewpoint positions | q−p _i It is determined whether or not | is equal to or less than a predetermined threshold value L. The threshold value L may be set as appropriate. For example, the threshold value L is set as an average value of the moving distance for one second on the shooting path when shooting video image data. If L or less, the process returns to step ST2, and if not, the process proceeds to step ST202.
[0066]
In step ST202, the frame selection device 30 sets a new image frame. This is p _i Distance between q and q _i An image frame that minimizes |. The variable i is replaced with the number of the new image frame, and the process proceeds to step ST2.
[0067]
That is, after the image frame 23 shown in FIG. 17 is used for display, if “NO” is determined in the step ST201 and the process proceeds to the step ST202, the frame selecting device 30 has an image frame having the viewpoint position 31 closest to the designated viewpoint position 8. 32 is selected.
Therefore, in the next step ST2, the image frame 32 is read instead of the image frame 24 continuous to the image frame 23, and the image frames are selected continuously from the image frame 32 as a starting point.
[0068]
As described above, in the second embodiment, when the distance between the designated viewpoint position and the photographing viewpoint position of the image frame is greatly separated, an appropriate image frame is selected again, and the image from the image frame is displayed. Since it is configured to shift, there is an effect that a spatial expression device can be obtained in which the distortion of the scene does not increase even if the pace is greatly changed, and the scene is prevented from being correctly expressed.
[0069]
Embodiment 3 FIG.
FIG. 19 is a block diagram showing a configuration of a spatial expression apparatus using video images according to Embodiment 3 of the present invention. In FIG. 19, the same or corresponding parts as those in FIG.
In FIG. 19, 33 is a transition point setting device (transition point setting means) that predetermines a transition point when switching the used image frame for video image data, and 34 is a frame with a function of selecting an image frame to be used next time at the transition point. A selection device (frame selection means).
[0070]
20, FIG. 21, and FIG. 22 are explanatory diagrams showing the operation of the space representation apparatus using video images according to the third embodiment of the present invention. The same or corresponding parts as those in FIG. Omitted. In the figure, 35 is the viewpoint position at the transition point, 36 is the image frame at the viewpoint position 35, 37 is the viewpoint position of the transition destination, 38 is the image frame at the viewpoint position 37, 39 is the image frame related to the transition point, 40 is a distance for determining the forward shift of the video image data 7, and 41 is a distance for determining the backward shift of the video image data 7.
[0071]
In this third embodiment, as in the first embodiment, the video image data 7 is converted to I _n (N = 0, 1, 2,... N) and the presentation image 13 is J. Further, it is assumed that the designated viewpoint position 8 is on the shooting route 6 and moves on the shooting route 6. As described in the second embodiment, the shooting viewpoint position p of the image frame in which the designated viewpoint position q is currently selected in the video generation process. _i If the distance is far away, the distortion will increase even after conversion, and the scene may not be represented correctly.
Therefore, in the third embodiment, a transition point as a check point is provided so that it can be dealt with even when the designated viewpoint is moved at a speed greatly different from the viewpoint moving speed at the time of video image data shooting, and an appropriate image frame is appropriately set. Configure to migrate.
[0072]
As a result, when the difference between the designated viewpoint position and the shooting viewpoint position of the image frame of the selected video image data 7 is increased, the generated image is selected by reselecting the image frame between preset image frames. Image quality can be maintained.
[0073]
Next, the operation will be described.
First, the transition point setting method by the transition point setting device 33 will be described. FIG. 23 is a flowchart showing a method for setting the transition point.
Each numerical value shown in the first embodiment is obtained as an initial state, and each p _i , It is assumed that the distance from the starting point 21 on the imaging route is obtained from the array X [i]. This is, for example, the difference in viewpoint position for adjacent image frames | p _i -P _(i-1) | Can be obtained by integrating.
[0074]
In step ST301, a variable i indicating an image frame of video image data is initialized to “1”. Also, the variable h indicating the image frame at the temporary transition point is initialized to Hmin.
This Hmin represents the minimum frame difference between the transition points, and is set in advance, for example, 30 frames per second. Also, the variable H indicating the transition point frame is initialized to Hmin. Further, a variable n indicating the number of transition points is initialized to “1”, and stored as a distance from the start point 21 on the first imaging path of the array T in which it is stored.
Further, the variable E indicating the error is initialized to a sufficiently large value (for example, a value obtained by multiplying the square of a value that the pixel value can take by the number of pixels).
[0075]
In step ST302, the degree of mismatch between the i-th image frame and the h-th image frame is calculated. This is because the i-th image frame is taken from the shooting viewpoint p. _h Converted into a scene from the image, and the converted images K and I _h This is done by taking the square sum of the error of the pixel value.
Let this value be e. The image conversion used at this time is the conversion described in the first embodiment.
[0076]
In step ST303, it is determined whether E> e. If “YES”, the process proceeds to step ST304, and if “NO”, the process proceeds to step ST305.
[0077]
In step ST304, the variable H indicating the transition point is updated to h, and the error E is updated to e. In step ST305, “1” is added to h.
[0078]
In step ST306, it is determined whether or not h is N or less. If “NO”, n is incremented by “1” in step ST307, and the last transition point H is set to T [n]. The setting process ends.
As a result, n transition points are set in the video image data, and their frame numbers are stored in the array T.
[0079]
In step ST308, it is determined whether hi is equal to or less than Hmax. Hmax represents the maximum frame difference between transition points, and is set in advance, for example, the number of frames per 5 seconds 5 × 30 = 150. If “YES” in the step ST308, the process returns to the step ST302, and if “NO”, the process proceeds to the step ST309.
[0080]
In step ST309, the variable n indicating the number of transition points H is increased by “1” and stored in the array T. This transition point H is an image frame that has the smallest discrepancy with the image frame at the previous transition point in the range of the frame width Hmin to Hmax from the previous transition point.
[0081]
In step ST310, the transition point H is set to a new i, the value obtained by adding Hmin to the i is set to a new h, and E and H are reinitialized.
[0082]
In step ST311, it is determined whether or not h is the number of frames N or less. If “YES”, the process returns to step ST302, and if “NO”, the transition point setting process is terminated.
[0083]
Through the above processing, n transition points having the minimum Hmin and maximum Hmax intervals are set. An example of this is shown in FIG. On the video image data 7, viewpoint positions 21, 35b, 35c, and 35d as transition points and their image frames 39a, 39b, 39c, and 39d are set. As a result, when an image frame is switched between adjacent transition points, the generated video can be configured so that a significant discontinuity does not occur because an error becomes small.
[0084]
Next, the video generation operation according to the third embodiment will be described.
FIG. 24 is a flowchart showing this video generation operation. In FIG. 24, steps that are the same as or equivalent to those in FIG.
In step ST351, m indicating the transition point number is set to 1, and the transition point is set to h.
In step ST352, it is determined whether or not h is greater than i. If “NO”, the transition point is sequentially changed in step ST353 until “YES”.
In step ST354, it is determined whether i is the transition point h. If the result is “YES”, the process proceeds to step ST355, and if “NO”, the process returns to step ST2.
[0085]
In step ST355, it is determined whether or not t−X [T [m + 1]] ≧ 0 between the viewpoint positions. As a result, if “YES”, the process proceeds to step ST356, and if “NO”, the process proceeds to step ST357.
However, when m + 1 is larger than n, there is no transition point for T [m + 1], and the process proceeds to step ST357.
[0086]
In step ST356, the image frame i is the transition point and p _i Accordingly, the designated viewpoint q is located beyond the next transition point T [m + 1] on the photographing route. Therefore, it can be expected that distortion is reduced by generating an image using a frame at the next transition point indicated by T [m + 1]. Therefore, T [m + 1] is replaced with i. Furthermore, T [m + 2] is set as h, and m + 2 is set as m newly. However, when m + 2 becomes larger than n, h and m are not updated.
This is shown in FIG. If the distance 40 used for the determination is positive or “0”, the subsequent image frame 36 (viewpoint position 35) is canceled with respect to the image frame 23 used for the previous display, and the image frame 38 close to the designated viewpoint position 8 is displayed. (Viewpoint position 37) is used for display. Since the image frame 36 and the image frame 38 are image frames at the transition point, smooth replacement and transition are possible.
[0087]
In Step ST357, it is determined whether or not X [T [m-1]]-t ≧ 0 between the viewpoint positions. As a result, if “YES”, the process proceeds to step ST358. However, when m is “1”, since there is no transition point for T [m−1], the process proceeds to step ST359 as in the case of “NO”.
In step ST359, the transition point is updated. However, no update is performed when m + 1 is greater than n.
[0088]
In step ST358, the image frame i is the transition point, and p _i Accordingly, the designated viewpoint q is located beyond the previous transition point T [m−1] on the photographing path. Therefore, it can be expected that distortion is reduced by generating an image using a frame at the previous transition point indicated by T [m−1]. Therefore, T [m−1] is replaced with i.
This is shown in FIG. If the distance 41 used for the determination is positive or “0”, the subsequent image frame 36 (viewpoint position 35) is canceled with respect to the image frame 23 used for the previous display, and the image frame 38 (close to the designated viewpoint position 8) ( The viewpoint position 37) is used for display. Since the image frame 36 and the image frame 38 are image frames at the transition point, smooth replacement and transition are possible.
[0089]
As described above, according to the third embodiment, the transition point for switching the video is set in advance so as not to cause discontinuity in the video, and the video is switched only between the set transition points. Therefore, for example, even if a scene where a person or a car crosses the target environment, a transition point is not set between them, and the car does not appear or disappear suddenly during the transition. Therefore, even when using video image data including moving objects, there is an effect that a space expression device capable of expressing a space without time discontinuity can be obtained.
[0090]
Embodiment 4 FIG.
FIG. 25 is a block diagram showing the configuration of a spatial expression apparatus using video images according to Embodiment 4 of the present invention. In FIG. 25, the same or corresponding parts as in FIG. In FIG. 25, reference numeral 42 synthesizes and interpolates the presentation image from two image frames, and gradually sets the ratio in the interpolation with respect to the reselected image frame, and gradually changes to the reselected image frame. This is an image composition device (image composition means) for preventing the discontinuity of the image from becoming noticeable.
[0091]
FIG. 26 is an explanatory diagram showing the operation of the space expression apparatus using video images. In FIG. 26, reference numeral 43 denotes a second converted image obtained by converting an image frame.
[0092]
In this spatial expression device, the transition is made so that the video is gradually changed with a predetermined frame width at the transition point in the generated video. However, scenes are not always matched at the transition point of video image data. Therefore, in this embodiment, in order not to make discontinuity conspicuous, video is generated by gradually changing from one video to the other.
[0093]
Next, the operation will be described.
FIG. 27 is a flowchart showing a video generation operation by the space video rendering apparatus according to Embodiment 4 of the present invention. In FIG. 27, steps that are the same as or equivalent to those in FIG.
In step ST401, a variable F indicating whether or not it is a transition area is initialized to “0”. When the variable F is “0”, normal image generation is performed. When the variable F is positive, the transition image is regarded as a transition area and two image frames are combined to obtain a presentation image.
[0094]
After the processing from step ST2 to step ST7, in step ST402, it is determined whether or not the variable F is “0”. If “YES”, the process proceeds to step ST201 to determine whether or not to transfer the video. If the variable F is not “0”, since it is already a transition area, the transition area processing from step ST404 is performed.
[0095]
When the process proceeds to step ST201 and it is determined that the video is to be transferred, that is, “NO”, the process proceeds to step ST403. In step ST403, the image frame number i is stored in j, and the variable F is set to W. W is a positive integer, and is the number of frames of the presentation image generated by combining two image frames, that is, the width of the transition area. This value is predetermined, for example, 15 frames for 0.5 seconds.
After step ST202, the transfer destination image frame 32 is newly represented by i. The process proceeds from step ST202 to step ST404, and after step ST404, the new image frame 32 and the image frame 24 (j) continuous with the image frame 23 displayed previously are mixed and gradually changed to the destination video. The generated video is synthesized as if going. In step ST404, which is the process of the transition area, in addition to the normal i-th image frame 32 and variables, the j-th image frame 24 and variables are read from the storage device.
[0096]
In step ST405, the i-th image frame 32 and the j-th image frame 24 are converted to generate a converted image 25 (K) and a second converted image 43 (K ′) shown in FIG. This conversion is the same as described above.
[0097]
In step ST406, in the image composition device 42, as shown in Expression (9), each converted video frame is expressed as follows:
[Equation 9]

As a result, the presentation image 13 (J) shown in FIG. 26 is obtained. This gradually increases the ratio of the video frame after the transition according to the variable F, and after the Wth frame, the video is completely switched to the video after the transition. Here, the coefficients for K and K ′ indicate that the pixel value of the image is multiplied by the coefficient.
[0098]
In step ST407, J is displayed by the image display device 20. Also in this case, the frame speed is made to coincide with the value at the time of shooting. In step ST408, the variable F is decremented by “1” and “1” is added to j. The process so far is the process of the transition area, and thereafter, the process proceeds to step ST5.
[0099]
As described above, according to the fourth embodiment, the transition area of the width W frame is set at the transition point for switching the video, and the video before and after the transition is smoothly and gradually changed in that area. There is an effect that a spatial expression device capable of synthesizing the generated video in which the transition part does not appear remarkably is obtained.
[0100]
Embodiment 5. FIG.
FIG. 28 is a block diagram showing the configuration of a spatial expression apparatus using video images according to Embodiment 5 of the present invention. In FIG. 28, the same or corresponding parts as in FIG. In FIG. 28, 61 is a frame selection device (frame selection means) that selects the most appropriate image frame for the start specified viewpoint position q, and 62 is a viewpoint position that initially sets the start specified viewpoint position q, the optical axis direction, and the angle of view. A setting device (viewpoint position setting means).
[0101]
FIGS. 29 and 30 are explanatory diagrams showing the operation of the video image spatial expression device according to the fifth embodiment of the present invention. The same or corresponding parts as those in FIGS. Is omitted. In the figure, 47 is an image frame selected at the initial stage by the frame selection device 61, and 48 is the viewpoint position of the image frame 47.
[0102]
In the fifth embodiment, the designated viewpoint position q can be freely designated not only on the shooting route.
[0103]
Next, the operation will be described.
FIG. 31 is a flowchart showing an image generation operation by the space expression apparatus for video images according to the fifth embodiment of the present invention. In FIG. 31, steps that are the same as or corresponding to those in FIG.
First, in step ST501, the viewpoint position setting device 62 initializes the start designated viewpoint position q, the optical axis direction Θ, and the angle of view Φ on the photographing path. In this case, the start designated viewpoint position q is not limited to the shooting route. In the subsequent step ST502, the most suitable image frame 47 (I _i ) Is selected. In this case, for example, the angle formed by the optical axis direction is a predetermined threshold, for example, 15 degrees or less, and the difference between viewpoints for the viewpoint position 48 | q−p _i Image frame I with minimum | _i Select as follows. The number of the image frame 47 is stored in the variable i.
[0104]
The operation from step ST2 to step ST4 is the same as the operation described in the first embodiment.
In step ST503, the viewpoint position setting device 62 designates the viewpoint movement v, the optical axis direction movement u, and the angle of view change s.
Subsequent steps ST6 and ST7 are the same as the operations described in the first embodiment.
[0105]
Here, the image conversion in step ST3 of this embodiment will be described.
Similar to the first embodiment, in this embodiment, the gazing point 26 of the presentation image 13 is drawn correctly at the center. For this purpose, as shown in FIG. 30, the gazing point a = D (cos Θ, sin Θ, 0) + q = (a _x , A _y , A _z As
[0106]
[Expression 10]

[0107]
So,
[0108]
## EQU11 ##

[0109]
Get. r _v This is the same as in the first embodiment. Magnification r _z Is the ratio of the distance from the viewpoint position to the gazing point 26 and the difference in the angle of view.
[0110]
[Expression 12]

[0111]
It becomes.
[0112]
As described above, according to the fifth embodiment, the user walks while changing the pace in a moving environment regardless of the shooting route at the time of shooting the video image data and the designated viewpoint position on the shooting route. There is an effect that a space expression device capable of giving a sense of the case can be obtained.
In particular, if the designated viewpoint position 8 is moved along the shooting path 6 and the optical axis direction of the presentation image 13 is changed in the vicinity of the optical axis direction of the image frame 47, the movement of the video image data 7 is recorded. It is possible to generate a video that walks in a free pace while looking over the left and right in a certain space.
[0113]
The storage device 16 in each embodiment described above is a main storage device of a computer, a magnetic disk device, a video tape device, or a video tape.
[0114]
Embodiment 6 FIG.
In each of the embodiments described above, the video image spatial representation device of the first embodiment includes the speed adjustment device 17, the frame selection device 18, and the image conversion device 19 as constituent elements. The frame selection device 30 is provided as a typical component, and the transition point setting device 33 and the frame selection device 34 are provided as characteristic components in the third embodiment, and a characteristic component in the fourth embodiment. In the fifth embodiment, the frame selecting device 61 and the viewpoint position setting device 62 are provided as characteristic constituent elements in the fifth embodiment. The operation of each constituent element is performed by software using a computer. It may be realized.
[0115]
Embodiment 7 FIG.
In each of the above embodiments, the frame speed of the generated video 14 is the same as that of the video video data 7, but it may be configured differently. In this case, if the frame rate of the generated video 14 is 1 / n of the video video data 7 (the frame to be presented per unit time is 1 / n), the operation of step ST6 is i ← i + n.
Conversely, if the frame rate of the generated video 14 is n times that of the video video data 7 (the number of frames to be presented per unit time is n times), the operation of step ST6 is performed every time step ST6 is executed n times. The operation is such that “1” is added.
[0116]
Embodiment 8 FIG.
In the fifth embodiment, continuous video image data 7 is used. However, a plurality of video image data 7 photographed separately may be used.
[0117]
Embodiment 9 FIG.
In each of the above embodiments, the horizontal plane is described as being in the xy plane. However, it may be set to another plane such as a vertical plane, and the same operation can be expected.
[0118]
Embodiment 10 FIG.
In each of the embodiments described above, the optical axis of the video image data and the presentation image is described as being parallel to the xy plane. However, the present invention is not limited to this, and the video image data and the optical axis of the presentation image are not parallel to the xy plane. Can be dealt with by simple image conversion.
[0119]
Embodiment 11 FIG.
In each of the embodiments, the horizontal axis of the video image data and the presentation image is described as being parallel to the xy plane. However, the present invention is not limited to this, and the horizontal axis of the video image data and the presentation image is not parallel to the xy plane. Can be dealt with by simple image conversion.
[0120]
Embodiment 12 FIG.
In the fourth embodiment, the calculation performed by the image synthesizing device 42 has been described as the interpolated mixture of pixel values shown in Expression (9). However, the correspondence of feature points representing the same point in the subject between two converted images to be synthesized It is also possible to perform so-called morphing, in which pixel values are mixed after being moved and deformed so that they are placed on the same pixel.
[0121]
Embodiment 13 FIG.
In each of the above-described embodiments, the image frames are selected and presented one by one in order. However, when the movement speed of the target object is not fixed and the designated viewpoint position moves quickly, for example, the image is displayed up to one frame. Select by thinning out the frame. Further, when the movement of the designated viewpoint position is slow, for example, the same image frame may be continuously presented up to twice.
[0122]
【The invention's effect】
As described above, according to the present invention, the image frame selected from the video image data is provided with a configuration for image conversion so as to match the scene at the designated viewpoint position designated with respect to the shooting path. Although the viewpoint position is different, an image frame that correctly shows the movement of the subject can be selected, and an image that correctly reproduces the movement of the subject can be generated.
[0123]
According to the present invention, since the frame selection means sequentially selects the successive image frames from the video image data one frame at a time, the image frames showing the correct movement of the subject with different viewpoint positions are sequentially ordered by one frame. It is possible to select images one by one and generate an image that correctly reproduces the movement of the subject.
[0124]
According to this invention, when the distance between the designated viewpoint position updated by the movement amount of the designated viewpoint given by the speed adjusting unit and the photographing viewpoint position of the selected image frame is more than a predetermined distance. Since the frame selection means for reselecting the image frame at the photographing viewpoint position close to the designated viewpoint position is provided, there is an effect that the video space can be reproduced without the scene to be reproduced being significantly different from the actual scene.
[0125]
According to this invention, the transition point setting means for setting in advance the transition point corresponding to the image frame selected by the frame selection means is provided, and the image frame at the photographing viewpoint position close to the designated viewpoint position is preliminarily set by the transition point setting means. Since the frame selection means selects between image frames at the set transition point, there is an effect that it is possible to generate a video in which the discontinuity of the video space reproduced at the transition point does not appear remarkably.
[0126]
According to the present invention, the image conversion unit converts the image frame already selected by the frame selection unit and the reselected image frame, the synthesized image is converted by the image conversion unit, and the presentation image is interpolated. Since the image synthesizing means for gradually increasing the ratio in the interpolation with respect to the reselected image frame is provided, it is possible to smoothly shift the image and generate an image in which the discontinuity is not remarkable. is there.
[0127]
According to the present invention, it is provided with the viewpoint position setting means for initially setting the start viewpoint, the optical axis direction, and the angle of view in the space where the video image data is photographed, and is optimal for the start viewpoint set by the viewpoint position setting means. The image selection means presents the image frame selected by the frame selection means based on the start viewpoint, the optical axis direction, and the angle of view set by the viewpoint position setting means. Since it is configured to convert the image into an image, the initial setting is performed in a moving space captured in the video image data regardless of the shooting path when shooting the video image data and the designated viewpoint position on the shooting path. It is possible to generate an image according to the start viewpoint, the optical axis direction, and the angle of view.
[0128]
According to the present invention, the image conversion means converts the selected image frame so that the subject at a fixed distance on the optical axis of the presentation image is correctly drawn on the presentation image. There is an effect that a presentation image in which the vicinity is correctly drawn can be generated.
[0129]
According to the present invention, since the image conversion means performs image conversion by translation and enlargement or reduction of the selected image frame, it is possible to generate a presentation image that correctly reproduces the movement of the subject with a simple calculation. There is.
[0130]
According to the present invention, the shooting viewpoint positions of the video image data are arranged on a straight line, the optical axis is on the straight line toward the traveling direction of the shooting viewpoint position, and the specified viewpoint position and the shooting viewpoint of the selected image frame are set. Since the image conversion means performs the image conversion by enlarging or reducing the image frame, which is determined by the distance from the position, there is an effect that it is possible to generate a video in which the forward speed is arbitrarily changed by a simple calculation.
[0131]
According to the present invention, in the processing using the video image data photographed in the direction orthogonal to the straight line on the straight photographing route, the image frame conversion is performed by the parallel movement of the image. It is possible to generate an image in which the translation speed is arbitrarily changed by simple calculation.
[0132]
According to the present invention, since the image conversion means generates the converted image smaller than the image frame of the video image data, the region that cannot be associated with the peripheral portion of the converted image generated in the conversion of the image frame is not affected. This has the effect of generating video.
[Brief description of the drawings]
BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a block diagram showing a configuration of a spatial expression device using video images according to Embodiment 1 of the present invention.
FIG. 2 is an explanatory diagram showing an operation of the space expression device using video images according to the first embodiment of the present invention.
FIG. 3 is an explanatory diagram showing a camera arrangement in the operation of the space representation apparatus for video images according to Embodiment 1 of the present invention;
FIG. 4 is an explanatory diagram showing an operation of the space expression device using video images according to the first embodiment of the present invention.
FIG. 5 is an explanatory diagram showing data stored in the storage device of the space representation device for video images according to Embodiment 1 of the present invention;
FIG. 6 is an explanatory diagram showing an operation of the space expression device using video images according to the first embodiment of the present invention.
FIG. 7 is an explanatory diagram showing a viewpoint arrangement in the movement of the space expression device using video images according to the first embodiment of the present invention;
FIG. 8 is an explanatory diagram showing image conversion in the operation of the space representation apparatus using video images according to Embodiment 1 of the present invention;
FIG. 9 is an explanatory diagram showing an operation of the spatial expression device using video images according to the first embodiment of the present invention.
FIG. 10 is an explanatory diagram showing an example of image conversion in the operation of the spatial expression device using video images according to the first embodiment of the present invention.
FIG. 11 is an explanatory diagram showing an operation of the space expression device using video images according to the first embodiment of the present invention.
FIG. 12 is an explanatory diagram showing an example of image conversion in the operation of the spatial expression device using video images according to the first embodiment of the present invention.
FIG. 13 is an explanatory diagram showing an example of image conversion in the operation of the spatial expression device using video images according to the first embodiment of the present invention.
FIG. 14 is an explanatory diagram showing an example of a generated image in the video image space representation device according to the first embodiment of the present invention.
FIG. 15 is a flowchart showing an image generation operation of the space expression device using video images according to the first embodiment of the present invention;
FIG. 16 is a block diagram showing a configuration of a space representation apparatus using video images according to Embodiment 2 of the present invention;
FIG. 17 is an explanatory diagram showing an example of image conversion in the operation of the space representation device using video images according to Embodiment 2 of the present invention;
FIG. 18 is a flowchart showing an image generation operation of the space expression device using video images according to the second embodiment of the present invention.
FIG. 19 is a block diagram showing a configuration of a space representation apparatus for video images according to Embodiment 3 of the present invention.
FIG. 20 is an explanatory diagram showing an operation of the space expression apparatus using video images according to the third embodiment of the present invention.
FIG. 21 is an explanatory diagram showing an operation of the space expression apparatus using video images according to the third embodiment of the present invention.
FIG. 22 is an explanatory diagram showing an operation of the space expression apparatus using video images according to the third embodiment of the present invention.
FIG. 23 is a flowchart showing a procedure for setting a transition point of the space representation device using video images according to the third embodiment of the present invention.
FIG. 24 is a flowchart showing an image generation operation of the space expression device using video images according to the third embodiment of the present invention.
FIG. 25 is a block diagram showing a configuration of a space expression device using video images according to Embodiment 4 of the present invention;
FIG. 26 is an explanatory diagram showing the operation of the space expression device based on video images according to the fourth embodiment of the present invention.
FIG. 27 is a flowchart showing an image generation operation of the space expression device using video images according to the fourth embodiment of the present invention;
FIG. 28 is a block diagram showing a configuration of a video space spatial expression device according to a fifth embodiment of the present invention.
FIG. 29 is an explanatory diagram showing an operation of the space expression device based on video images according to the fifth embodiment of the present invention.
FIG. 30 is an explanatory diagram showing viewpoint arrangement in the operation of the space representation device based on video images according to Embodiment 5 of the present invention;
FIG. 31 is a flowchart showing an image generation operation of the space expression device using video images according to the fifth embodiment of the present invention;
FIG. 32 is a block diagram showing a configuration of a conventional space expression device.
FIG. 33 is an explanatory diagram showing an operation of a conventional space expression device.
FIG. 34 is an explanatory diagram showing an operation of a conventional space expression device.
FIG. 35 is an explanatory diagram showing an example of a generated image by a conventional space expression device.
FIG. 36 is an explanatory diagram showing an example of a generated image by a conventional space expression device.
[Explanation of symbols]
16 storage device (storage means), 17 speed adjustment device (speed adjustment means), 18, 30, 34, 61 frame selection device (frame selection means), 19 image conversion device (image conversion means), 20 image display device (image Display means), 33 transition point setting device (transition point setting means), 42 image composition device (image composition means), 62 viewpoint position setting device (viewpoint position setting means).

Claims (11)

Storage means for storing the shooting viewpoint position, the optical axis direction, and the angle of view of each image frame of the video image data and video image data;
A speed adjusting means for inputting a movement amount of a designated viewpoint designated for a shooting route of video image data;
Frame selecting means for selecting image frames to be sequentially presented every predetermined number of frames from the video image data stored in the storage means;
Image conversion means for sequentially converting the image frames selected by the frame selection means into a presentation image at the specified viewpoint at that time ;
Image display means for sequentially displaying the presentation images converted by the image conversion means at a predetermined number of display times per unit time and expressing a video space by a presentation image showing a scene that matches the movement of the designated viewpoint; Spatial expression device using video images. The frame selection means selects consecutive image frames from the video image data one by one in order ,
The image display means sequentially displays the presentation images sequentially obtained from the image conversion means at a display frame number per unit time that matches the number of frames taken per unit time of the video image data . Spatial expression device using video images. When the distance between the designated viewpoint position updated by the amount of movement of the designated viewpoint given by the speed adjusting means and the shooting viewpoint position of the selected image frame is more than a predetermined distance, 3. The apparatus according to claim 1, wherein an image frame at a photographing viewpoint position close to the specified viewpoint position is reselected. A transition point setting means for presetting a transition point corresponding to an image frame selected by the frame selection means;
Frame selection means
4. The apparatus according to claim 3, wherein an image frame at a photographing viewpoint position close to a designated viewpoint position is selected between image frames at transition points preset by the transition point setting means. Image conversion means
The frame selection means converts both the image frame already selected and the reselected image frame,
The image conversion means comprises image synthesis means for synthesizing the converted images obtained by image conversion, interpolating the presented image, and gradually increasing the ratio in the interpolation with respect to the reselected image frame. 5. A space representation apparatus for video images according to claim 3 or claim 4. Provided with a viewpoint position setting means for initially setting the start viewpoint, the optical axis direction, and the angle of view in the space where the video image data was shot,
Frame selection means
Selecting an optimal image frame for the start viewpoint set by the viewpoint position setting means;
Image conversion means
The image frame selected by the frame selection unit is converted into a presentation image based on a start viewpoint, an optical axis direction, and an angle of view set by the viewpoint position setting unit. Spatial expression device using video images. Image conversion means
The selected image frame is converted so that a subject at a certain distance on the optical axis of the instruction image is correctly drawn on the presented image. A spatial expression device using the described video image. Image conversion means
8. The apparatus according to claim 7, wherein image conversion is performed by parallel movement and enlargement or reduction of the selected image frame. The shooting viewpoint positions of the video image data are arranged on a straight line, and the optical axis is on the straight line toward the traveling direction of the shooting viewpoint position.
Image conversion means
The image conversion is performed by enlarging or reducing the image frame determined by a distance between the designated viewpoint position and the photographing viewpoint position of the selected image frame. Spatial expression device using video images. The shooting viewpoint positions of the video image data are arranged on a straight line, and the optical axis is orthogonal to the straight line.
Image conversion means
7. The image conversion according to claim 1, wherein image conversion is performed by parallel movement of an image frame determined by a distance between a designated viewpoint position and the shooting viewpoint position of a selected image frame. Spatial expression device using video images. The image display means displays a central portion of a presentation image obtained from the image conversion means with a size smaller than an image frame of the video image data, according to any one of claims 1 to 10. Spatial expression device using video images.

Images