JP4599775B2 - Image processing apparatus, image processing method, image processing program, and information recording medium - Google Patents

Description

【００５０】
この閾値の設定では、図１１に示すように動き量のヒストグラムを求めて、このヒストグラムの極小値を利用して閾値Ｔh1，Ｔh2を求めるものとしても良い。このように、動き量の分布に応じて閾値Ｔhが動的に変化されるので、画像が１つのレイヤだけに分類されてしまうことがなく、動き量の分布に応じて良好にレイヤ分類を行うことができる。
【００５１】
分類処理部２６３では、閾値設定部２６４からの閾値Ｔhと動きベクトルＭＶの動き量に基づいて、各フレームの各検出領域がレイヤパターン情報ＬＰで作成が指示されたレイヤのいずれに属するかを判別してレイヤ分類を行う。また、水平方向に移動する検出領域やズームアウト動作方向の検出領域は、それぞれ対応するレイヤへの割当てを行う。この分類処理部２６３で注目フレームの各検出領域のレイヤ分類が完了したときには、このレイヤ分類の結果を示すレイヤ分類情報ＬＡを分類補正部２６５に供給する。
【００５２】
ここで、分類処理部２６３では、遠景，中景，近景の３つのレイヤを作成する際には、動き検出処理部２２からの動きベクトルＭＶを用いて、検出領域毎に前ｍフレーム分と後ｎフレーム分の動き量の平均値を算出する。例えば、正面画像の右側端部に設けた検出領域において、動きベクトルＭＶの動き量が時間の経過と共に図１２Ａに示すように変化したとき（図の数字は動き量を示すものとする）には、Ｆpフレームでの検出領域ＡR1に対して、Ｆpフレームでの検出領域ＡR1の動き量と、Ｆp-１〜Ｆp-mフレームの検出領域ＡR1の動き量と、Ｆp+１〜Ｆp+nフレームの検出領域ＡR1の動き量から、動き量の平均値を算出する。さらに、検出領域ＡR1の動き量として算出された平均値と、上述のようにして設定された閾値Ｔh1，Ｔh2を比較することで、このＦpフレームでの検出領域ＡR1が遠景，中景，近景のいずれのレイヤに属するかを判別する。このように、注目フレームの検出領域ＡR1について、前フレームや後フレームの動き量も用いて動き量の平均値を算出して、この平均値を用いて検出領域ＡR1のレイヤ分けを行うので、検出領域ＡR1の動き量に誤りが生じても、この検出領域ＡR1のレイヤ分類することができると共に、被写体の大きさの差や被写体までの距離の差によってフレーム毎の動き量の変動が生じても、これらの変動の影響を防止できる。また、他の領域や他のフレームに対しても同様に処理することで、図１２Ｂに示すように各検出領域を遠景，中景，近景のいずれかのレイヤに分類して、レイヤ分類情報ＬＡを分類補正部２６５に通知する。なお図１２Ｂにおいて、クロスハッチで示す検出領域は近景、斜線で示す検出領域は中景、他の検出領域は遠景として分類されたことを示している。
【００５３】
分類補正部２６５では、各検出領域のレイヤ分類情報ＬＡを参照して、同じレイヤに属する領域の連結数が所定の数よりも少ない領域は周囲に合わせたレイヤに補正する。例えば図１２Ｂでは、Ｆp+4フレームの領域ＡR4-gやＦp-2フレームの領域ＡR6-hでは、同じレイヤに属する領域が連結していないので、これらの領域を図１２Ｃに示すように周囲に合わせた領域に補正して、いずれの領域がいずれのレイヤとされているかを示すレイヤ分類情報ＬＢを、情報生成処理部２７に供給する。
【００５４】
また、追越し車両があるときには、上述の遠景，中景，近景だけでなく、図１２Ｄに示すように追い越し車両の画像が含まれる後退レイヤが生成される。ここで、追越し車両は、遠景や中景および近景の画像とは動きベクトルＭＶの方向が逆であることから、後退レイヤの時間軸方向は、他のレイヤとは方向を逆として生成する。
【００５５】
図１３は情報生成処理部２７の構成を示している。レイヤ分類部２６から供給されたレイヤ分類情報ＬＢは、動き量平均値算出部２７１と画像読み込み部２７２に供給される。
【００５６】
動き量平均値算出部２７１では、各検出領域の動き量を用いて、レイヤ毎の動き量の平均値をフレーム毎に算出する。例えばＦeフレームのときに、遠景レイヤとされた検出領域がｎy個であるときには、このｎy個の検出領域の動き量を用いて平均値を算出して画像読み込み部２７２に供給すると共に、この算出した動き量ＭＹvを中間画像情報として蓄積部３０の中間画像情報蓄積領域３２に記憶させる。
【００５７】
画像読み込み部２７２では、レイヤ分類部２６から供給されたレイヤ分類情報ＬＢに基づき、側端部から動き量平均値算出部２７１で算出された動き量ＭＹvに応じてレイヤ毎に画像信号ＳＤCを抽出してレイヤ毎の中間画像信号ＧＦvを生成して、中間画像信号補間部２７３に供給する。
【００５８】
ここで、各レイヤの動き量ＭＹvに応じた信号量だけ画像信号ＳＤCを側端部から読み出して中間画像信号を生成すると、例えば中間画像信号に基づく画像は図１４に示すものとなる。なお、図１４Ａは遠景レイヤの中間画像信号に基づく画像、図１４Ｂは中景レイヤの中間画像信号に基づく画像、図１４Ｃは近景レイヤの中間画像信号に基づく画像を示している。
【００５９】
中間画像信号補間部２７３では、動き量ＭＹvの異なる各レイヤの画像を遠景，中景，近景の順に貼り付けても、画像のないアンカバード(Uncovered)領域が生ずることがないように中間画像信号ＧＦvの補正を行い、補正された信号を中間画像信号ＧＹvとして中間画像情報蓄積領域３２に記憶させる。この中間画像信号の補正では、手前のレイヤに対して全てを補間すると、奥のレイヤが隠れてしまうこととなる。したがって、最も奥のレイヤに対しては隙間全部を補間するものとし、中間に位置するレイヤではこのレイヤよりも手前のレイヤに属する領域によって生成された隙間に対してのみ、水平方向に隣接する信号を用いた補間処理を行い、画像の無い隙間領域の画像信号を作り出して中間画像信号に貼り付ける。このように補間処理を行うことで、図１４Ａに示す中間画像信号ＧＦvに基づく画像は、画像の無い部分（斜線で塗りつぶした領域）が水平方向に隣接する信号を用いて補間されて、図１５Ａに示すような画像となる。同様に、図１４Ｂに示す中間画像信号ＧＦvに基づく画像は図１５Ｂに示すように補正された画像となる。なお、最も手前に位置するレイヤが例えば図１４Ｃに示す近景レイヤであるものとしたとき、最も手前に位置するレイヤではアンカバード領域が生じないので、補間処理は不要となり、図１５Ｃに示す近景レイヤは図１４Ｃと等しいものである。
【００６０】
蓄積部３０の中間画像情報蓄積領域３２には、中間画像情報生成ブロック２５で生成されたレイヤ毎の動き量ＭＹvとレイヤ毎の中間画像信号ＧＹvをフレーム毎に関係付けて、中間画像情報として記憶される。ここで、中間画像情報蓄積領域３２に記憶された遠景レイヤの中間画像信号は、動き量が少ないことから図１５Ａに示すように画像信号の信号量は少なくなる。また近景レイヤの中間画像信号では、動き量が大きいことから図１５Ｃに示すように信号量も多くなる。また中景レイヤでは図１５Ｂに示すように遠景と中景の中間である信号量となる。なお、後退レイヤでは、動き方向が逆であることから後退レイヤの中間画像信号では、図１５Ａ〜図１５Ｃとはフレーム方向が逆となる。
【００６１】
レイヤパターン情報ＬＰに基づいて水平移動レイヤを作成するものと指示されたときには、動きベクトルの方向が水平方向とされている検出領域の画像信号を、この水平移動レイヤの中間画像信号に設定する。ここで、水平方向に移動する画像では、正面画像から外に移動する画像と中に入り込む画像が生じる。例えば、右折時には、正面画像の右側端部から画像が入り込み左側端部から画像が出ていく。このため、外に移動する画像の水平移動レイヤは遠景レイヤ等と時間軸方向を等しくすると共に、中に入り込む画像の水平移動レイヤは後退レイヤと同様に、遠景レイヤ等とは時間軸方向を逆方向とする。
【００６２】
このようにして、中間画像情報蓄積領域３２には、遠景，中景，近景の各レイヤの中間画像信号と後退レイヤや水平移動レイヤの中間画像信号が動き量に応じて生成されて記憶される。また、各レイヤの動き量も上述のように合わせて記憶される。
【００６３】
次に、蓄積部３０の中間画像情報蓄積領域３２に記憶された中間画像情報を用いて右側面の画像の周辺画像信号ＳＤRを生成する場合について説明する。このように、周辺画像信号ＳＤRを生成する場合、中間画像情報蓄積領域３２から中間画像情報として記憶されている動き量ＭＹvを読み出して、レイヤ毎の中間画像信号ＧＹvの信号読み出し量を動き量ＭＹvに基づいて決定する。さらに、この動き量ＭＹvに応じた信号量の中間画像信号をレイヤ毎に使用して、遠景レイヤ，中景レイヤ、近景レイヤの順に貼り付ける。また、遠景，中景，近景のレイヤとは異なるレイヤが設けられているときには、このレイヤの画像の貼り付け処理も行い、周辺画像信号ＳＤRを生成できる。この生成した周辺画像信号ＳＤRを正面画像の画像信号ＳＤCの出力タイミングに合わせてプロジェクタ１２Rに供給することで、正面画像だけでなく右側面の画像も連続的に表示させることができる。
【００６４】
図１６は、中間画像情報蓄積領域３２に記憶されている中間画像情報を読み出して周辺画像信号を生成する周辺画像信号生成ブロック４０の構成を示している。周辺画像信号生成ブロック４０の画像生成制御部４１では、中間画像情報蓄積領域３２に中間画像情報として記憶されているレイヤ毎の動き量ＭＹvをフレーム順に読み出して、この読み出した動き量ＭＹvに基づきレイヤ毎の中間画像信号の読み出し量を判別する。この画像生成制御部４１で判別された読み出し量に基づいて、中間画像情報蓄積領域３２から中間画像信号ＧＹvをレイヤ毎に読み出して、レイヤ画像生成処理部４２に設けられているレイヤ毎の貼り付け部４２１-1〜４２１-5に供給する。
【００６５】
ここで、右左折時に水平方向に移動する水平移動レイヤの中間画像信号は貼り付け部４２１-1に供給する。また、遠景レイヤの中間画像信号は貼り付け部４２１-2に供給すると共に、中景レイヤの中間画像信号および近景レイヤの中間画像信号は貼り付け部４２１-3，４２１-4に供給する。さらに追越し車両の画像が含まれる後退レイヤの中間画像信号は貼り付け部４２１-5に供給する。
【００６６】
貼り付け部４２１-1には、後述する画像シフト部４２３-1が接続されており、画像シフト部４２３-1から供給された画像信号に、中間画像情報蓄積領域３２から読み出した対応するレイヤの画像信号を貼り付ける。この貼り付け部４２１-1によって画像の貼り付けが行われて得られた水平移動レイヤの画像信号は、１フレーム遅延部４２２-1と画像信号生成部４４の遠景貼り付け部４４１に供給される。
【００６７】
１フレーム遅延部４２２-1では、貼り付け部４２１-1から供給された画像信号を１フレーム分だけ遅延させて画像シフト部４２３-1に供給する。画像シフト部４２３-1では、１フレーム遅延部４２２-1から供給された画像信号に基づく画像を、画像生成制御部４１から供給された水平移動レイヤの動き量ＭＹvに基づいて水平方向に移動させる。さらにこの水平方向に移動させた画像の画像信号を貼り付け部４２１-1に供給する。
【００６８】
このように１フレーム分遅延された画像信号に基づく画像を、画像生成制御部４１から供給された動き量ＭＹvに基づいて水平方向に移動させると共に、この水平方向に移動された画像に中間画像情報蓄積領域３２から動き量ＭＹv分だけ読み出した水平移動レイヤの中間画像信号ＧＹvを貼り付けることで、移動する画像の画像信号を生成できる。
【００６９】
同様に、１フレーム遅延部４２２-2〜４２２-5で１フレーム分遅延された画像信号に基づく画像を、画像シフト部４２３-2〜４２３-5で画像生成制御部４１から供給された各レイヤの動き量ＭＹvに基づいて水平方向に移動させると共に、この水平方向に移動された画像に対して、中間画像情報蓄積領域３２から各レイヤの動き量分だけ読み出した中間画像信号ＧＹvを貼り付け部４２１-2〜４２１-5に供給して、この読み出した中間画像信号ＧＹvに基づく画像を貼り付けることで、順次画像が移動する画像信号をレイヤ毎に生成できる。
【００７０】
また、中間画像信号は、正面画像信号から読み出した信号であることから、この中間画像信号に基づく画像は、スクリーン１０Cと同じ面上の画像となる。しかし、周辺画像を表示するスクリーン１０Rは正面のスクリーン１０Cに対して傾きを持って設けられている。このため、遠景，中景，近景レイヤや後退レイヤのように、動きベクトルの方向が画像基準位置ＣＰの方向あるいは逆方向であるレイヤの画像信号を用いて、各レイヤの画像の貼り付けを行い周辺画像信号ＳＤRを生成して画像表示を行うものとすると、スクリーン１０Rに表示される画像は、スクリーン１０Cに対するスクリーン１０Rの傾きによって正しい形状とならない。このため、動きベクトルの方向が画像基準位置ＣＰの方向あるいは逆方向であるレイヤの画像信号は、射影変換部４３の各変換処理部４３１〜４３４に供給して、この各レイヤの画像をスクリーン１０L，１０Rに表示したときに、動きベクトルＭＶのベクトル方向に画像が正しく移動するように射影変換を行う。また、水平移動レイヤの画像は、動きベクトルＭＶの方向が水平方向であることから、遠景，中景，近景レイヤ，後退レイヤのような射影変換は行わないものとする。
【００７１】
射影変換部４３の変換処理部４３１〜４３４では、貼り付け部４２１-2〜４２１-5から供給された画像信号に対して、画像がスクリーン１０Rの向きに対応した形状となるように射影変換を行う。ここで、正面画像から画像信号を順次読み出すと、読み出した画像の表示面１０CRとスクリーン１０Rは、図１７Ａおよび図１７Ｂに示すように同一面でないことから、正面画像から読み出した画像をスクリーン１０Rに表示したときには、表示面１０CRの画像をそのままスクリーン１０Rの面上に投射したものとなり、正しい形状とならない。
【００７２】
このため、射影変換を行って、垂直方向は画像のズームインあるいはズームアウトの中心Ｃzmから距離に比例した拡大率で拡大すると共に、水平方向はその速度がズームインあるいはズームアウトの中心Ｃzmからの距離に比例するような拡大率で拡大する。
【００７３】
ここで、図１７Ｃに示すように、右側面画像の画面前端から中心Ｃzmまでの長さをＬ、右側面画像の周辺方向の長さをＡ、正面画像と右側面画像とが成す角度「θs」に基づいた比例定数をαとすると、正面画面端からそのまま読み出した画像の位置（ｘa，ｙa）と右側面画像上に正しく画像表示できるように変換処理した後の位置（Ｘa，Ｙa）は、垂直方向については式（３）の関係となり、水平方向については式（４）の関係となる。このため、図１７Ｄに示すように、スクリーン１０R上の（Ｘa，Ｙa）の画像信号は、式（３），式（４）を満たす位置（ｘa，ｙa）の信号を用いることで、図１７Ｅに示すようにスクリーン１０R上に正しく右側面画像を表示できる。
【００７４】

【００７５】
この変換処理部４３１で得られた射影変換後の遠景レイヤの画像信号は、画像信号生成部４４の遠景貼り付け部４４１に供給される。また、変換処理部４３２で得られた射影変換後の中景レイヤの画像信号は中景貼り付け部４４２に供給されると共に、変換処理部４３３で得られた射影変換後の近景レイヤの画像信号は近景貼り付け部４４３に供給される。さらに、変換処理部４３４で得られた後退レイヤの画像信号は縮小貼り付け部４４４に供給される。
【００７６】
遠景貼り付け部４４１では、貼り付け部４２１と変換処理部４３１から供給された画像信号に基づいて、水平移動レイヤの画像に遠景レイヤの画像を貼り付けた画像信号を生成して中景貼り付け部４４２に供給する。
【００７７】
中景貼り付け部４４２では、変換処理部４３２と遠景貼り付け部４４１から供給された画像信号に基づいて、遠景レイヤの画像が貼り付けた画像に中景レイヤの画像を貼り付けた画像信号を生成して近景貼り付け部４４３に供給する。
【００７８】
近景貼り付け部４４３では、変換処理部４３３と中景貼り付け部４４２から供給された画像信号に基づいて、中景レイヤの画像が貼り付けた画像に近景レイヤの画像を貼り付けた画像信号を生成して縮小貼り付け部４４４に供給する。
【００７９】
縮小貼り付け部４４４では、変換処理部４３４と近景貼り付け部４４３から供給された画像信号に基づいて、近景レイヤの画像が貼り付けた画像に、時間と共に画像が縮小する後退レイヤの画像を貼り付けた画像信号を生成する。この縮小貼り付け部４４４で生成された画像信号は、水平移動レイヤから縮小レイヤまでの各レイヤが貼り付けられた側面画像の画像信号となる。
【００８０】
このため、縮小貼り付け部４４４で生成された画像信号を周辺画像信号ＳＤRとしてプロジェクタ１２Rに供給することで、正面画像と連続した右側面の画像をスクリーン１０Rに表示することができる。
【００８１】
また、中間画像情報記憶領域３２に記憶された中間画像信号に対して、画像の無い部分を補間する処理が行われていないとき、あるいは縮小貼り付け部４４４で生成された画像信号に画像の無い部分が生ずるときには、前補間処理部を設けて、画像の無い部分と隣接する領域でどのようなレイヤの画像が貼り合わされたを判別して、奥行きの深いレイヤの画像を用いて補間処理を行う。例えば、中景レイヤと近景レイヤの画像が貼り合わされている領域と水平方向に隣接する部分で画像の無い部分が生じてときには、奥行きの深い中景レイヤの画像を用いて補間処理を行うことで、画像の欠落のない良好な周辺画像信号ＳＤRを生成できる。
【００８２】
また、周辺画像信号ＳＤRと同様にして周辺画像信号ＳＤLを生成してプロジェクタ１２Lに供給することで、正面画像と連続した左側面の画像をスクリーン１０Lに表示することができる。
【００８３】
さらに、蓄積部３０には、中間画像情報を蓄積するものであり、この中間画像情報は動き量とレイヤ毎の中間画像信号であることから、周辺画像の画像信号を蓄積する場合に比べて情報量を少なくできる。このため、大容量の記録媒体等を用いることなく広画角での画像表示を行うことができる。
【００８４】
ところで、上述の実施の形態では、正面画像の側端側に設けた検出領域の動きベクトルＭＶから、遠景，中景，近景等の画像のレイヤ分けや動き量の判別等を行い、中間画像情報を生成して中間画像情報蓄積領域３２に記憶させると共に、周辺画像を表示させる際には、中間画像情報蓄積領域３２に蓄積されている中間画像情報を用いて、各レイヤの動き量ＭＹvに応じて中間画像信号ＧＹvを読み出して画像の貼り付けを行うことで周辺画像信号を生成するものとしたが、周辺画像信号は動きベクトルＭＶを用いることなく生成することもできる。次に第２の実施の形態として、動きベクトルＭＶを用いることなく周辺画像信号を生成できる画像信号処理装置について説明する。
【００８５】
この第２の実施の形態の画像信号処理装置５５では、幾何変換によって正面画像から側面方向の射影画像を生成し、この射影画像を積分処理して積分画像を生成すると共に、この積分画像から画像のレイヤ分けを行い、レイヤ毎の動き量ＭＹsやレイヤ毎の中間画像信号を生成して、中間画像情報として中間画像情報蓄積領域３２に記憶させるものである。
【００８６】
図１８は、画像処理装置５５の構成を示している。正面画像の画像信号ＳＤCは、幾何変換部６１に供給される。幾何変換部６１では、正面画像の画像信号ＳＤCを用いて正面画像を進行方向の面である射影面に表示した画像の射影画像信号ＳＤPを幾何変換処理によって生成する。この幾何変換部６１によって生成された射影画像信号ＳＤPは、動き検出ブロック６２の動き判別画像生成処理部６３と積分画像生成処理部６４および中間画像情報生成ブロック７０の情報生成処理部７２に供給される。
【００８７】
動き判別画像生成処理部６３では、幾何変換部６１より供給された射影画像信号ＳＤPから、フレーム毎に正面画像と接する端部側から所定範囲の射影画像信号ＳＤPを抽出して順次貼り付けることにより動き判別画像信号ＳＭＤを生成する。この動き判別画像信号ＳＭＤは、動き量判別部６５に供給される。
【００８８】
積分画像生成処理部６４では、水平方向に位置をずらしながらｍsフレーム分の射影画像信号ＳＤPを足し込んで平均化することにより１つの積分画像信号を生成する。また、位置のずらし量をｎs個の異なる値としてｎs個の積分画像信号を生成する。この積分画像生成処理部６４で生成したｎs個の積分画像信号ＳＡは動き量判別部６５に供給される。
【００８９】
動き量判別部６５では、ｎs個の積分画像信号ＳＡと動き判別画像信号ＳＭＤとを比較して、動き判別画像信号ＳＭＤに基づく画像とマッチングする積分画像信号ＳＡに基づく画像を検出して、検出結果に基づき注目フレームの所定位置の動き量を判別する。ここで、積分画像では、位置のずらし量と画像の移動量が一致したときに、この被写体が積分画像に現れることとなる。また、ずらし量を異なるものとしてｎs個の積分画像が生成されていることから、ｎs個の移動量の異なる画像が積分画像信号ＳＡによって示されるものとなる。このため、ｎs個の積分画像信号ＳＡと動き判別画像信号ＳＭＤとの相関を判別して、この判別結果に基づき、動き判別画像を構成する被写体の移動量すなわち注目フレームの所定位置の動き量を判別できる。ここで、被写体の画像が遠景であるときには移動量が小さく、被写体の画像が近景であるときには移動量が大きいことから、この移動量は被写体の画像の奥行きを示すものである。このようにして、動き量判別部６５で射影画像の各位置の動き量を判別すると、判別結果を示す動き量情報ＭＦを生成して中間画像情報生成ブロック７０のレイヤ分類部７１と情報生成処理部７２に供給する。
【００９０】
レイヤ分類部７１では、供給された動き量情報ＭＦに基づき、注目フレームの所定位置が複数レイヤのいずれに属するかを示すレイヤ分類情報ＬＢを生成して情報生成処理部７２に供給する。
【００９１】
情報生成処理部７２では、各レイヤにおいて、フレーム毎に前後のフレームの移動量を用いて平均値を求めて、各レイヤの該当フレームの動き量ＭＹsとして蓄積部３０の中間画像情報蓄積領域３２に供給する。また、求めたレイヤ毎の動き量ＭＹsと、レイヤ分類部７１から供給されたレイヤ分類情報ＬＢに基づき、幾何変換部６１より供給された画像信号ＳＤPをレイヤ毎に分類して中間画像信号ＧＦsを生成する。すなわち、動き量分だけ画像信号を水平方向に読み出して該当するレイヤ毎に順次貼り付けることにより、図１４と同様なレイヤ毎の中間画像信号ＧＦsを生成する。
【００９２】
また、遠景レイヤや中景レイヤおよび近景レイヤの画像の動き方向を基準とすると、後退レイヤでは動き方向が逆方向となる。このため、動き量の絶対値分だけ正面画像と接する端部側から射影面画像信号を読み込み左右反転を行って貼り付けることで、後退レイヤの中間画像信号を生成できる。
【００９３】
さらに、作成したレイヤ毎の中間画像信号ＧＦsに対して、画像の無いアンカバード領域が生じないように補間処理を行って画像の無い部分の埋め込みを行う。この情報生成処理部７２で補間処理を行うことにより得られたレイヤ毎の中間画像信号ＧＹsは、各レイヤの動き量ＭＹsと関係付けられて中間画像情報として中間画像情報蓄積領域３２に記憶される。
【００９４】
このようにして生成された中間画像情報を中間画像情報蓄積領域３２から読み出して、上述の周辺画像信号生成ブロック４０と同様に処理することで、右側面画像の周辺画像信号を生成できる。
【００９５】
次に各ブロック等について詳細に説明する。図１９は幾何変換部６１での幾何変換処理を説明するための図である。この図１９において、位置Ｏは正面画像の撮影位置である。幾何変換部６１では、正面画像の位置ｐの画像を進行方向と平行な面上の位置ｑに射影して、正面画像から射影画像を生成する。ここで、射影面上の進行方向の位置ｘbと正面画像上の水平位置Ｘbの関係は、式（５）に示すものとなる。また、射影面上の高さ方向の位置ｙbと正面画像上の垂直位置Ｙbの関係は、式（６）に示すものとなる。
【００９６】
【数３】

【００９７】
なお、式（５）（６）において、「θg」は正面画像の水平画角を半分とした角度を示しており、正面画像を撮影したカメラの水平画角が明らかであるときには、この水平画角の半分の角度を「θg」とする。また、カメラの水平画角が明らかでないときには、使用したレンズの焦点距離に応じて予め設定された値を用いるものとする。「ｄ」は正面画像の水平方向の長さ（画枠の水平方向の長さ）を１／２にした値である。
【００９８】
このように式（５）（６）を用いることで、正面画像の各フレームの画像信号から左右の射影画像の画像信号をフレーム単位で生成することができる。また、入力画像が車載前進系の正面画像の場合、左右の射影画像は側方画像となっており、この射影画像の動き量は車速に比例する。また、この射影画像の動き量は奥行きに反比例するものであり、射影画像が遠景の画像であって奥行きが大きいときには動き量が小さくなると共に、射影画像が近景の画像であって奥行きが小さいときには動き量が大きくなる。
【００９９】
図２０は、動き判別画像生成処理部６３の動作を説明するための図である。動き判別画像生成処理部６３では、図２０Ａに示すように、フレーム毎に射影面画像の正面画像側端部（右側射影面画像では右側端部）から水平方向に所定画素数ｗａ（例えば４画素）の幅で画像信号を切り出すと共に、この切り出した画像信号を図２０Ｂに示すようにフレーム順に所定フレーム分だけ貼り付けることで動き判別画像の画像信号ＳＭＤを生成する。なお、動き判別画像では、被写体の動き量が水平方向にｗａ画素／フレームで、画像信号の切り出し画素数ｗａと等しいとき（図２０では家）、被写体の画像は不連続点を有しない連続した画像となる。画像信号の切り出しよりも動きの遅い被写体（図２０では山）の画像では、繰り返し切り出される部分が生じるため切り出された画像信号の繋ぎ部分で画像の不連続を生じる。また、画像信号の切り出しよりも動きの速い被写体の画像（図２０では木）では、動きが速いために切り出されない部分が生じて繋ぎ部分で画像の不連続を生じる。
【０１００】
積分画像生成処理部６４では、動き判別画像の各画素の動き量を判別するために用いる積分画像信号ＳＡを生成する。ここで図２１を用いて積分画像について説明する。例えば右方向に移動する被写体Ｊaと被写体Ｊaの２倍の速度で右方向に移動する被写体Ｊbを撮影すると、撮影画像は図２１Ａに示すものとなる。ここで、最初のフレームＦ1に対して次のフレームＦ2の画像を被写体Ｊaの移動方向とは逆方向で被写体Ｊaの画素量ｄｘだけシフトさせる。また、次のフレームＦ３の画像も同様に、フレームＦ2に対して被写体Ｊaの移動方向とは逆方向で被写体Ｊaの画素量ｄｘだけシフトさせる。このように、シフトした各フレームの画像を加算して平均化した画像を積分画像とすると、被写体Ｊaの位置が一定であると共に被写体Ｊbの位置は移動することから、被写体Ｊbの画像信号の信号レベルは被写体Ｊaに比べて小さいものとなり、図２１Ｃに示すように積分画像によって被写体Ｊaのみを抽出できる。
【０１０１】
また、図２１Ｄに示すように、各フレームの画像のずらし量を２倍（２ｄｘ）として積分画像を生成したときには、被写体Ｊbの位置が一定となると共に被写体Ｊaの位置が移動することから、図２１Ｅに示すように積分画像によって被写体Ｊbのみを抽出できる。
【０１０２】
このように、各フレームの画像を所定量だけ移動させて加算して平均化することにより積分画像を形成すると共に、この各フレームの画像のずらし量を変えて複数の積分画像を生成すれば、いずれの積分画像で被写体が正しく表示されたかを判別することで、１つの画像に動き量の異なる被写体が含まれていても、積分画像を用いて各被写体の動き量を判別できる。すなわち、各フレームの画像を構成する被写体の動き量が判別できるので、注目フレームの所定位置の動き量を判別できる。
【０１０３】
図２２は、積分画像生成処理部６４の構成を示す図である。幾何変換部６１から供給された射影画像信号ＳＤPは積分画像生成処理部６４の演算処理部６４１-1〜６４１-nsに供給される。
【０１０４】
演算処理部６４１-1では、予め設定された画素数ＳＨ-1だけ前フレームよりも水平方向に移動させた射影面画像信号を、ｍsフレーム分（例えば１０フレーム分）加算して加算信号を順次算出する。さらに、この加算信号を１／ｍ倍して平均化した信号を画素切り出し部６４２-1に供給する。
【０１０５】
画素切り出し部６４２-1では、供給された画像信号からフレーム毎に正面画像側端部（右側射影面画像では右側端部）より水平方向に所定画素数分ｗｂの画像信号を切り出して、この切り出した画像信号を順次貼り付けることにより積分画像信号ＳＡ-1として出力する。この画像切り出し部６４２-1から出力された積分画像信号ＳＡ-1は、予め設定された画素数ＳＨ-1の動き量の被写体を示した画像信号となる。
【０１０６】
演算処理部６４１-2〜６４１-nsも演算処理部６４１-1と同様に、予め設定された画素数ＳＨ-2〜ＳＨ-nsだけ前フレームよりも水平方向に移動させた射影面画像信号を、ｍsフレーム分加算して加算信号を順次算出して、この加算信号を１／ｍ倍して平均化したのち画素切り出し部６４２-2〜６４２-nsに供給する。さらに、画素切り出し部６４２-2〜６４２-nsでは、所定画素数分ｗｂの画像信号を切り出して順次貼り付けて積分画像信号として出力する。この画素切り出し部６４２-2〜６４２-nsから出力された積分画像信号ＳＡ-2〜ＳＡ-nsは、それぞれ予め設定された画素数ＳＨ-2〜ＳＨ-3の動き量の被写体を示した画像信号となる。
【０１０７】
動き判別画像生成処理部６３と積分画像生成処理部６４の画素切り出し部６４２で切り出す画素数を等しいものとすると（ｗａ＝ｗｂ）、動き判別画像と積分画像における被写体の位置の整合性をとることができる。
【０１０８】
図２３は積分画像生成処理部６４の動作を示す図である。画像ずらし量を画素数ＳＨ-rとしたときには、図２３Ａに示すようにフレーム毎に画像が画素数ＳＨ-r（＜ｗｂ）だけ移動されて足し込まれて、図２３Ｂに示すように山の画像が抽出される。画像ずらし量を画素数ＳＨ-rよりも多い画素数ＳＨ-s（＝ｗｂ）としたときには、図２３Ｃに示すようにフレーム毎に画像が画素数ＳＨ-sだけ移動されて足し込まれて、図２３Ｄに示すように山よりも動き量の大きい家の画像が抽出される。また、画像ずらし量を画素数ＳＨ-sよりも多い画素数ＳＨ-t（＞ｗｂ）としたときには、図２３Ｅに示すようにフレーム毎に画像が画素数ＳＨ-tだけ移動されて足し込まれて、図２３Ｆに示すように山よりも動き量の大きい家の画像が抽出される。
【０１０９】
ここで、動き判別画像生成部６３で動き判別画像を生成する際に最初に切り出しを行ったフレームＦaを基準として切り出しを開始すると共に、動き判別画像と等しい切り出し量（ｗａ＝ｗｂ）で同じフレーム数だけ画像の切り出しを行い積分画像を生成すると、積分画像と動き判別画像は水平方向の画像の長さが等しくなると共に、被写体の位相が積分画像と動き判別画像とで一致する。
【０１１０】
また、動き判別画像では、切り出し量と動き量が等しくないときには被写体が不連続が形状となるが、積分画像の場合も同様に、画素数ＳＨ-rが画素数ｗｂよりも少ないときには、画像の切り出しが重なりを持って行われることから図２３Ｇに示すように、動き量に応じた不連続な画像となる。また、画素数ＳＨ-ｓが画素数ｗｂと等しいときには、図２３Ｈのように動き量に応じた連続した画像となる。さらに、画素数ＳＨ-tが画素数ｗｂよりも多いときには、画像の切り出しで欠落部分が生じて図２３Ｊのように、動き量に応じた不連続な画像となる。このため、次に述べる動き量判別部６５で動き判別画像と積分画像のマッチングを行い、マッチングすると判別された積分画像の画像ずらし量である画素数ＳＨに基づいて、被写体の動き量を判別することができる。
【０１１１】
動き量判別部６５では、動き判別画像の各画素と積分画像との相関値を求めて、この相関値に基づき、動き判別画像の各画素がいずれの積分画像と等しいか判別することで、動き判別画像の各画素の動き量を判別する。
【０１１２】
相関値の算出では、画像Ａの画素と画像Ｂとの相関値を算出する際に、画像Ａの注目画素（相関値を算出する画素）を中心とした所定範囲を設定して、この所定範囲の画像信号を用いて注目画素の相関値を求める。例えば注目画素を中心としてｘ方向に±ｍvc画素、ｙ方向に±ｎvc画素の矩形状範囲（注目画素を中心とした例えば水平３１画素，垂直５画素の矩形状範囲）を設定し、式（７）に基づいて相関値ＶＣを算出する。
【０１１３】
【数４】

【０１１４】
なお、式（７）において、画像Ａの矩形状範囲内における各画素の信号レベルをＤＡｉ（ｉ＝１〜(ｍvc×ｎvc)）、画像Ｂの矩形状範囲内における各画素の信号レベルをＤＢｉ（ｉ＝１〜(ｍvc×ｎvc)）とし、矩形状範囲内における各画素の信号レベルの平均値をＤＡav，ＤＢavとする。
【０１１５】
例えば、式（７）に基づいて図２４Ａに示す動き判別画像における山の画像が図２４Ｂに示す積分画像とマッチングするものと判別されたときには、この山の画像を構成する各画素の動き量が画素数ＳＨ-rに対応した動き量であると判別できる。また、図２４Ａに示す動き判別画像におけるの家の画像が図２４Ｃに示す積分画像とマッチングするものと判別されたときには、この家の画像を構成する各画素の動き量が画素数ＳＨ-sに対応した動き量であることを判別できる。さらに、図２４Ａに示す動き判別画像おける木の画像が図２４Ｄに示す積分画像とマッチングするものと判別されたときには、この木の画像を構成する各画素の動き量が画素数ＳＨ-tに対応した動き量であることを判別できる。このようにして、動き量判別部６５で各画素に対して動き量を判別したときには、この判別結果を示す動き量情報ＭＦをレイヤ分類部７１と情報生成処理部７２に供給する。
【０１１６】
なお、図示せずも、積分画像生成処理部６４では、画像のずらし方向を図２３Ａ，図２３Ｃ，図２３Ｅとは逆方向も含むように設定（例えば画像のずらし量ＳＨを４８画素〜−２４画素に設定）すれば、木や家や山の画像に対して逆方向に移動する追越し車両等の画像を抽出できる。また、動き方向が遠景レイヤ等とは逆方向の画像の動き量を判別するため、図２５Ａに示すように、各フレームの画像から画像の切り出しを行い、この画像を左右反転してから貼り付けることにより図２５Ｂに示す動き判別画像を生成する。この場合、図２５Ｂに示す動き判別画像では、追越し車両のように動き方向が逆方向の画像が、遠景や中景および近景レイヤと同様な向きで正しく表示される。なお図２５Ｃは画像の左右反転を行わないで貼り付けた場合の動き判別画像である。このため、画像のずらし方向を逆方向とした積分画像の画像信号と、切り出した画像を反転させて貼り付けることにより生成された動き判別画像の画像信号を、基準となるフレームから順次比較することで、逆方向に移動する被写体の動き量を判別できる。
【０１１７】
レイヤ分類部７１では、動き量情報ＭＦに基づいて動き判別画像の各画素を奥行き方向にレイヤ化する。例えば、奥行きを予め遠景，中景，近景の３層と逆方向の動きを持つ層の４層にレイヤ化するものとして、各画素がいずれのレイヤに属するかレイヤ分類を行う。
【０１１８】
このレイヤ分類では、上述の閾値設定部２６４と同様にして閾値を設定すると共に、この閾値と各画素の動き量を比較して、各画素がいずれのレイヤに属するか判別してレイヤ分類情報ＬＢを生成する。この生成されたレイヤ分類情報ＬＢは情報生成処理部７２に供給される。
【０１１９】
図２６は、情報生成処理部７２の構成を示している。情報生成処理部７２の動き量平均値算出部７２１では、動き量判別部６５からの動き量情報ＭＦとレイヤ分類部７１から供給されたレイヤ分類情報ＬＢに基づき、それぞれのレイヤにおいて、注目フレーム毎に前後ｎzフレームの動き量の平均値を求め、そのレイヤの注目フレームの速度とする。この算出した各レイヤのフレーム毎の動き量ＭＹsは、画像読み込み部７２２に供給されると共に中間画像情報蓄積領域３２に記憶される。
【０１２０】
画像読み込み部７２２では、レイヤ分類情報ＬＢに基づき、射影画像の画像信号ＳＤPから動き量平均値算出部７２１で算出された動き量ＭＹsに応じてレイヤ毎に信号を抽出して、レイヤ毎の中間画像信号ＧＦsを生成する。この中間画像信号ＧＦsは、中間画像信号補間部７２３に供給される。このとき、遠景レイヤの動き量は小さく近景レイヤの動き量は大きいことから上述の図１４と同様に、遠景レイヤの中間画像は水平方向の長さが短く、近景レイヤの中間画像では長くなる。また、後退レイヤでは動きが逆方向であることから、動き量の絶対値分だけ側面画像から画像を読み込み左右反転して貼り付けていくことで、後退レイヤの中間画像を作成する。
【０１２１】
中間画像信号補間部７２３は、中間画像信号補間部２７３と同様に、動き量ＭＹsの異なる各レイヤの画像を遠景，中景，近景の順に貼り付けても、動き量の違いかかわらず画像のないアンカバード(Uncovered)領域が生ずることがないように中間画像信号ＧＦsの補正を行い、補正された信号を中間画像信号ＧＹsとして中間画像情報蓄積領域３２に記憶させる。
【０１２２】
また、追越し車両等の後退レイヤがあるときには、近景レイヤの手前のレイヤであることから補間処理を行わないものとする。また、右左折時に遠景に現れる建物等を示す水平移動レイヤが設けられているときには、遠景レイヤと同様に補間処理を行う。なお、後退レイヤや水平移動レイヤの被写体の判別は、右側射影画像と左側射影画像の動き判別画像の動き方向に基づいて図１０の場合と同様に動きパターンを判別して、判別された動きパターンから判別できる。例えば左側画像の画面上部と右側画像の画面上部での動きが異なる方向を示しているときには右折あるいは左折動作中であることを判別できる。また、左側画像と右側画像の画面上部の動きが同じ方向を示しており、これらとは逆方向の動きである被写体は、後退レイヤと判別できる。
【０１２３】
次に、各レイヤの中間画像情報を用いて周辺画像を生成する動作について説明する。この周辺画像の生成では、上述の周辺画像信号生成ブロック４０と同様にして、周辺画像信号を生成できる。すなわち、各レイヤの画像をその動き量分だけ読み出すと共に、遠景レイヤ、中景レイヤ、近景レイヤの順に貼り付ける。また水平移動レイヤや後退レイヤが設定されているときにも周辺画像信号生成ブロック４０と同様にして、周辺画像信号を生成できる。
【０１２４】
また、画像の貼り付けでは、例えば周辺画像として右側面画像を生成するときには、各レイヤの同一フレーム部分が、生成する右側面画像の左端で重なるように貼り付ける。また、最初に正面画像から変換した射影画像の右端と、生成した側面画像の左端が等しくなるようにする。その後、レイヤ毎に、レイヤ毎の動き量に基づいて画像を移動させて、この移動された画像に次の画像をレイヤ毎に貼り付けることで、図２７に示すように射影面画像と連続する側面画像を生成することができる。
【０１２５】
ところで、中間画像情報を用いてレイヤ毎の中間画像を貼り付けて生成された画像は、図２８に示すように射影面と等しい面上の画像である。一方周辺画像が表示されるスクリーン１０Rは、射影面とは傾きを持って設置されている。このため、各レイヤの画像を貼り付けてスクリーン１０Rに表示させたとき、画像が正しい形状で移動するように画像の変換処理を行う。
【０１２６】
ここで、図２８に示すように、正面画像の水平方向の長さの１／２を距離ｄ、正面画像の撮影位置である位置Ｏが、スクリーン１０Rの中心からの法線上となるようにスクリーン１０Rを傾けて設置した場合、スクリーン１０R上の位置（Ｘc，Ｙc）の画像信号は、式（８）（９）に基づいて算出された変換前の画像位置（ｘc，ｙc）の画像信号を抽出すれば、射影変換後の右側面画像の周辺画像信号ＳＤRを容易に生成できる。また、この画像信号ＳＤRを用いてスクリーン１０Rに右側面動画像を表示させることで、臨場感の高い右側面画像を表示できる。
【０１２７】
【数５】

【０１２８】
なお、周辺画像信号ＳＤLも周辺画像ＳＤRと同様にして生成できることは勿論である。
【０１２９】
さらに、上述の各処理ブロックで行われる処理はハードウェアだけでなくソフトウェアで実現するものとしても良い。この場合の構成を図２９に示す。
【０１３０】
コンピュータ９０は、図２９に示すようにＣＰＵ(Central Processing Unit)９０１を内蔵しており、このＣＰＵ９０１にはバス９２０を介してＲＯＭ９０２，ＲＡＭ９０３，ハード・ディスク・ドライブ９０４，入出力インタフェース９０５が接続されている。さらに、入出力インタフェース９０５には入力部９１１や記録媒体ドライブ９１２，通信部９１３，画像入出力部９１４が接続されている。
【０１３１】
キーボードやマウス等の操作入力手段あるいはマイク等の音声入力手段等を用いて構成された入力部９１１から、命令が入力されると、この命令が入出力インタフェース９０５を介してＣＰＵ９０１に供給される。
【０１３２】
ＣＰＵ９０１では、ＲＯＭ９０２やＲＡＭ９０３あるいはハード・ディスク・ドライブ９０４に記憶されているプログラムを実行して、供給された命令に応じた処理を行う。さらに、ＲＯＭ９０２やＲＡＭ９０３あるいはハード・ディスク・ドライブ９０４には、上述の各ブロックでの処理を実行するためのプログラムも記憶させて、側面画像の生成も行う。
【０１３３】
ここで、中間画像情報の生成や中間画像情報から周辺画像信号を生成して出力する画像処理プログラムは、情報記録媒体であるＲＯＭ９０２やハード・ディスク・ドライブ９０４に予め記憶しておくことができる。
【０１３４】
あるいは、磁気や光を利用したリムーバブル記録媒体や半導体素子等を用いて構成されたリムーバブルの情報記録伝送媒体、例えばフロッピー（Ｒ）ディスクやＣＤ−ＲＯＭ等の光ディスク、ＭＯディスク等の光磁気ディスク、テープカートリッジ、あるいは半導体メモリ等に画像生成プログラムを記録するものとして、これらのリムーバブル情報記録伝送媒体を記録媒体ドライブ９１２に装着して記録されている画像処理プログラムを読み出し、読み出したプログラムを入出力インタフェース９０５やバス９２０を介してハード・ディスク・ドライブ９０４等に記憶させることでインストールしても良い。
【０１３５】
さらに、通信部９１３を設けて、有線や無線の伝送路、例えばＬＡＮやインターネット等のネットワーク、あるいは衛星放送波や地上放送波等を利用して供給された画像処理プログラムを受信して、この受信した画像処理プログラムを、入出力インタフェース９０５及びバス９２０を介してハード・ディスク・ドライブ９０４等にインストールすることもできる。
【０１３６】
ここで、画像処理プログラムが実行されて中間画像情報の生成が行われたときには、ハード・ディスク・ドライブ９０４あるいは記録媒体ドライブ９１２に装着された記録媒体に記録されている正面画像の画像信号ＳＤCが読み出されて、上述のような処理が行われて中間画像情報が、例えば画像信号ＳＤCと共にハード・ディスク・ドライブ９０４や記録媒体ドライブ９１２に装着された記録媒体に記録される。また、周辺画像の画像信号の出力が要求されたときには、中間画像情報が読み出されて左右側面画像の画像信号ＳＤR，ＳＤLが生成されると共に、この画像信号ＳＤR，ＳＤLが正面画像の画像信号ＳＤCとタイミングを合わせて出力される。このため、正面と左右のスクリーンを用いて、広画角の連続した画像を表示することができる。
【０１３７】
このように、上述の実施の形態によれば、正面画像の動きを判別して、正面画像から判別された動きに基づいて異方向の周辺画像を生成できる。このため、ビデオカメラ等で撮影された正面動画像から同時系列で異方向の動画像を生成できるので、臨場感が高く広画角である多画面の画像提示ができる。
【０１３８】
また、１つの入力画像から異方向の画像を生成できるので、複数のビデオカメラや広角レンズを用いたカメラ等を用いる必要がなく、撮影を容易に行うことができる。
【０１３９】
さらに、実写画像を用いて異方向の画像を生成するものであることから、コンピュータグラフィックスによる３次元仮想空間の画像よりもリアルで臨場感の高い画像を表示させることができると共に、奥行きを２次元平面の階層として表現するため、３次元の演算処理が不要となり、信号処理も容易である。
【０１４０】
また、入力画像信号として蓄積されている信号を用いることで、時間の経過と共に正面画像内に入り込む画像を、正面画像で表示される前に周辺画像で表示することができる。このようにリアルタイムの画像信号を用いたときには表示することができない画像を周辺画像で表示することができるので、より現実的で臨場感の高い広画角の画像表示を行うことができる。また、既に存在している莫大な画像ソースを入力画像信号として用いることで、これらの画像を臨場感が高く広画角で楽しむこともできる。
【０１４１】
なお、上述の実施の形態では、正面画像の側端部に複数の検出領域を設けて検出領域毎に動きベクトルを求めたり、射影画像から動き判別画像を生成して積分画像と比較することにより、注目フレームの所定位置の動きを判別して中間画像情報を生成し、この中間画像情報を利用して周辺画像信号を生成したが、動き検出は、注目フレームの所定位置の動きを判別できるものであれば良く、上述の方法は例示的なものであって、限定的なものではないことは勿論である。
【０１４２】
【発明の効果】
この発明によれば、複数フレームの入力画像信号を用いて、入力画像信号における注目フレームの所定位置の動きが検出されて、この検出された動きと複数フレームの入力画像信号に基づいて中間画像情報が生成される。さらに、この中間画像情報に基づいて、注目フレームに存在しない画角の周辺画像信号が生成される。このため、実写した入力画像信号を用いて注目フレームに存在しない画角の周辺画像信号を生成することにより、この周辺画像をリアルで臨場感の高い画像として表示させることができる。また、複数のビデオカメラや特殊レンズを用いたビデオカメラを用いることなく、１つの入力画像から異方向の周辺画像を生成して広画角の画像提示を容易に行うことができる。さらに、既に存在している莫大な画像ソースを利用して臨場感が高く広画角の画像提示を行うこともできる。
【０１４３】
また、所定位置の動きに基づいたレイヤ分類を行って、所定位置を複数レイヤのいずれかに分類するものとし、分類結果と入力画像信号と所定位置の動きに基づき、レイヤ毎の動き量とレイヤ毎の画像信号が中間画像情報とされる。このため、２次元平面上での動きに基づいて奥行き方向の階層化を行い、各階層の画像信号を貼り付けることで、３次元処理を行うことなく３次元での画像提示を行うことができる。
【図面の簡単な説明】
【図１】表示システムの構成を示す図である。
【図２】画像投影状態を示す図である。
【図３】画像処理装置の構成を示す図である。
【図４】各検出領域の動きベクトルを示す図である。
【図５】動きベクトルの判別方法を説明するための図である。
【図６】ズーム時の領域サイズの変化を説明するための図である。
【図７】画像基準位置の判別方法を示す図である。
【図８】動き検出処理部の構成を示す図である。
【図９】レイヤ分類部の構成を示す図である。
【図１０】ベクトル方向と動きパターンの関係を説明するための図である。
【図１１】閾値の設定方法を説明するための図である。
【図１２】レイヤ分類情報の生成動作を説明するための図である。
【図１３】情報生成処理部の構成を示す図である。
【図１４】レイヤ毎の中間画像を示す図である。
【図１５】補間処理後のレイヤ毎の中間画像を示す図である。
【図１６】周辺画像信号生成ブロックの構成を示す図である。
【図１７】画像変換処理を説明するための図である。
【図１８】画像信号処理装置の他の構成を示す図である。
【図１９】幾何変換処理を説明するための図である。
【図２０】動き判別画像生成部の動作を説明するための図である。
【図２１】積分画像の生成動作を説明するための図である。
【図２２】積分画像生成部の構成を示す図である。
【図２３】積分画像生成部の動作を説明するための図である。
【図２４】動き量判別部の動作を説明するための図である。
【図２５】後退レイヤの動き判別画像の生成動作を説明するための図である。
【図２６】情報生成処理部の構成を示す図である。
【図２７】射影面画像と生成した側面画像を示す図である。
【図２８】スクリーンで正しく画像表示を行うための画像変換処理を説明するための図である。
【図２９】コンピュータを用いたときの構成を示す図である。
【符号の説明】
１０L，１０C，１０R・・・スクリーン、１２L，１２C，１２R・・・プロジェクタ、１５，５５・・・画像処理装置、２０，６２・・・動き検出ブロック、２１・・・フレーム遅延処理部、２２・・・動き検出処理部、２５，７０・・・中間画像情報生成ブロック、２６，７１・・・レイヤ分類部、２７，７２・・・情報生成処理部、３０・・・蓄積部、３１・・・正面画像信号蓄積領域、３２・・・中間画像情報蓄積領域、４０・・・周辺画像信号生成ブロック、４１・・・画像生成制御部、４２・・・レイヤ画像生成処理部、４３・・・射影変換部、４４・・・画像信号生成部、６１・・・幾何変換部、６３・・・動き判別画像生成処理部、６４・・・積分画像生成処理部、６５・・・動き量判別部、９０・・・コンピュータ、２２１・・・サイズ変換処理部、２２２・・・誤差和算出部、２２３・・・比較処理部、２２４・・・データ保持部、２２５・・・探索制御部、２６１・・・パターン判定部、２６２・・・レイヤ作成判定部、２６３・・・分類処理部、２６４・・・閾値設定部、２６５・・・分類補正部、２７１，７２１・・・動き量平均値算出部、２７２，７２２・・・画像読み込み部、２７３，７２３・・・中間画像信号補間部、４２１-1〜４２１-5・・・貼り付け部、４２２-1〜４２２-5・・・１フレーム遅延部、４２３-1〜４２３-5・・・画像シフト部、４３１〜４３４・・・変換処理部、４４１・・・遠景貼り付け部、４４２・・・中景貼り付け部、４４３・・・近景貼り付け部、４４４・・・縮小貼り付け部、６４１-1〜６４１-ns・・・演算処理部、６４２-1〜６４２-ns・・・画素切り出し部、９０１・・・ＣＰＵ、９０２・・・ＲＯＭ、９０３・・・ＲＡＭ、９０４・・・ハード・ディスク・ドライブ、９０５・・・入出力インタフェース、９１１・・・入力部、９１２・・・記録媒体ドライブ、９１３・・・通信部、９１４・・・画像入出力部、９２０・・・バス[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and an image. processing The present invention relates to a method, an image processing program, and an information recording medium. Specifically, it is assumed that a motion at a predetermined position of the target frame is detected using a plurality of frames of input image signals, and intermediate image information is generated based on the detected motion and the plurality of frames of input image signals. Based on the intermediate image information, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated.
[0002]
[Prior art]
In recent years, a multi-screen display system, a curved display, a wide-angle display, a head-mounted display, and the like have been put into practical use in order to display an image with enhanced realism.
[0003]
Here, in order to display an image with a higher sense of presence using the display as described above, for example, a three-dimensional virtual space is constructed by computer graphics. In addition, a large number of video cameras are used or a wide-angle lens is attached to the video camera to capture a wide range of space, and the captured image is converted into a flat or curved surface such as a multi-surface display or a head-mounted display and displayed. Yes.
[0004]
[Problems to be solved by the invention]
By the way, when displaying an image with a high sense of reality using a multi-screen display, a wide-angle display, or the like, a high-speed computer device is used to construct a three-dimensional virtual space with computer graphics. Since it is necessary to perform arithmetic processing, a great deal of cost and time are required, and an image with less realism and realism than a real image is obtained.
[0005]
In addition, when using live-action images, it is necessary to shoot the entire range without gaps in order to present a wide range of space, such as a large number of video cameras and video cameras with special lenses. A capture device is required, and a large amount of cost is required.
[0006]
Furthermore, when an image source shot with a general video camera already exists, an image with a wide angle of view cannot be presented unless the same scene is retaken with a plurality of video cameras.
[0007]
Therefore, in the present invention, an image processing apparatus and an image that can easily present wide-angle images without using a plurality of video cameras or video cameras using special lenses. processing A method, an image processing program, and an information recording medium are provided.
[0008]
[Means for Solving the Problems]
The image processing apparatus according to the present invention uses a plurality of frames of input image signals to detect a motion at a predetermined position of a frame of interest in the input image signal, the motion detected by the motion detector, Intermediate image information generating means for generating intermediate image information based on input image signals of a plurality of frames, and a peripheral image having an angle of view that does not exist in the frame of interest based on the intermediate image information generated by the intermediate image information generating means And peripheral image signal generating means for generating a signal.
[0009]
The image processing method is to detect a motion at a predetermined position of a target frame in the input image signal using a plurality of frames of input image signals, and based on the detected motion and the input images of the plurality of frames. Image information is generated, and based on the generated intermediate image information, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated.
[0010]
Further, the image processing program uses a motion detection procedure for detecting a motion at a predetermined position of a frame of interest in the input image signal using a plurality of frames of the input image signal to a computer that processes the image signal, and the motion detection procedure. An intermediate image information generation procedure for generating intermediate image information based on the detected motion and the input image signals of the plurality of frames, and an existing image in the frame of interest based on the intermediate image information generated by the intermediate image information generation procedure A peripheral image signal generation procedure for generating a peripheral image signal having a non-viewing angle is executed.
[0011]
Furthermore, the information recording medium is a computer that uses a plurality of frames of input image signals to detect a motion at a predetermined position of a frame of interest in the input image signal, and a motion detected by the motion detection procedure. An intermediate image information generation procedure for generating intermediate image information based on the input image signals of the plurality of frames, and a surrounding of an angle of view that does not exist in the frame of interest based on the intermediate image information generated by the intermediate image information generation procedure A program for executing a peripheral image signal generation procedure for generating an image signal is recorded.
[0012]
In the present invention, the stored input image signal is used to detect the correlation between the image at the predetermined position of the frame of interest and the image of the previous frame or the subsequent frame with respect to the frame of interest, and From the position of the image having the highest correlation, the movement direction and the amount of movement of the image at the predetermined position are detected. A motion pattern is detected based on the detected motion direction, and a plurality of layers indicating the depth are set according to the detection result. In addition, a threshold value used for layer classification of the predetermined position based on the detected motion amount is set. Layer classification is performed using this threshold value, and a motion amount for each layer and an image signal for each layer are generated as intermediate image information based on a layer classification result at a predetermined position, an input image signal, and a motion at a predetermined position.
[0013]
In addition, an image signal of a projected image obtained by projecting an image based on the input image signal onto a predetermined plane is generated, and an image signal of a predetermined area of the projected image is extracted for each frame using the image signal of the projected image. Image signals are generated. Further, an average value image signal for a predetermined frame is sequentially generated using an image signal obtained by shifting the position of the projected image for each frame, and an image signal is extracted for each frame from the average value image signal to obtain an image of the integral image. A signal is generated. A plurality of image signals of the integral image are generated by changing the shift amount of the projected image. Further, the correlation between the image signal of the motion determination image and the image signals of the plurality of integral images is detected, and the predetermined amount of the motion determination image is determined from the shift amount when the image signal of the integral image having the highest correlation is generated. The amount of movement of the position is determined. Layer classification at a predetermined position is performed based on the amount of motion, and the amount of motion and the image signal for each layer are used as intermediate image information.
[0014]
When generating an image signal for each layer, if there is an area where the image disappears when the images based on the image signal for each layer are pasted together in a predetermined order, the image signal of the peripheral area of the area where the image disappears is used. Interpolation processing is performed.
[0015]
When outputting the image signal of the peripheral image, the intermediate image information is read out, and the image signal is read out for each layer based on the motion amount for each layer and pasted in a predetermined layer order. The image signal used for pasting is moved according to the amount of movement for each layer, and a new image signal is pasted for each layer on the moved signal. The image signal pasted in the predetermined layer order is output in correspondence with the input image signal as an image signal indicating a peripheral image having a field angle that does not exist in the frame of interest.
[0016]
In addition, when there is a region without an image in the image signal in which the image signal for each layer is pasted in a predetermined layer order, interpolation processing is performed using the image signal of the peripheral region of the region without the image, and the image signal An image signal of an area without the image is generated. Further, projective transformation is applied to the image signal pasted for each layer according to the positional relationship between the image display surface of the image based on the input image signal and the image display surface of the peripheral image, or pasted for each layer. Projection conversion is performed on an image signal obtained by pasting the attached image signal in a predetermined layer order.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows the overall configuration of a display system using an image processing apparatus according to the present invention. In this display system, for example, three screens are arranged on the front and both sides of the user, and images are projected from projectors 12L, 12C, and 12R corresponding to the screens 10L, 10C, and 10R. The projectors 12L, 12C, and 12R are connected to the image processing device 15.
[0018]
The image processing device 15 stores a captured image signal, for example, an image signal SDC of a front image captured by an in-vehicle camera or the like. The image processing apparatus 15 supplies the stored image signal SDC to the projector 12C, and displays a front image based on the image signal SDC on the screen 10C located in front as shown in FIG. Further, in the image processing device 15, an image signal indicating a peripheral image having an angle of view that is not included in the field-of-view range of the in-vehicle camera, for example, left and right side images continuous with the front image, from the stored image signal SDC of the front image. Intermediate image information for generating peripheral image signals SDL and SDR, which are image signals indicating the above, is generated and stored. When a front image is displayed on the screen 10C, peripheral image signals SDL and SDR of a peripheral image continuous with the front image are generated using the accumulated intermediate image information, and the peripheral image signal SDL is supplied to the projector 12L. At the same time, the peripheral image signal SDR is supplied to the projector 12R. Therefore, a left side image having continuity with the front image is displayed on the screen 10L located on the left side, and a right side image having continuity with the front image is displayed on the screen 10R located on the right side. Corner image presentation can be performed.
[0019]
FIG. 3 shows an outline of the configuration of the image processing apparatus 15. The image signal SDC accumulated in the front image signal accumulation area 31 of the accumulation unit 30 is converted into a 1-frame delay processing unit 21 and a motion detection processing unit 22 of the motion detection block 20 and an information generation processing unit 27 of the intermediate image information generation block 25. To be supplied.
[0020]
The 1-frame delay processing unit 21 delays the image signal SDC by 1 frame and supplies it to the motion detection processing unit 22 as the image signal SDCa. The motion detection processing unit 22 sets a plurality of detection areas divided as indicated by broken lines in FIG. 4 on the side edge side of the front image, and the image signal SDC of the frame of interest in this detection area and the previous frame. Comparison with the image signal SDCa is performed for each detection region, and a motion vector MV indicating the motion of the image is determined for each detection region and supplied to the layer classification unit 26 of the intermediate image information generation block 25.
[0021]
When the layer classification unit 26 determines the motion pattern of the front image from the motion vector MV of each detection area and generates the intermediate image information, for example, a layer of a distant view image obtained by photographing a distant subject and a close subject are photographed. The layer setting is performed by determining how to provide a layer of the foreground image, a layer of the middle scene image positioned between the distant view image and the foreground image, and a layer different from these layers. For example, when a front image is taken with an in-vehicle camera, when the vehicle goes straight in one direction, the front image is sequentially zoomed in. In addition, when the vehicle is moving backward, the front image is sequentially zoomed out. Furthermore, when the overtaking vehicle is photographed, the overtaking vehicle is displayed as a zoomed-out image in the zoom-in image. When a right turn or a left turn is made, for example, the upper side of the front image is moved in the horizontal direction and the lower side becomes a zoom-in image. For this reason, it is assumed that the motion pattern of the front image is determined from the motion vector MV of each detection region, and layer setting is performed based on the determined motion pattern. For example, when it is determined that the motion pattern goes straight in one direction, a layer of each image of the far view, middle view, and foreground is generated, and when it is determined that the motion pattern is a right turn or left turn, the distant view, middle view, A layer including an image moved in the horizontal direction is created as well as a layer of each image in the foreground. In addition, when the overtaking vehicle has a certain motion pattern, the layer setting is performed so as to create not only a layer of each image of a distant view, a middle view, and a close view but also a layer including a zoom-out image.
[0022]
Further, the layer classification unit 26 performs layer classification on which layer each detection region set on the side edge side of the front image belongs based on the motion vector MV supplied from the motion detection processing unit 22. In this layer classification, layer classification using a layer set according to the motion pattern is performed. For example, when a distant view, a middle view, and a foreground are divided into three layers, and a layer that includes an image that is moved in the horizontal direction or a layer that includes a zoom-out image is generated, the layer classification including these layers is also performed. I do. By this layer classification, layer classification information LB indicating which detection region belongs to which layer is generated, and the generated layer classification information LB is supplied to the information generation processing unit 27.
[0023]
The information generation processing unit 27 divides each detection region set in the front image based on the layer classification information LB from the layer classification unit 26 into layers, and uses the image signal of this detection region for each layer in the frame order. Each intermediate image signal is generated. Further, an average value of the motion amount is calculated for each layer from the motion amount of the motion vector MV of the detection area divided into layers. The intermediate image signal GYv generated by the information generation processing unit 27 and the calculated motion amount (average value) MYv for each layer are stored in the intermediate image information storage region 32 of the storage unit 30 as intermediate image information.
[0024]
When images are displayed on the screens 10C, 10R, and 10L, the stored image signal SDC is read, and an image based on the image signal SDC is displayed on the screen 10C. Further, the intermediate image information stored in the intermediate image information storage area 32 is read out by the read control signal RC from the peripheral image signal generation block 40 connected to the storage unit 30, and according to the motion amount MYv for each layer. The intermediate image signal GYv of each layer is sequentially used as much as the past, and the images of each layer are pasted in the order of distant view, middle view, and close view. Further, when there are provided layers that do not belong to the three layers of distant view, middle view, and close view, image pasting processing of these layers is also performed, and peripheral image signals SDL and SDR are generated. By supplying the peripheral image signal SDL to the projector 12L at a timing corresponding to the image signal SDC of the front image, the left side image can be displayed on the screen 10L continuously with the front image, and the peripheral image signal SDR can be displayed on the front image. By supplying the projector 12R with the timing corresponding to the image signal SDC, the right side image can be displayed on the screen 10R in succession to the front image.
[0025]
Next, each part which comprises the motion detection block 20 is demonstrated in detail. In addition, in order to simplify description, in the following description, only the right side surface will be described, and description of the left side surface will be omitted.
[0026]
The motion detection block 20 determines the motion vector MV for each detection region as described above. Here, when the front image has a moving center of the image, that is, when the front image is captured by the vehicle-mounted camera, the image at time T shown in FIG. 5A is, for example, time T after one frame time has elapsed. As shown in FIG. 5B, it becomes substantially equal to a zoom-in operation image centered on a position CP where the image appears to appear (hereinafter referred to as “image reference position”).
[0027]
Here, as shown in FIG. 5C, the enlargement ratio Z is set to reduce the detection area of the target frame to (1 / Z), and the position of the reduced detection area is moved, and the image one frame before is displayed. Calculate the error sum with the signal. Further, the enlargement ratio Z is changed, and the error sum is calculated while moving the position in the same manner. In this way, by detecting the position where the error sum is the minimum value, the motion vector MV of each detection region can be determined for each frame of interest. Further, the enlargement ratio Z when the error sum becomes the minimum value is set as the motion amount.
[0028]
By the way, when the reduction process is performed, a pixel in which the coordinate value of the pixel in the region is not an integer value is generated. On the other hand, in the image one frame before which the reduction process is not performed, the coordinate value of the pixel in the region is an integer value. Therefore, linear interpolation is performed on the reduced image, and the signal level at the position where the coordinate value becomes an integer value is calculated. For example, as shown in FIG. 6, when the image size becomes Kb × Kb pixel size by performing the reduction process to (1 / Z) on the image of Ka × Ka pixel, linear interpolation is performed. The signal level of the image with the number of pixels “Kb × Kb” is calculated from the signal of Ka × Ka pixels. By calculating the sum of errors between the calculated signal level and the signal level one frame before the position corresponding to the reduced image, the amount of motion can be accurately determined.
[0029]
When the image reference position CP is not clear, as shown in FIG. 7, when the center of the detection area ARa is changed from the position Pa1 to the position Pa2, the error sum becomes a minimum value, and the center of the detection area ARb is changed from the position Pb1 to the position Pb2. If the error sum becomes a minimum value when the image is detected, the image reference position CP can be detected by detecting a point where the direction of the motion vector MV of the detection area ARa and the direction of the motion vector of the detection area ARb intersect.
[0030]
Further, when turning right or turning left, for example, the upper side image of the front image moves horizontally. For this reason, if the minimum value of the error sum is not detected even when the zoom-in operation is performed with the image reference position CP as the center, the minimum value of the error sum is determined by moving the image in the detection region in the horizontal direction. At this time, the amount of movement of the detection region until the error sum becomes the minimum value can be set as the amount of motion of the motion vector MV.
[0031]
Next, since the image appears to be sucked into the image reference position CP at the time of backward movement, the image at the time of backward movement is substantially equal to the zoom-out operation image centered on the image reference position CP. For this reason, the enlargement ratio Z is set to “1” or less. That is, when moving backward, the movement of the image is opposite to that when moving forward, so a plurality of detection areas are set on the side edge side of the previous frame, and each detection area is reduced to (1 / Z). The error sum with the image of the frame of interest is calculated while moving the position of the region. Alternatively, each detection area of the target frame is reduced to (1 / Z), and the error sum with the image after one frame is calculated while moving the position of the reduced area. Further, the enlargement ratio Z is changed, and the error sum is calculated while moving the position in the same manner. By detecting the position where the error sum becomes the minimum value in this way, the motion vector MV during backward movement can also be determined.
[0032]
When the motion vector MV is discriminated as described above, the amount of motion of the motion vector MV is small because the image of the distant view has little motion, and the amount of motion of the motion vector MV is large because the image of the foreground has much motion. Become.
[0033]
FIG. 8 shows the configuration of the motion detection processing unit 22. The image signal SDC is supplied to the size conversion processing unit 221, and the image signal SDCa supplied from the 1-frame delay processing unit 21 is the error sum calculation unit 222. To be supplied. The size conversion processing unit 221 sets the detection area by dividing the side edge portion of the front image into a plurality of detection areas, for example, in a unit of 16 × 16 pixel size. Further, the enlargement ratio Z is set in the size conversion processing unit 221 from a search control unit 225 described later, and is detected with the image signal FEz obtained by multiplying the image of the detection region by (1 / Z) and the image reference position CP. The coordinate value Pz converted by multiplying the image of the region by (1 / Z) is supplied to the error sum calculation unit 222. The image signal FEz is a signal when the coordinate value is converted into an integer value by interpolation processing as shown in FIG. 6 described above, and the coordinate value Pz is the coordinate when the coordinate value is converted into an integer value by interpolation processing. The value, that is, the pixel position “Kb × Kb” in FIG.
[0034]
The error sum calculation unit 222 selects a signal at the position indicated by the coordinate value Pz from the size conversion processing unit 221 from the image signal SDCa, and calculates the error sum between the selected signal and the image signal FEz. The comparison processing unit 223 is notified.
[0035]
The comparison processing unit 223 compares the error sum minimum value with the error sum calculated by the error sum calculation unit 222. Here, when the error sum minimum value is not set, the error sum calculated first is set as the error sum minimum value. When the calculated error sum is smaller than the error sum minimum value, this error sum is set as a new error sum minimum value, and the data holding unit 224 is notified that the error sum minimum value has been updated. In addition, the search controller 225 is notified by the signal ES that the comparison between the minimum error sum and the error sum calculated by the error sum calculator 222 has been completed. The minimum error sum value may be set in advance to a value larger than the calculated error sum.
[0036]
The data holding unit 224 is notified of the enlargement factor Z from the search control unit 225, and stores the notified enlargement factor Z when notified that the minimum error sum has been updated. When the enlargement ratio is already stored, the stored enlargement ratio is updated with the notified enlargement ratio Z. Further, when the search control unit 225 notifies the completion of the enlargement ratio changing process by the signal ET, the motion vector MV with the stored enlargement ratio as the motion amount and the image reference position direction as the vector direction is classified into layers. To the unit 26.
[0037]
In the search control unit 225, the lower limit value of the enlargement ratio is set to “1” and the upper limit value is also set in advance. First, the size conversion processing unit 221 and the data holding unit 224 are set with the lower limit value as the enlargement ratio Z. Notify Thereafter, each time the comparison processing unit 223 notifies that the comparison between the minimum error sum value and the error sum calculated by the error sum calculation unit 222 has been completed, the enlargement ratio Z is sequentially increased to increase the size conversion processing unit 221. To the data holding unit 224. Thereafter, when the enlargement ratio Z reaches the upper limit value, the data holding unit 224 is notified of the completion of the enlargement ratio change process.
[0038]
Further, when the minimum error sum obtained by the comparison processing unit 223 is not small, that is, when an image equal to the side edge portion of the front image cannot be detected, the search control unit 225 sets the enlargement ratio Z to “1” and performs size conversion. The processing unit 221 is notified, and the error area calculation unit 222 is supplied with the image signal FEz of the detection area. Further, the control signal RP is supplied to the error sum calculation unit 222, and a signal at a position obtained by moving the detection region of the image signal FEz by a predetermined amount in the horizontal direction is selected from the image signal SDCa. Thereafter, by moving the position selected from the image signal SDCa by the control signal RP in the horizontal direction and determining the minimum error sum, the motion vector MV of the image to be moved in the horizontal direction can also be obtained. Further, although not shown, a signal after one frame is supplied to the error sum calculation unit 222, or a detection region is set in the image one frame before and a signal of the image of the frame of interest is supplied to the error sum calculation unit 222. Thus, it is also possible to determine the amount of motion of the image that is sucked into the image reference position CP.
[0039]
In this way, by moving the search direction in the direction of the image reference position CP or in the horizontal direction and detecting the image position of the other frame that minimizes the error sum with the image of the detection area, the image of the detection area is turned to the right. The motion vector MV can be obtained correctly even for an image of a left turn, an image of a car or a person crossing the front, or an image of an overtaking vehicle.
[0040]
The layer classification unit 26 determines what kind of motion pattern the front image is based on the motion vector MV of each detection region, sets a layer based on the determined motion pattern, and determines which region is which The layer classification information LB indicating whether the layer belongs to the information generation processing unit 27 is supplied.
[0041]
FIG. 9 shows the configuration of the layer classification unit 26. The motion pattern determination unit 261 of the layer classification unit 26 accumulates the motion vector MV of each detection area supplied from the motion detection processing unit 22 in units of frames, and determines a motion pattern based on the accumulated motion vector MV.
[0042]
Here, the direction of the motion vector MV of each detection area is set as the radial direction from the image reference position CP, and it is determined whether or not all the images are in the zoom-in operation direction where the image boils from the image reference position CP. Is in a zoom-in operation direction that is boiling from the image reference position CP, it is determined that the operation is a straight-ahead operation. For example, when the vector direction of the motion vector MV is a radial direction from the image reference position CP as indicated by an arrow in FIG.
[0043]
Next, when it is not determined to be a straight-ahead operation, it is determined whether the direction of the motion vector MV of each detection region is opposite to the radial direction, and whether or not all the images are in a zoom-out operation direction that is sucked into the image reference position CP. When all the images are in the zoom-out operation direction in which the images are sucked into the image reference position CP, it is determined that the operation is the backward operation. For example, when the direction of the image reference position CP is set as indicated by the arrow in FIG. 10B and it is detected that the zoom-out operation direction in which all images are sucked into the image reference position CP is detected, the backward movement operation is determined.
[0044]
Further, when it is not determined that the vehicle is moving straight forward or backward, it is determined whether only a part is in the zoom-out operation direction, and when only part is in the zoom-out operation direction, it is determined that there is an overtaking vehicle. For example, as shown in FIG. 10C, it is determined that there is an overtaking vehicle when it is detected that the movements of the detection areas at the left and right end portions are in the zoom-in operation direction and only a part is in the zoom-out operation direction. Further, when it is not determined that there is a straight traveling operation, a reversing operation, or an overtaking vehicle, in the detection area on the upper side of the front image, when the vector direction of the motion vector MV is the horizontal direction as shown in FIG. It is determined as a right turn or left turn depending on the direction. Further, when the vector direction of the motion vector MV is the horizontal direction as shown in FIG. 10E in a part of the detection area, it is determined that the crossing operation is performed. The motion pattern MP determined in this way is notified to the layer creation determination unit 262.
[0045]
The layer creation determination unit 262 determines whether or not the determined motion pattern continues for a predetermined number of frames or more based on the motion pattern MP determined by the motion pattern determination unit 261. Here, the layer pattern information LP corresponding to the motion pattern determined when the motion pattern continues for a predetermined number of frames or more is generated.
[0046]
Here, when it is determined that the motion pattern in which the entire screen is enlarged as in the forward motion is continuously more than a predetermined number of frames, for example, layer pattern information LP for instructing creation of a far view, middle view, and foreground layer Is generated and notified to the classification processing unit 263. In addition, when it is determined that the movement pattern in which the upper part moves in the horizontal direction, such as a right turn or a left turn, is continuously determined for a predetermined number of frames or more, it moves in the horizontal direction as well as the far, middle, and foreground layers. Layer pattern information LP instructing creation of a layer including the image to be generated is generated and notified to the classification processing unit 263. In addition, when it is determined that the motion pattern includes an image that shrinks with time, such as when the vehicle is overtaking or when moving backwards, the layers of the distant view, the middle view, and the foreground are determined continuously. In addition, layer pattern information LP for instructing creation of a backward layer including an image to be reduced is generated. In addition, when it is determined that a part of the moving pattern moves in the horizontal direction, such as a crossing object, for a predetermined number of frames or more, the creation of a layer including an image moving in the horizontal direction is instructed. Layer pattern information LP is generated. As described above, when the determined motion pattern continues for a predetermined number of frames or more, the layer pattern information LP corresponding to the determined motion pattern MP is generated, so that a frame in which the motion pattern is erroneously determined is generated. However, the layer pattern information LP corresponding to the correct motion pattern can be generated.
[0047]
The threshold value setting unit 264 uses the motion vector MV in a predetermined time range (for example, 30 frames before and after the frame of interest) and uses the average value of the motion amount of the motion vector MV whose vector direction is the radial direction from the image reference position CP. Classification processing is performed by obtaining Vavg, the maximum value Vmax, and the minimum value Vmin, and setting a threshold Th for classifying the layer indicated by the layer pattern information LP based on the average value Vavg, the maximum value Vmax, and the minimum value Vmin. To the unit 263.
[0048]
For example, when the layer pattern information LP indicates that the layer is divided into a distant view, a middle view, and a foreground layer, a threshold value Th1 indicating the division position between the distant view layer and the middle view layer is calculated based on Expression (1). Further, a threshold Th2 indicating the division position of the middle scene layer and the foreground layer is calculated based on the formula (2).
[0049]
[Expression 1]

[0050]
In setting the threshold value, as shown in FIG. 11, a motion amount histogram may be obtained, and the threshold values Th1 and Th2 may be obtained using the minimum value of the histogram. As described above, since the threshold Th is dynamically changed according to the motion amount distribution, the image is not classified into only one layer, and the layer classification is favorably performed according to the motion amount distribution. be able to.
[0051]
Based on the threshold Th from the threshold setting unit 264 and the amount of motion of the motion vector MV, the classification processing unit 263 determines which of the layers instructed to be created by the layer pattern information LP, each detection region of each frame. Layer classification. Further, the detection area moving in the horizontal direction and the detection area in the zoom-out operation direction are assigned to the corresponding layers. When the classification processing unit 263 completes the layer classification of each detection region of the frame of interest, the layer classification information LA indicating the result of the layer classification is supplied to the classification correction unit 265.
[0052]
Here, the classification processing unit 263 uses the motion vector MV from the motion detection processing unit 22 to create three layers of a distant view, a middle view, and a foreground for the previous m frames and the subsequent steps for each detection region. The average value of the motion amount for n frames is calculated. For example, in the detection area provided at the right end of the front image, when the amount of motion of the motion vector MV changes as time passes as shown in FIG. 12A (the numbers in the figure indicate the amount of motion). , With respect to the detection area AR1 in the Fp frame, the motion amount of the detection area AR1 in the Fp frame, the motion amount of the detection area AR1 in the Fp-1 to Fp-m frames, and the Fp + 1 to Fp + n frames An average value of the motion amount is calculated from the motion amount of the detection area AR1. Further, by comparing the average value calculated as the amount of motion of the detection area AR1 with the threshold values Th1 and Th2 set as described above, the detection area AR1 in this Fp frame is a distant view, a middle view, and a foreground view. It is determined to which layer it belongs. In this way, for the detection area AR1 of the target frame, the average value of the motion amount is calculated using the motion amount of the previous frame and the subsequent frame, and the detection area AR1 is divided into layers using this average value. Even if an error occurs in the movement amount of the area AR1, the layer of the detection area AR1 can be classified, and even if the movement amount varies from frame to frame due to the difference in the size of the subject and the difference in the distance to the subject. , The influence of these fluctuations can be prevented. Further, by performing the same processing for other regions and other frames, each detection region is classified into a distant view, a middle view, or a foreground layer as shown in FIG. Is notified to the classification correction unit 265. In FIG. 12B, the detection area indicated by cross hatching is classified as a foreground, the detection area indicated by diagonal lines is classified as a middle scene, and the other detection areas are classified as a distant view.
[0053]
The classification correction unit 265 refers to the layer classification information LA of each detection area, and corrects an area where the number of connected areas belonging to the same layer is smaller than a predetermined number to a layer that matches the surrounding area. For example, in FIG. 12B, in the area AR4-g of the Fp + 4 frame and the area AR6-h of the Fp-2 frame, the areas belonging to the same layer are not connected, so these areas are surrounded as shown in FIG. 12C. The layer classification information LB indicating which area is set to which layer is corrected to the combined area, and is supplied to the information generation processing unit 27.
[0054]
Further, when there is an overtaking vehicle, a reverse layer including an image of the overtaking vehicle is generated as shown in FIG. Here, since the passing vehicle has the direction of the motion vector MV opposite to that of the distant view, the middle view, and the foreground image, the time axis direction of the backward layer is generated with the direction opposite to that of the other layers.
[0055]
FIG. 13 shows the configuration of the information generation processing unit 27. The layer classification information LB supplied from the layer classification unit 26 is supplied to the motion amount average value calculation unit 271 and the image reading unit 272.
[0056]
The motion amount average value calculation unit 271 calculates the average value of the motion amount for each layer for each frame using the motion amount of each detection region. For example, in the case of Fe frame, when there are ny detection areas that are the distant view layer, an average value is calculated using the amount of motion of the ny detection areas and supplied to the image reading unit 272. The amount of movement MYv is stored in the intermediate image information storage area 32 of the storage unit 30 as intermediate image information.
[0057]
Based on the layer classification information LB supplied from the layer classification unit 26, the image reading unit 272 extracts the image signal SDC for each layer according to the motion amount MYv calculated by the motion amount average value calculation unit 271 from the side end portion. Then, an intermediate image signal GFv for each layer is generated and supplied to the intermediate image signal interpolation unit 273.
[0058]
Here, when the image signal SDC is read from the side end portion by a signal amount corresponding to the motion amount MYv of each layer to generate an intermediate image signal, for example, an image based on the intermediate image signal is as shown in FIG. 14A shows an image based on the intermediate image signal of the distant view layer, FIG. 14B shows an image based on the intermediate image signal of the intermediate view layer, and FIG. 14C shows an image based on the intermediate image signal of the foreground layer.
[0059]
In the intermediate image signal interpolation unit 273, an intermediate image signal is generated so that an uncovered area without an image does not occur even if images of layers having different movement amounts MYv are pasted in the order of distant view, medium view, and close view. The GFv is corrected, and the corrected signal is stored in the intermediate image information accumulation area 32 as the intermediate image signal GYv. In the correction of the intermediate image signal, if all of the previous layers are interpolated, the inner layer will be hidden. Therefore, for the innermost layer, the entire gap is interpolated, and in the intermediate layer, only the gap generated by the area belonging to the layer before this layer is the signal adjacent in the horizontal direction. Is used to create an image signal in a gap area without an image and paste it on the intermediate image signal. By performing the interpolation processing in this way, the image based on the intermediate image signal GFv shown in FIG. 14A is interpolated using a signal in which a portion without an image (a region filled with diagonal lines) is adjacent in the horizontal direction. An image as shown in FIG. Similarly, the image based on the intermediate image signal GFv shown in FIG. 14B is an image corrected as shown in FIG. 15B. Note that when the foreground layer is the foreground layer shown in FIG. 14C, for example, the uncovered area does not occur in the foreground layer, so no interpolation processing is required, and the foreground layer shown in FIG. Is equivalent to FIG. 14C.
[0060]
In the intermediate image information storage area 32 of the storage unit 30, the motion amount MYv for each layer generated by the intermediate image information generation block 25 and the intermediate image signal GYv for each layer are associated with each other and stored as intermediate image information. Is done. Here, the intermediate image signal of the distant view layer stored in the intermediate image information storage area 32 has a small amount of motion, so that the signal amount of the image signal is small as shown in FIG. 15A. Further, the intermediate image signal in the foreground layer has a large amount of motion, and thus the amount of signal increases as shown in FIG. 15C. In the middle scene layer, as shown in FIG. 15B, the signal amount is intermediate between the far scene and the middle scene. In the backward layer, since the movement direction is reverse, the frame direction is reverse to that in FIGS. 15A to 15C in the intermediate image signal of the backward layer.
[0061]
When it is instructed to create a horizontal movement layer based on the layer pattern information LP, the image signal of the detection area in which the direction of the motion vector is the horizontal direction is set as the intermediate image signal of this horizontal movement layer. Here, in the image that moves in the horizontal direction, an image that moves outward from the front image and an image that enters inside are generated. For example, when turning right, the image enters from the right end of the front image and the image exits from the left end. For this reason, the horizontal movement layer of the image moving outside has the same time axis direction as the distant view layer, etc., and the horizontal movement layer of the image entering inside is reverse to the distant view layer etc. in the time axis direction, like the receding layer. The direction.
[0062]
In this way, in the intermediate image information storage area 32, the intermediate image signal of each of the distant view, the middle view, and the foreground layer and the intermediate image signal of the backward layer and the horizontal movement layer are generated and stored according to the amount of motion. . In addition, the amount of motion of each layer is also stored as described above.
[0063]
Next, a case where the peripheral image signal SDR of the right side image is generated using the intermediate image information stored in the intermediate image information storage area 32 of the storage unit 30 will be described. As described above, when the peripheral image signal SDR is generated, the motion amount MYv stored as the intermediate image information is read from the intermediate image information accumulation region 32, and the signal read amount of the intermediate image signal GYv for each layer is determined as the motion amount MYv. Determine based on. Further, an intermediate image signal having a signal amount corresponding to the motion amount MYv is used for each layer, and is pasted in the order of a distant view layer, a middle view layer, and a foreground layer. Further, when a layer different from the distant, middle, and foreground layers is provided, the peripheral image signal SDR can be generated by pasting the image of this layer. By supplying the generated peripheral image signal SDR to the projector 12R in accordance with the output timing of the image signal SDC of the front image, it is possible to continuously display not only the front image but also the right side image.
[0064]
FIG. 16 shows a configuration of a peripheral image signal generation block 40 that reads the intermediate image information stored in the intermediate image information storage area 32 and generates a peripheral image signal. The image generation control unit 41 of the peripheral image signal generation block 40 reads out the motion amount MYv for each layer stored as intermediate image information in the intermediate image information storage area 32 in the order of frames, and based on the read motion amount MYv The readout amount of the intermediate image signal for each is determined. Based on the read amount determined by the image generation control unit 41, the intermediate image signal GYv is read from the intermediate image information accumulation region 32 for each layer and pasted for each layer provided in the layer image generation processing unit 42. Parts 421-1 to 421-5 are supplied.
[0065]
Here, the intermediate image signal of the horizontal movement layer that moves in the horizontal direction when turning left or right is supplied to the pasting unit 421-1. Further, the intermediate image signal of the far view layer is supplied to the pasting unit 421-2, and the intermediate image signal of the middle scene layer and the intermediate image signal of the foreground layer are supplied to the pasting units 421-3 and 421-4. Further, the intermediate image signal of the reverse layer including the image of the overtaking vehicle is supplied to the pasting unit 421-5.
[0066]
An image shift unit 423-1, which will be described later, is connected to the pasting unit 421-1. The image signal supplied from the image shift unit 423-1 is added to the corresponding layer read from the intermediate image information storage area 32. Paste the image signal. The image signal of the horizontal movement layer obtained by pasting the image by the pasting unit 421-1 is supplied to the 1-frame delay unit 422-1 and the far-field pasting unit 441 of the image signal generating unit 44. .
[0067]
The 1-frame delay unit 422-1 delays the image signal supplied from the pasting unit 421-1 by one frame and supplies the delayed image signal to the image shift unit 423-1. The image shift unit 423-1 moves the image based on the image signal supplied from the 1-frame delay unit 422-1 in the horizontal direction based on the amount of movement MYv of the horizontal movement layer supplied from the image generation control unit 41. . Further, the image signal of the image moved in the horizontal direction is supplied to the pasting unit 421-1.
[0068]
Thus, the image based on the image signal delayed by one frame is moved in the horizontal direction based on the movement amount MYv supplied from the image generation control unit 41, and intermediate image information is added to the image moved in the horizontal direction. By pasting the intermediate image signal GYv of the horizontal movement layer read from the accumulation area 32 by the amount of movement MYv, an image signal of the moving image can be generated.
[0069]
Similarly, an image based on the image signal delayed by one frame by the one-frame delay units 422-2 to 422-5 is displayed on each layer supplied from the image generation control unit 41 by the image shift units 423-2 to 423-5. The intermediate image signal GYv read out from the intermediate image information storage area 32 by the amount of movement of each layer is attached to the image moved in the horizontal direction based on the amount of movement MYv. By supplying to 421-2 to 421-5 and pasting the image based on the read intermediate image signal GYv, it is possible to generate an image signal for sequentially moving the image for each layer.
[0070]
Further, since the intermediate image signal is a signal read from the front image signal, the image based on the intermediate image signal is an image on the same plane as the screen 10C. However, the screen 10R for displaying the peripheral image is provided with an inclination with respect to the front screen 10C. For this reason, the image of each layer is pasted using the image signal of the layer whose direction of the motion vector is the direction of the image reference position CP or the reverse direction, such as the distant view, the middle view, the foreground layer and the backward layer. If the peripheral image signal SDR is generated to display an image, the image displayed on the screen 10R does not have a correct shape due to the inclination of the screen 10R with respect to the screen 10C. For this reason, the image signal of the layer whose direction of the motion vector is the direction of the image reference position CP or the reverse direction is supplied to the conversion processing units 431 to 434 of the projection conversion unit 43, and the image of each layer is displayed on the screen 10L. , 10R, projective transformation is performed so that the image moves correctly in the vector direction of the motion vector MV. In addition, since the direction of the motion vector MV is horizontal in the image of the horizontal movement layer, projective transformation such as a distant view, a middle view, a foreground layer, and a backward layer is not performed.
[0071]
In the conversion processing units 431 to 434 of the projection conversion unit 43, the projective conversion is performed on the image signals supplied from the pasting units 421-2 to 421-5 so that the image has a shape corresponding to the direction of the screen 10R. Do. Here, when image signals are sequentially read from the front image, the display surface 10CR of the read image and the screen 10R are not the same surface as shown in FIGS. 17A and 17B, so the image read from the front image is applied to the screen 10R. When displayed, the image on the display surface 10CR is projected on the surface of the screen 10R as it is, and the correct shape is not obtained.
[0072]
For this reason, projective transformation is performed so that the vertical direction is enlarged at an enlargement ratio proportional to the distance from the center Czm of the zoomed-in or zoomed-out image, and the horizontal direction is the distance from the center Czm of the zoomed-in or zoomed-out. Magnify at a proportional scale.
[0073]
Here, as shown in FIG. 17C, the length from the front end of the right side image to the center Czm is L, the length in the peripheral direction of the right side image is A, and the angle “θs between the front image and the right side image is formed. Assuming that the proportionality constant based on “α” is α, the position (xa, ya) of the image read as it is from the front screen edge and the position (Xa, Ya) after the conversion processing so that the image can be correctly displayed on the right side image are In the vertical direction, the relationship is expressed by the equation (3), and in the horizontal direction, the relationship is expressed by the equation (4). For this reason, as shown in FIG. 17D, the image signal of (Xa, Ya) on the screen 10R uses the signal of the position (xa, ya) satisfying the expressions (3) and (4), thereby FIG. As shown in FIG. 4, the right side image can be correctly displayed on the screen 10R.
[0074]

[0075]
The image signal of the distant view layer after the projective conversion obtained by the conversion processing unit 431 is supplied to the distant view pasting unit 441 of the image signal generating unit 44. Further, the image signal of the foreground layer after the projective transformation obtained by the conversion processing unit 432 is supplied to the middle scene pasting unit 442 and the image signal of the foreground layer after the projective transformation obtained by the transformation processing unit 433. Is supplied to the foreground pasting unit 443. Further, the image signal of the backward layer obtained by the conversion processing unit 434 is supplied to the reduced paste unit 444.
[0076]
In the distant view pasting unit 441, based on the image signals supplied from the pasting unit 421 and the conversion processing unit 431, an image signal in which the image of the far view layer is pasted on the image of the horizontal movement layer is generated and pasted in the middle scene To the unit 442.
[0077]
Based on the image signals supplied from the conversion processing unit 432 and the distant view pasting unit 441, the mid view pasting unit 442 outputs an image signal obtained by pasting the mid view layer image to the image pasted with the distant view layer image. It is generated and supplied to the foreground pasting unit 443.
[0078]
In the foreground pasting unit 443, based on the image signals supplied from the conversion processing unit 433 and the middle scene pasting unit 442, an image signal obtained by pasting the foreground layer image to the image pasted with the middle scene layer image is output. Generated and supplied to the reduced paste unit 444.
[0079]
Based on the image signals supplied from the conversion processing unit 434 and the foreground pasting unit 443, the reduction pasting unit 444 pastes the backward layer image that shrinks over time to the image pasted with the foreground layer image. The attached image signal is generated. The image signal generated by the reduced paste unit 444 becomes an image signal of a side image in which each layer from the horizontal movement layer to the reduced layer is pasted.
[0080]
For this reason, by supplying the image signal generated by the reduced paste unit 444 to the projector 12R as the peripheral image signal SDR, an image on the right side continuous with the front image can be displayed on the screen 10R.
[0081]
Further, when the intermediate image signal stored in the intermediate image information storage area 32 is not subjected to the process of interpolating a portion without an image, or the image signal generated by the reduced paste unit 444 has no image. When a part is generated, a pre-interpolation processing unit is provided to determine what layer image is pasted in an area adjacent to a part without an image, and an interpolation process is performed using an image of a deep layer . For example, when a portion without an image occurs in a portion adjacent to the region where the images of the foreground layer and the foreground layer are combined in the horizontal direction, interpolation processing is performed using the image of the deep foreground layer. Therefore, it is possible to generate a good peripheral image signal SDR with no image loss.
[0082]
Further, by generating the peripheral image signal SDL in the same manner as the peripheral image signal SDR and supplying it to the projector 12L, an image on the left side continuous with the front image can be displayed on the screen 10L.
[0083]
Further, the storage unit 30 stores intermediate image information. Since this intermediate image information is an intermediate image signal for each motion amount and layer, the information is compared with the case of storing an image signal of a peripheral image. The amount can be reduced. Therefore, it is possible to display an image with a wide angle of view without using a large capacity recording medium or the like.
[0084]
By the way, in the above-described embodiments, intermediate image information is determined by layering images such as a distant view, a middle view, and a foreground from the motion vector MV of the detection area provided on the side edge side of the front image, and determining the amount of movement. Is generated and stored in the intermediate image information storage area 32, and when the peripheral image is displayed, the intermediate image information stored in the intermediate image information storage area 32 is used in accordance with the motion amount MYv of each layer. The peripheral image signal is generated by reading the intermediate image signal GYv and pasting the image, but the peripheral image signal can also be generated without using the motion vector MV. Next, as a second embodiment, an image signal processing apparatus capable of generating a peripheral image signal without using a motion vector MV will be described.
[0085]
In the image signal processing device 55 of the second embodiment, a projected image in the lateral direction is generated from the front image by geometric transformation, and an integrated image is generated by integrating the projected image, and an image is generated from the integrated image. In other words, the movement amount MYs for each layer and the intermediate image signal for each layer are generated and stored as intermediate image information in the intermediate image information storage area 32.
[0086]
FIG. 18 shows the configuration of the image processing device 55. The image signal SDC of the front image is supplied to the geometric conversion unit 61. The geometric conversion unit 61 generates a projected image signal SDP of an image in which the front image is displayed on the projection plane that is the plane in the traveling direction by using the image signal SDC of the front image by the geometric conversion process. The projected image signal SDP generated by the geometric conversion unit 61 is supplied to the motion determination image generation processing unit 63 of the motion detection block 62, the integral image generation processing unit 64, and the information generation processing unit 72 of the intermediate image information generation block 70. The
[0087]
The motion discriminating image generation processing unit 63 extracts a projection image signal SDP within a predetermined range from the end side in contact with the front image for each frame from the projection image signal SDP supplied from the geometric conversion unit 61 and sequentially pastes it. A motion discrimination image signal SMD is generated. The motion determination image signal SMD is supplied to the motion amount determination unit 65.
[0088]
The integral image generation processing unit 64 generates one integral image signal by adding and averaging the projection image signals SDP for ms frames while shifting the position in the horizontal direction. Also, ns integrated image signals are generated with the position shift amount being ns different values. The ns integrated image signals SA generated by the integrated image generation processing unit 64 are supplied to the motion amount determination unit 65.
[0089]
The motion amount determination unit 65 compares the ns integral image signals SA and the motion determination image signal SMD, detects an image based on the integrated image signal SA that matches the image based on the motion determination image signal SMD, and detects the detected image. Based on the result, the amount of motion at a predetermined position of the frame of interest is determined. Here, in the integrated image, the subject appears in the integrated image when the position shift amount and the image moving amount match. Since ns integral images are generated with different shift amounts, ns different images with different movement amounts are indicated by the integral image signal SA. Therefore, the correlation between the ns integral image signals SA and the motion discrimination image signal SMD is discriminated, and based on the discrimination result, the movement amount of the subject constituting the motion discrimination image, that is, the motion amount at a predetermined position of the target frame is determined. Can be determined. Here, since the movement amount is small when the subject image is a distant view, and the movement amount is large when the subject image is a close view, this movement amount indicates the depth of the subject image. In this way, when the motion amount discriminating unit 65 discriminates the motion amount of each position of the projected image, the motion amount information MF indicating the discrimination result is generated, and the layer classification unit 71 of the intermediate image information generation block 70 and the information generation processing To the unit 72.
[0090]
The layer classification unit 71 generates layer classification information LB indicating which of a plurality of layers the predetermined position of the frame of interest belongs to based on the supplied motion amount information MF, and supplies the layer classification information LB to the information generation processing unit 72.
[0091]
In each layer, the information generation processing unit 72 obtains an average value using the movement amount of the preceding and following frames for each frame and stores it in the intermediate image information accumulation region 32 of the accumulation unit 30 as the movement amount MYs of the corresponding frame of each layer. Supply. Further, based on the obtained amount of movement MYs for each layer and the layer classification information LB supplied from the layer classification unit 71, the image signal SDP supplied from the geometric conversion unit 61 is classified for each layer to obtain an intermediate image signal GFs. Generate. That is, the intermediate image signal GFs for each layer similar to that in FIG. 14 is generated by reading out the image signal in the horizontal direction by the amount of motion and sequentially pasting it for each corresponding layer.
[0092]
In addition, when the movement direction of the images in the distant view layer, the middle view layer, and the foreground layer is used as a reference, the movement direction is reverse in the backward layer. For this reason, the intermediate image signal of the receding layer can be generated by reading the projection plane image signal from the end side in contact with the front image by the absolute value of the amount of motion, and performing left-right reversal and pasting.
[0093]
Further, interpolation processing is performed on the created intermediate image signal GFs for each layer so that an uncovered area without an image is generated, thereby embedding a portion without an image. The intermediate image signal GYs for each layer obtained by performing the interpolation processing in the information generation processing unit 72 is stored in the intermediate image information accumulation region 32 as intermediate image information in association with the motion amount MYs of each layer. .
[0094]
By reading out the intermediate image information generated in this way from the intermediate image information storage area 32 and processing it in the same manner as the above-described peripheral image signal generation block 40, the peripheral image signal of the right side image can be generated.
[0095]
Next, each block will be described in detail. FIG. 19 is a diagram for explaining the geometric transformation process in the geometric transformation unit 61. In FIG. 19, a position O is a photographing position of the front image. The geometric conversion unit 61 projects the image at the position p of the front image onto the position q on the plane parallel to the traveling direction, and generates a projected image from the front image. Here, the relationship between the position xb in the traveling direction on the projection plane and the horizontal position Xb on the front image is as shown in Expression (5). Further, the relationship between the height position yb on the projection surface and the vertical position Yb on the front image is as shown in Expression (6).
[0096]
[Equation 3]

[0097]
In Equations (5) and (6), “θg” indicates an angle obtained by halving the horizontal angle of view of the front image. When the horizontal angle of view of the camera that captured the front image is clear, this horizontal image is displayed. The angle half of the angle is “θg”. When the horizontal angle of view of the camera is not clear, a value set in advance according to the focal length of the used lens is used. “D” is a value obtained by halving the horizontal length of the front image (the horizontal length of the image frame).
[0098]
In this way, by using the equations (5) and (6), the image signals of the left and right projected images can be generated in units of frames from the image signals of each frame of the front image. When the input image is a front image of an in-vehicle forward system, the left and right projected images are side images, and the amount of movement of the projected image is proportional to the vehicle speed. The amount of motion of the projected image is inversely proportional to the depth. Increases the amount of movement.
[0099]
FIG. 20 is a diagram for explaining the operation of the motion discrimination image generation processing unit 63. In the motion discrimination image generation processing unit 63, as shown in FIG. 20A, a predetermined number of pixels wa (for example, 4 pixels) in the horizontal direction from the front image side end of the projection plane image (right end in the right projection plane image) for each frame. The image signal SMD of the motion discrimination image is generated by cutting out the image signal with a width of) and pasting the cut-out image signal by a predetermined number of frames in the frame order as shown in FIG. 20B. In the motion determination image, when the amount of motion of the subject is wa pixels / frame in the horizontal direction and is equal to the number of cutout pixels wa of the image signal (house in FIG. 20), the subject image is continuous without discontinuous points. It becomes an image. In an image of a subject (moving in FIG. 20) whose movement is slower than the cutout of the image signal, a portion that is cut out repeatedly is generated, so that a discontinuity of the image occurs at a connecting portion of the cut out image signals. Also, in the subject image (the tree in FIG. 20) that moves faster than the cutout of the image signal, the portion that is not cut out due to the fast movement, and the discontinuity of the image occurs at the connecting portion.
[0100]
The integrated image generation processing unit 64 generates an integrated image signal SA used for determining the amount of motion of each pixel of the motion determination image. Here, the integral image will be described with reference to FIG. For example, when a subject Ja moving rightward and a subject Jb moving rightward at twice the speed of the subject Ja are photographed, the photographed image is as shown in FIG. 21A. Here, the image of the next frame F2 is shifted by the pixel amount dx of the subject Ja in the direction opposite to the moving direction of the subject Ja with respect to the first frame F1. Similarly, the image of the next frame F3 is also shifted by the pixel amount dx of the subject Ja in the direction opposite to the moving direction of the subject Ja with respect to the frame F2. As described above, when an image obtained by adding the images of the shifted frames and averaging them is an integrated image, the position of the subject Ja is constant and the position of the subject Jb moves. The level is smaller than that of the subject Ja, and only the subject Ja can be extracted from the integrated image as shown in FIG. 21C.
[0101]
Further, as shown in FIG. 21D, when the integral image is generated with the shift amount of the image of each frame being doubled (2dx), the position of the subject J becomes constant and the position of the subject Ja moves. As shown in 21E, only the subject Jb can be extracted from the integrated image.
[0102]
In this way, the image of each frame is moved by a predetermined amount, added and averaged to form an integral image, and when a plurality of integral images are generated by changing the shift amount of the image of each frame, By determining in which integrated image the subject is correctly displayed, the amount of motion of each subject can be determined using the integrated image even if subjects with different amounts of motion are included in one image. That is, since the amount of motion of the subject constituting the image of each frame can be determined, the amount of motion at a predetermined position of the frame of interest can be determined.
[0103]
FIG. 22 is a diagram illustrating a configuration of the integral image generation processing unit 64. The projected image signal SDP supplied from the geometric conversion unit 61 is supplied to the arithmetic processing units 641-1 to 641-ns of the integral image generation processing unit 64.
[0104]
In the arithmetic processing unit 641-1, the projection plane image signal moved in the horizontal direction from the previous frame by the preset number of pixels SH-1 is added for ms frames (for example, 10 frames), and the addition signal is sequentially added. calculate. Further, a signal obtained by averaging the added signal by 1 / m is supplied to the pixel cutout unit 642-1.
[0105]
The pixel cutout unit 642-1 cuts out an image signal of a predetermined number of pixels from the supplied image signal in the horizontal direction from the front image side end portion (right side end portion in the right projection plane image) for each frame. By sequentially pasting the image signals, the integrated image signal SA-1 is output. The integrated image signal SA-1 output from the image cutout unit 642-1 is an image signal indicating a subject with a motion amount of a preset number of pixels SH-1.
[0106]
Similarly to the arithmetic processing unit 641-1, the arithmetic processing units 641-2 to 641-ns also output projection plane image signals that are moved in the horizontal direction from the previous frame by a preset number of pixels SH-2 to SH-ns. , Add the ms frames, sequentially calculate the added signal, multiply the added signal by 1 / m, average it, and supply it to the pixel cutout units 642-2 to 642-ns. Further, in the pixel cutout units 642-2 to 642-ns, image signals of a predetermined number of pixels wb are cut out and sequentially pasted and output as integrated image signals. The integrated image signals SA-2 to SA-ns output from the pixel cutout units 642-2 to 642-ns are images showing subjects having a predetermined amount of motion of the pixels SH-2 to SH-3, respectively. Signal.
[0107]
If the number of pixels cut out by the pixel segmentation unit 642 of the motion discrimination image generation processing unit 63 and the integral image generation processing unit 64 is equal (wa = wb), the consistency of the position of the subject in the motion discrimination image and the integration image is taken. Can do.
[0108]
FIG. 23 is a diagram illustrating the operation of the integral image generation processing unit 64. When the image shift amount is the number of pixels SH-r, the image is moved and added by the number of pixels SH-r (<wb) for each frame as shown in FIG. 23A. An image is extracted. When the image shift amount is set to a pixel number SH-s (= wb) larger than the pixel number SH-r, the image is moved and added by the pixel number SH-s for each frame as shown in FIG. 23C. As shown in FIG. 23D, an image of a house having a larger amount of movement than a mountain is extracted. Further, when the image shift amount is set to the number of pixels SH-t (> wb) larger than the number of pixels SH-s, the image is moved and added by the number of pixels SH-t for each frame as shown in FIG. 23E. Then, as shown in FIG. 23F, an image of a house having a larger amount of movement than a mountain is extracted.
[0109]
Here, when the motion discrimination image generation unit 63 generates the motion discrimination image, the clipping starts with the frame Fa first cut out as a reference, and the same frame with the same cutout amount (wa = wb) as the motion discrimination image. When an integral image is generated by cutting out a number of images, the integral image and the motion determination image have the same horizontal image length, and the phase of the subject matches between the integral image and the motion determination image.
[0110]
In the motion discrimination image, when the cutout amount and the motion amount are not equal, the subject has a discontinuous shape. Similarly, in the case of the integral image, when the pixel number SH-r is smaller than the pixel number wb, Since the cutout is performed with overlap, as shown in FIG. 23G, a discontinuous image corresponding to the amount of motion is obtained. When the number of pixels SH-s is equal to the number of pixels wb, continuous images corresponding to the amount of motion are obtained as shown in FIG. 23H. Further, when the number of pixels SH-t is larger than the number of pixels wb, a missing portion is generated by cutting out the image, and a discontinuous image corresponding to the amount of motion is obtained as shown in FIG. For this reason, the motion discriminating unit 65 described below performs matching between the motion discriminating image and the integrated image, and discriminates the amount of motion of the subject based on the number of pixels SH which is the image shift amount of the integrated image discriminated to be matched. be able to.
[0111]
The motion amount discriminating unit 65 obtains a correlation value between each pixel of the motion discrimination image and the integrated image, and based on this correlation value, discriminates which integrated image each pixel of the motion discrimination image is equal to. The amount of movement of each pixel of the discrimination image is discriminated.
[0112]
In the calculation of the correlation value, when calculating the correlation value between the pixel of the image A and the image B, a predetermined range centering on the target pixel of the image A (pixel for calculating the correlation value) is set, and this predetermined range is set. The correlation value of the target pixel is obtained using the image signal. For example, a rectangular range of ± mvc pixels in the x direction and ± nvc pixels in the y direction around the target pixel (for example, a rectangular range of 31 horizontal pixels and 5 vertical pixels centered on the target pixel) is set. ) To calculate a correlation value VC.
[0113]
[Expression 4]

[0114]
In Expression (7), the signal level of each pixel in the rectangular range of the image A is DAi (i = 1 to (mvc × nvc)), and the signal level of each pixel in the rectangular range of the image B is DBi. (I = 1 to (mvc × nvc)), and the average value of the signal level of each pixel in the rectangular range is DAav, DBav.
[0115]
For example, when it is determined that the mountain image in the motion determination image shown in FIG. 24A matches the integrated image shown in FIG. 24B based on the equation (7), the motion amount of each pixel constituting the mountain image is It can be determined that the amount of motion corresponds to the number of pixels SH-r. When it is determined that the house image in the motion determination image shown in FIG. 24A matches the integrated image shown in FIG. 24C, the amount of movement of each pixel constituting the house image becomes the number of pixels SH-s. It can be determined that the amount of movement is corresponding. Further, when it is determined that the tree image in the motion determination image shown in FIG. 24A matches the integrated image shown in FIG. 24D, the motion amount of each pixel constituting the tree image corresponds to the number of pixels SH-t. Can be determined. In this way, when the motion amount determination unit 65 determines the motion amount for each pixel, the motion amount information MF indicating the determination result is supplied to the layer classification unit 71 and the information generation processing unit 72.
[0116]
Although not shown, in the integrated image generation processing unit 64, the image shift direction is set so as to include the reverse direction of FIGS. 23A, 23C, and 23E (for example, the image shift amount SH is 48 pixels to −24). If set to pixels), it is possible to extract an image of an overtaking vehicle or the like that moves in the opposite direction to the image of a tree, house, or mountain. In addition, in order to determine the amount of movement of an image whose direction of movement is opposite to that of a distant view layer or the like, as shown in FIG. 25A, the image is cut out from the image of each frame, and this image is reversed and pasted. Thus, the motion discrimination image shown in FIG. 25B is generated. In this case, in the motion determination image shown in FIG. 25B, an image with the opposite direction of movement, such as an overtaking vehicle, is correctly displayed in the same direction as the distant view, middle view, and foreground layer. Note that FIG. 25C is a motion determination image when the image is pasted without being reversed horizontally. For this reason, the image signal of the integral image with the image shift direction being the reverse direction and the image signal of the motion discrimination image generated by reversing and pasting the clipped image are sequentially compared from the reference frame. Thus, the amount of movement of the subject moving in the reverse direction can be determined.
[0117]
The layer classification unit 71 layers each pixel of the motion determination image in the depth direction based on the motion amount information MF. For example, assuming that the depth is previously layered into four layers having a movement in the opposite direction to the three layers of the distant view, the middle view, and the foreground, layer classification is performed as to which layer each pixel belongs to.
[0118]
In this layer classification, a threshold value is set in the same manner as the threshold value setting unit 264 described above, and the threshold value is compared with the amount of motion of each pixel to determine which layer each pixel belongs to. Is generated. The generated layer classification information LB is supplied to the information generation processing unit 72.
[0119]
FIG. 26 shows the configuration of the information generation processing unit 72. Based on the motion amount information MF from the motion amount determination unit 65 and the layer classification information LB supplied from the layer classification unit 71, the motion amount average value calculation unit 721 of the information generation processing unit 72 performs each frame of interest in each layer. Then, an average value of the motion amounts of the preceding and following nz frames is obtained and set as the speed of the target frame of the layer. The calculated amount of movement MYs of each layer of each layer is supplied to the image reading unit 722 and stored in the intermediate image information storage area 32.
[0120]
Based on the layer classification information LB, the image reading unit 722 extracts a signal for each layer in accordance with the motion amount MYs calculated by the motion amount average value calculation unit 721 from the image signal SDP of the projected image, An image signal GFs is generated. The intermediate image signal GFs is supplied to the intermediate image signal interpolation unit 723. At this time, since the motion amount of the distant view layer is small and the motion amount of the foreground layer is large, the intermediate image of the distant view layer is short in the horizontal direction and is long in the intermediate image of the close view layer, as in FIG. Also, since the movement in the backward layer is in the reverse direction, an intermediate image of the backward layer is created by reading the image from the side image by the absolute value of the amount of movement and pasting it in the left-right direction.
[0121]
Similar to the intermediate image signal interpolation unit 273, the intermediate image signal interpolation unit 723 does not have an image regardless of the amount of movement even if images of layers having different movement amounts MYs are pasted in the order of distant, middle, and foreground. The intermediate image signal GFs is corrected so that an uncovered area does not occur, and the corrected signal is stored in the intermediate image information storage area 32 as the intermediate image signal GYs.
[0122]
Further, when there is a backward layer such as an overtaking vehicle, the interpolation process is not performed because it is a layer in front of the foreground layer. In addition, when a horizontal movement layer indicating a building or the like that appears in a distant view at the time of turning left or right is provided, interpolation processing is performed in the same manner as the distant view layer. Note that the subject of the backward layer or the horizontal movement layer is determined by determining the motion pattern based on the motion direction of the motion determination image of the right projection image and the left projection image in the same manner as in FIG. Can be determined. For example, it can be determined that a right turn or a left turn is being performed when the movements of the left image and the right image indicate different directions. In addition, the movement of the upper part of the screen of the left image and the right image shows the same direction, and a subject that moves in the opposite direction can be determined as a backward layer.
[0123]
Next, an operation for generating a peripheral image using the intermediate image information of each layer will be described. In the generation of the peripheral image, a peripheral image signal can be generated in the same manner as the peripheral image signal generation block 40 described above. That is, the image of each layer is read by the amount of movement, and pasted in the order of the distant view layer, the middle view layer, and the foreground layer. Even when the horizontal movement layer or the receding layer is set, the peripheral image signal can be generated in the same manner as the peripheral image signal generation block 40.
[0124]
In addition, when a right side image is generated as a peripheral image, for example, the same frame portion of each layer is pasted so as to overlap at the left end of the generated right side image. In addition, the right end of the projected image first converted from the front image is set to be equal to the left end of the generated side image. After that, the image is moved for each layer based on the amount of movement for each layer, and the next image is pasted to the moved image for each layer, so that it is continuous with the projection plane image as shown in FIG. A side image can be generated.
[0125]
By the way, the image generated by pasting the intermediate image for each layer using the intermediate image information is an image on the same plane as the projection plane as shown in FIG. On the other hand, the screen 10R on which the peripheral image is displayed is installed with an inclination with respect to the projection plane. For this reason, when an image of each layer is pasted and displayed on the screen 10R, an image conversion process is performed so that the image moves in a correct shape.
[0126]
Here, as shown in FIG. 28, the screen is such that ½ of the length of the front image in the horizontal direction is the distance d, and the position O that is the shooting position of the front image is on the normal line from the center of the screen 10R. When 10R is inclined and installed, the image signal at the position (Xc, Yc) on the screen 10R is the image signal at the image position (xc, yc) before conversion calculated based on the equations (8) and (9). If extracted, the peripheral image signal SDR of the right side image after projective transformation can be easily generated. Further, by using the image signal SDR to display the right side moving image on the screen 10R, it is possible to display a right side image with a high sense of presence.
[0127]
[Equation 5]

[0128]
Of course, the peripheral image signal SDL can be generated in the same manner as the peripheral image SDR.
[0129]
Furthermore, the processing performed in each processing block described above may be realized not only by hardware but also by software. The configuration in this case is shown in FIG.
[0130]
The computer 90 includes a CPU (Central Processing Unit) 901 as shown in FIG. 29. A ROM 902, a RAM 903, a hard disk drive 904, and an input / output interface 905 are connected to the CPU 901 via a bus 920. ing. Further, an input unit 911, a recording medium drive 912, a communication unit 913, and an image input / output unit 914 are connected to the input / output interface 905.
[0131]
When a command is input from an input unit 911 configured using an operation input unit such as a keyboard or a mouse or a voice input unit such as a microphone, the command is supplied to the CPU 901 via the input / output interface 905.
[0132]
The CPU 901 executes a program stored in the ROM 902, RAM 903, or hard disk drive 904, and performs processing according to the supplied command. Further, the ROM 902, the RAM 903, or the hard disk drive 904 also stores a program for executing the processing in each block described above, and generates a side image.
[0133]
Here, generation of intermediate image information and an image processing program for generating and outputting a peripheral image signal from the intermediate image information can be stored in advance in a ROM 902 or a hard disk drive 904 that is an information recording medium.
[0134]
Alternatively, a removable recording medium using magnetic or light, a removable information recording / transmission medium configured using a semiconductor element, for example, an optical disk such as a floppy (R) disk or a CD-ROM, a magneto-optical disk such as an MO disk, An image generation program is recorded on a tape cartridge, a semiconductor memory or the like, and the removable information recording / transmission medium is mounted on a recording medium drive 912 to read the recorded image processing program, and the read program is input / output interface You may install by memorize | storing to the hard disk drive 904 etc. via 905 or the bus | bath 920. FIG.
[0135]
Further, a communication unit 913 is provided to receive an image processing program supplied using a wired or wireless transmission path, for example, a network such as a LAN or the Internet, a satellite broadcast wave, a terrestrial broadcast wave, and the like. The image processing program can be installed in the hard disk drive 904 or the like via the input / output interface 905 and the bus 920.
[0136]
Here, when the intermediate image information is generated by executing the image processing program, the image signal SDC of the front image recorded on the recording medium mounted on the hard disk drive 904 or the recording medium drive 912 is obtained. After being read out, the above-described processing is performed, and the intermediate image information is recorded on a recording medium mounted on the hard disk drive 904 or the recording medium drive 912 together with, for example, the image signal SDC. When the output of the image signal of the peripheral image is requested, the intermediate image information is read to generate the image signals SDR and SDL of the left and right side images, and the image signals SDR and SDL are the image signals of the front image. Output in synchronization with SDC. For this reason, it is possible to display a continuous image with a wide angle of view using the front and left and right screens.
[0137]
Thus, according to the above-described embodiment, it is possible to determine the movement of the front image and generate a peripheral image in a different direction based on the movement determined from the front image. For this reason, since moving images in different directions can be generated simultaneously from front moving images taken with a video camera or the like, a multi-screen image with a high sense of realism and a wide angle of view can be presented.
[0138]
In addition, since images in different directions can be generated from one input image, it is not necessary to use a plurality of video cameras, cameras using wide-angle lenses, or the like, and shooting can be performed easily.
[0139]
Furthermore, since an image in a different direction is generated by using a live-action image, it is possible to display an image that is more realistic and realistic than an image in a three-dimensional virtual space by computer graphics, and has a depth of 2 Since it is expressed as a hierarchy of a three-dimensional plane, three-dimensional arithmetic processing is not necessary and signal processing is easy.
[0140]
Further, by using the signal accumulated as the input image signal, an image that enters the front image as time passes can be displayed as a peripheral image before being displayed as the front image. In this way, since an image that cannot be displayed when using a real-time image signal can be displayed as a peripheral image, a more realistic and highly realistic image display with a wide angle of view can be performed. In addition, by using an enormous image source that already exists as an input image signal, it is possible to enjoy these images with a high realistic angle and a wide angle of view.
[0141]
In the above-described embodiment, a plurality of detection areas are provided at the side edges of the front image to obtain a motion vector for each detection area, or a motion discrimination image is generated from a projected image and compared with an integral image. The intermediate image information is generated by discriminating the movement of the target frame at the predetermined position, and the peripheral image signal is generated using the intermediate image information. The motion detection can determine the movement at the predetermined position of the target frame. Of course, the above method is illustrative and not limiting.
[0142]
【The invention's effect】
According to the present invention, a motion at a predetermined position of a target frame in an input image signal is detected using the input image signal of a plurality of frames, and intermediate image information is based on the detected motion and the input image signal of the plurality of frames. Is generated. Further, based on this intermediate image information, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated. For this reason, by generating a peripheral image signal having an angle of view that does not exist in the frame of interest using the actually captured input image signal, the peripheral image can be displayed as a realistic and highly realistic image. In addition, a wide-angle image can be easily presented by generating peripheral images in different directions from one input image without using a plurality of video cameras or video cameras using special lenses. In addition, it is possible to present an image with a high sense of realism and a wide angle of view using a huge image source that already exists.
[0143]
Further, the layer classification based on the movement of the predetermined position is performed to classify the predetermined position into any one of a plurality of layers. Based on the classification result, the input image signal, and the movement of the predetermined position, the motion amount and the layer for each layer are classified. Each image signal is used as intermediate image information. For this reason, layering in the depth direction is performed based on the movement on the two-dimensional plane, and image signals in each layer can be pasted, so that three-dimensional image presentation can be performed without performing three-dimensional processing. .
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of a display system.
FIG. 2 is a diagram illustrating an image projection state.
FIG. 3 is a diagram illustrating a configuration of an image processing apparatus.
FIG. 4 is a diagram illustrating a motion vector of each detection area.
FIG. 5 is a diagram for explaining a motion vector discrimination method;
FIG. 6 is a diagram for explaining a change in area size during zooming;
FIG. 7 is a diagram illustrating a method for determining an image reference position.
FIG. 8 is a diagram illustrating a configuration of a motion detection processing unit.
FIG. 9 is a diagram illustrating a configuration of a layer classification unit.
FIG. 10 is a diagram for explaining a relationship between a vector direction and a motion pattern.
FIG. 11 is a diagram for explaining a threshold setting method;
FIG. 12 is a diagram for explaining a generation operation of layer classification information.
FIG. 13 is a diagram illustrating a configuration of an information generation processing unit.
FIG. 14 is a diagram illustrating an intermediate image for each layer.
FIG. 15 is a diagram illustrating an intermediate image for each layer after interpolation processing;
FIG. 16 is a diagram illustrating a configuration of a peripheral image signal generation block.
FIG. 17 is a diagram for explaining image conversion processing;
FIG. 18 is a diagram illustrating another configuration of the image signal processing apparatus.
FIG. 19 is a diagram for explaining a geometric conversion process;
FIG. 20 is a diagram for explaining an operation of a motion determination image generation unit;
FIG. 21 is a diagram for explaining an integral image generation operation;
FIG. 22 is a diagram illustrating a configuration of an integral image generation unit.
FIG. 23 is a diagram for explaining the operation of the integral image generation unit.
FIG. 24 is a diagram for explaining the operation of a motion amount determination unit;
FIG. 25 is a diagram for explaining the operation of generating the motion discrimination image of the backward layer.
FIG. 26 is a diagram illustrating a configuration of an information generation processing unit.
FIG. 27 is a diagram showing a projected surface image and a generated side image.
FIG. 28 is a diagram for describing image conversion processing for correctly displaying an image on a screen.
FIG. 29 is a diagram illustrating a configuration when a computer is used.
[Explanation of symbols]
10L, 10C, 10R ... screen, 12L, 12C, 12R ... projector, 15, 55 ... image processing device, 20, 62 ... motion detection block, 21 ... frame delay processing unit, 22 ... motion detection processing unit, 25, 70 ... intermediate image information generation block, 26, 71 ... layer classification unit, 27, 72 ... information generation processing unit, 30 ... storage unit, 31 ..Front image signal storage region, 32... Intermediate image information storage region, 40 .. Peripheral image signal generation block, 41... Image generation control unit, 42. Projection conversion unit, 44 ... image signal generation unit, 61 ... geometric conversion unit, 63 ... motion discrimination image generation processing unit, 64 ... integral image generation processing unit, 65 ... motion amount discrimination Part, 90... Computer, 221. Size conversion processing unit 222 ... Error sum calculation unit 223 ... Comparison processing unit 224 ... Data holding unit 225 ... Search control unit 261 ... Pattern determination unit 262 ... Layer creation determination unit, 263, classification processing unit, 264, threshold setting unit, 265, classification correction unit, 271, 721, motion amount average value calculation unit, 272, 722, image read Part, 273,723 ... intermediate image signal interpolation part, 421-1 to 421-5 ... pasting part, 422-1 to 422-5 ... 1 frame delay part, 423-1 to 423-5 ... Image shift part, 431-434 ... Conversion processing part, 441 ... Distant view pasting part, 442 ... Middle view pasting part, 443 ... Near view pasting part, 444 ... Reduction Pasting unit, 641-1 to 641-ns ... arithmetic processing unit, 642-1 to 42-ns: pixel cutout unit, 901 ... CPU, 902 ... ROM, 903 ... RAM, 904 ... hard disk drive, 905 ... input / output interface, 911 ... Input unit, 912, recording medium drive, 913, communication unit, 914, image input / output unit, 920, bus

Claims (22)

Using a plurality of frames of input image signals, motion detecting means for detecting a motion of a predetermined position of the frame of interest in the input image signals;
Intermediate image information generating means for generating intermediate image information based on the motion detected by the motion detecting means and the input image signals of the plurality of frames;
The intermediate image data based on the generated intermediate image information generating means, images processing device Ru and a peripheral image signal generation means for generating a peripheral image signal of the angle of view that does not exist in the frame of interest. The intermediate image information generating means includes
Layer classification means for performing layer classification based on the movement of the predetermined position and classifying the predetermined position into one of a plurality of layers;
There is information generation processing means for generating the amount of movement for each layer and the image signal for each layer as the intermediate image information based on the classification result in the layer classification means, the input image signal, and the movement of the predetermined position. the image processing apparatus 請 Motomeko 1, wherein that. The layer classification means includes
The threshold used when performing the layer classification, the image processing apparatus 請 Motomeko 2 wherein that having a threshold setting means for determining based on the motion amount of the predetermined position of a predetermined number of frames. The information in the generating unit, the image processing apparatus performs the 請 Motomeko 2 wherein that generates an image signal of each layer extracts the signal from the input image signal based on the motion amount of each of the layers for each layer. The information generation processing means
When there is a region where the image disappears when the images based on the image signals for each layer are pasted together in a predetermined order, an interpolation process is performed using the image signal of the peripheral region of the region where the image disappears, and the image disappears the image processing apparatus 請 Motomeko 2 wherein that having a intermediate image interpolation processing means for generating an image signal of a region. The multiple layers when performing the layer classified by the layer classification unit, the image processing apparatus of the shown to請 Motomeko 2 wherein the depth. The motion detection means detects a correlation between an image at a predetermined position of the frame of interest and an image of a previous frame or a subsequent frame with respect to the frame of interest, and starts from the position of the image having the highest correlation with the predetermined position of the frame of interest. the image processing apparatus 請 Motomeko 2 wherein you detect the motion direction and motion amount. The said layer classification means, from the input image signal of the plurality of frames, the detection is performed motion pattern based on the movement direction of the predetermined position, the detection result to the 請 Motomeko 7, wherein to set the plurality of layers in accordance Image processing device. Wherein in the layer classification means, based on said amount of motion of the predetermined position detected by the motion detection means, the image processing apparatus of rows cormorants請 Motomeko 8, wherein the classification into the plurality of layers. Having a geometric conversion means for generating an image signal of a projected image obtained by projecting an image based on an input image signal onto a predetermined plane;
The motion detection means detects a motion at a predetermined position of the frame of interest using an image signal of the projected image,
It said intermediate in the image information generating means, the image processing apparatus detected 請 Motomeko 1 wherein the motion and on the basis of the image signal of the projection image that generates the intermediate image information in the motion detecting means. The motion detection means includes
A motion discriminating image generating means for generating an image signal of the motion discriminating image by extracting an image signal of a predetermined region of the projected image for each frame;
An average image signal for a predetermined frame is sequentially generated using an image signal in which the position of the projected image is shifted for each frame, and an image signal of an integrated image is extracted from the average image signal for each frame. An integrated image generating means for generating a plurality of image signals of the integrated image by changing the shift amount of the projected image, and
The correlation between the image signal of the motion discrimination image generated by the motion discrimination image generation unit and the image signal of the plurality of integration images generated by the integral image generation unit is detected, and the correlation is highest with the predetermined position. the image processing apparatus 請 Motomeko 10, wherein that having a motion amount determination means for determining a motion amount of the predetermined position from the shift amount when the generated image signal of the integrated image that is a. Wherein in the layer classification means, depending on the amount of motion is determined by the movement amount determination means, the image processing apparatus of the layer classification row cormorants請 Motomeko 11 wherein. The peripheral image signal generation means includes
Said peripheral image signal can be output in correspondence with the input image signal, billed that having a control means for controlling the generation of the intermediate image information reading and the peripheral image signal from the intermediate image data storage unit Item 6. The image processing apparatus according to Item 1. The peripheral image signal generation means includes
The image processing apparatus 請 Motomeko 2 wherein that having a picture signal generating means for generating said peripheral image signal an image signal of each of the layers adhered with a predetermined layer order. The peripheral image signal generation means includes
When there is a region without an image in the image signal in which the image signal for each layer is pasted in a predetermined layer order, interpolation processing is performed using the image signal of the peripheral region of the region without the image, and the image signal is the image processing apparatus 請 Motomeko 14, wherein that having a peripheral image interpolation processing means for generating an image signal with no region. The peripheral image signal generation means includes
Layer image generation processing means for moving the pasted image signal for each layer based on the amount of movement for each layer and then pasting a new image signal for each layer and supplying the image signal generation processing means the image processing apparatus 請 Motomeko 14 wherein that Yusuke. The peripheral image signal generation means includes
Projection conversion of the image signal generated by the layer image generation processing means and supply to the image signal generation processing means, or projection output of the image signal generated by the image signal generation processing means the image processing apparatus 請 Motomeko 16, wherein that having a conversion means. Wherein the projective transformation means performs a projective transformation in accordance with the positional relationship between the image display surface of the image based on the image display surface and the surrounding image signal of an image based on the input image signal, that generates said surrounding image signal the image processing apparatus 請 Motomeko 17 wherein. The input image signal is an image processing apparatus of the signal der Ru請 Motomeko 1 wherein accumulated. Using the input image signal of a plurality of frames, the movement of a predetermined position of the frame of interest in the input image signal shall be detected,
Generating intermediate image information based on the detected motion and the input image signal of the plurality of frames;
And based on the generated intermediate image information, images processing method that generates a surrounding image signal of the angle of view that does not exist in the frame of interest. In the computer that processes the image signal,
Using a plurality of frames of input image signals, a motion detection procedure for detecting a motion of a predetermined position of the frame of interest in the input image signals;
Intermediate image information generation procedure for generating intermediate image information based on the motion detected in the motion detection procedure and the input image signal of the plurality of frames;
The intermediate image data based on the generated intermediate image information generation procedure, images processing program Ru to execute the peripheral image signal generation step of generating a peripheral image signal of the angle of view that does not exist in the frame of interest. In the computer that processes the image signal,
Using a plurality of frames of input image signals, a motion detection procedure for detecting a motion of a predetermined position of the frame of interest in the input image signals;
Intermediate image information generation procedure for generating intermediate image information based on the motion detected in the motion detection procedure and the input image signal of the plurality of frames;
The intermediate image data based on the generated intermediate image information generation procedure, the frame of interest to information recorded thereon a program for executing a peripheral image signal generation step of generating a peripheral image signal of the angle of view does not exist recorded Medium.

Images