JP3926575B2 - Imaging device - Google Patents

Description

但し、式（２）において、補間しようとするＲ画素位置の（ｘ、ｙ）座標を（ｍ、ｎ）とし、Ｇ_AVE（ｍ、ｎ）はＧ１とＧ２の平均値を、Ｒ_AVE（ｍ、ｎ）はＲ１とＲ２の平均値を示し、２つの二等辺三角形の線分相似比として１/２を乗じている。
【００７１】
上記例では、Ｒ画素位置においてＧ色信号成分を補間する例を示したが、ＲをＢに置き換えることでＢ画素位置におけるＧ色信号成分の補間を行うことが同様に可能である。
【００７２】
また、上記例は、Ｒ画素位置のＧ成分を色補間する場合を示したが、任意のＪ画素位置におけるＫ色の色補間に同様の考え方を当てはめることが可能である。図１３（ｂ）及び図１３（ｃ）は、他の画素位置における他の色成分補間を行う際の参照色の参照画素位置と補間色の参照画素位置の関係を示したものである。図１３（ｂ）は、Ｇ画素位置におけるＲ成分及びＢ成分補間の場合であり、各色成分について、図１４（ｂ）に示す三角形について上記モデルを適用する。
【００７３】
即ち、これらの３点は三角形を形成するため注目画素以外の２点をそれらの平均値で置きかえることにより二等辺三角形を作成し、これと補間色が相似二等辺三角形を構成するように線分相似比である１/２を乗ずる。さらに、ＲとＢが入れ替わった配置のＧ画素位置におけるＲ成分及びＢ成分の補間も同様に行うことが可能である。
【００７４】
また、Ｂ画素位置におけるＲ成分補間時の参照画素位置の様子を図１３（ｃ）に示す。この場合は、図１４（ｃ）に示すＧ画素位置における撮像Ｇ成分及び注目画素のＢ画素位置における補間済みＧ成分を参照色用三角形を形成する点として用い、Ｇ画素位置における生成済みＲ成分を補間色用三角形を形成する点として用い、各二等辺三角形の線分相似比を１/１とすることで、これまでの例と同様の扱いをすることができる。さらに、ＲとＢを入れ替えて考えることで、Ｒ画素位置におけるＢ成分の補間が可能である。
【００７５】
以上の説明を元に、式（２）を一般化すると、式（３）のように記述することができる。
【数２】

尚、式（３）におけるＫ（ｍ，ｎ）は座標（ｍ，ｎ）における補間すべき非撮像色信号、Ｊは座標（ｍ，ｎ）における参照色としての撮像色信号を示し、ＣｄはＪ色がなす二等辺三角形に対するＫ色がなす二等辺三角形の線分相似比を示している。
【００７６】
このようにして、色補間対象の画素位置を水平及び垂直方向に順次走査しながら各画素における全色成分補間を繰返し行うことにより、１画面分の撮影画素における色補間処理を実行することができる。
【００７７】
このようにして、第一の印加磁界条件下で撮影した第一の撮影画像に対する色補間を補間処理部１１で実行し、第二の画像メモリ１２７に一時記憶する。続いて、第二の印加磁界条件下で撮影した第二の撮影画像を同様に信号レベル補正処理部１０を施した後補間処理部１１において色補間処理を実施し、第二の画像メモリ１２７に一時記憶する。
【００７８】
ここで、第二の画像メモリ１２７に記憶する第一の補間画像と第二の補間画像は、後述する合成処理部１２８における空白画素での画素補間処理のために、図１１のように、１枚の合成画像内の所定の位置を示す物理アドレスに従って格納してもよいが、物理的には個別の画像記憶領域に記憶しておき、次の合成処理部１２８において、図１１の各画素の位置関係を念頭に適宜読み出しを行うようにしてもよい。
【００７９】
次に、２枚の補間画像の１枚の画像への合成処理について説明する。
【００８０】
始めに、空白画素位置におけるＧ成分生成について詳細に説明する。
【００８１】
図１５の網掛け部分は、この段階で画像中に存在するＧ成分の位置関係を示したものである。空白画素位置のＧ成分を補間する場合、周辺画素を参照して画像中のエッジにあたる部分が注目画素を通過するか否かを判定し、その場合はエッジ線分の稜線方向に画素補間を実行することでシャープなエッジ再現が可能である。図１５における矢印の示す方向は、注目画素を中心とする７×７画素からなる参照画素ウィンドウを元に検出可能な線分方向を示している。
【００８２】
線分角度検出は以下の手順に従って実行する。
【００８３】
注目画素及び８近傍画素に存在するＧ成分に対して信号レベル平均値Ｄａｖを算出する。次に、算出した信号レベル平均値Ｄａｖをしきい値として７×７画素ウィンドウ内の各Ｇ画素の信号値を１または０に２値化する。２値化されたウィンドウ内のＧ画素は予め定められている複数のパターンと比較され、パターンの一致により中心画素を通る線分角度及び線分のどちら側に明あるいは暗があるかという線分角度情報が認識される。これらの線分角度情報に基づいて、線分の稜線方向に存在する複数の画素値を線形補間して注目画素のＧ成分が算出される。
【００８４】
例えば、７×７画素ウィンドウ内の画素が図１６の斜線部画素（２値化データ：０）及びたすき線部画素（２値化データ：１）に２値化された場合、パターンマッチングにより図の矢印方向に画像のエッジが通っていることが検出され、図の太枠で示された画素の上下左右の４つの画素値を単純平均あるいは、注目画素からの距離に対する重み付け平均等により注目画素のＧ成分を補間する。
【００８５】
線分の角度を検出し稜線方向に補間することで、一般的な８近傍画素の単純平均値により成分補間する場合に比べ、エッジをシャープかつ線分を滑らかに補間することが可能となる。これにより、画像の観察者に対して高い解像感を与えることが可能となる。
【００８６】
以上の方法で、全画素のＧ成分が高い解像度で生成され、この結果を参照することでＲ及びＢ成分に関しても高解像度に補間生成することができる。
【００８７】
また、空白画素位置におけるＲ及びＢ成分補間に関しては、前記空白画素位置におけるＧ成分補間と同様のパターンマッチングを用いた線分に対する稜線ベクトル検出により補間を行う。この時、ＲＧＢ成分の個別のパターンマッチング結果の相違から生じる誤補間を防止するために、Ｇ成分補間時に検出した線分角度結果に基づきＲ及びＢ成分補間を行うように構成してもよい。
【００８８】
以上のように、撮像素子の２×２倍の画素数を持つ４画素分の高解像度画像が合成処理部１２８により生成され、生成されたフルカラー画像は画像補正処理部１２において光学系及び撮像系の画像歪を補正するノイズ除去フィルタ等の処理が施され、画像圧縮処理部１３によりＪＰＥＧ方式の画像符号化の後、インタフェース部１４に送られる。このようにして、２枚の画素ずらし画像から撮像素子２のナイキスト周波数を超える高解像度の画像が形成される。
【００８９】
本実施の形態１では、画素ずらし手段を用いて−４５度方向に１／２画素ずらして撮影した２枚の画像を用いて高解像度化処理する例を示したが、この限りでない。すなわち、撮像素子２の各受光素子は有限数が面状に配置されており受光素子面における入射光を積分値として信号出力するため、これにより撮影画像の空間周波数は制約（ナイキスト周波数）を受けていることが標本化定理より説明できる。これに対し、「２台のカメラを用いる超高精細画像取得法」（小松、相澤、斎藤：テレビジョン学会誌 Voｌ.４５, Nｏ.１０, pp.１２５６〜１２６２）によると、開口率１００％の撮像素子を用いて画素ずらし画像を統合した場合２倍以内の解像度まで改善可能であることが明示されている。本方式を用いて実際の撮像素子の開口率や受光素子の形状等を考慮してずらし方向、ずらし量及び撮影枚数を最適化することで合成画像における解像度を最適化することができる。
【００９０】
また、本実施の形態１では、合成処理部１２８におけるパターンマッチングの際、７×７画素からなるウィンドウを用いて８方向の線分検出を行っているが、この限りでなく、実行するソフトウェアの高速化あるいは回路規模の低減を図る場合は、ウィンドウサイズを小型化し４５度刻みの４方向あるいは水平及び垂直のみの２方向の画素間のエッジ線分の相関を用いるように構成してもよい。
【００９１】
また、本実施の形態１では、画素ずらし光学系１の撮像条件を変化させた１枚目の画像を撮影し、第一の画像メモリ７に記憶し信号レベル補正処理部１０及び補間処理部１１の処理を終えた後に、第二の印加磁界条件下での２枚目の撮影を開始する構成としたが、この限りでない。即ち、第一の画像メモリ７に２枚分の撮影画像を記憶させる容量を確保しておき、２枚の撮影画像が揃った時点で画像処理部９における画像処理を開始する構成にしても同様な効果を得ることができる。こられは、実際の装置の設計思想によって柔軟に変更することができる。
【００９２】
また、本実施の形態１では、１枚目の撮影画像に対して２枚目の撮影画像を−４５度方向に画素ずらしする例を示したが、この限りでなく、４５度方向、１３５度方向あるいは−１３５度方向にずらしても同一の処理が適用可能であることは言うまでもない。
【００９３】
実施の形態２．
以下、実施の形態２を図について説明する。
【００９４】
図１７は、実施の形態２の全体構成を表すブロック図である。図２において、図１に示す実施の形態１と同一部分は同一符号を付してその説明は省略する。新たな符号として、２０は撮影した各画像データを固定長圧縮方式にて符号化する固定長圧縮回路、２１は撮像部５から出力される撮像デジタル信号のうち固定長圧縮回路２０での圧縮時に使用するライン数分を遅延蓄積するラインバッファ、２２は固定長圧縮回路２０で圧縮された画像データを蓄積する圧縮画像メモリ、２３は固定長伸張処理部である。
【００９５】
次に、動作について説明する。画素ずらし機構を含む撮像部５の動作については実施の形態１と同様である。画素ずらし光学系１を用いて撮像部５から入力された撮影画像は、固定長圧縮回路２０で画像圧縮するためにラインバッファ２１に一時記憶される。以下に、固定長圧縮回路２０における圧縮動作について詳細に説明する。
【００９６】
固定長符号化手段２０における固定長符号化アルゴリズムとしては、例えば４×４画素を１単位とするブロック毎に隣接画素情報の冗長性を排除する符号化法を行うものを用いる。この場合、図２の画素配列では隣接する画素位置に同じ色の画素が存在しないため、固定長圧縮回路２０内において同色画素をブロック化する。
【００９７】
図１８は、固定長圧縮回路２０により図２の画素配列が並べ替えられる様子を示した図である。撮影画像データは撮像部５から順次ラインバッファ２１に蓄積され、未処理のデータが８ライン分揃った段階で画素配列の並べ替えを始めに、固定長圧縮回路２０における符号化が実施される。画素並べ替えは、８×８画素単位に行い各色４×４画素を単位ブロックとしてブロック化する。並べ替え後の画像信号は、ブロックの左上方向のアドレスと右下方向のアドレスにＧ成分が集められて配列され、左下方向のアドレスにＢ成分、右上方向のアドレスにＲ成分が集められて配列される。
【００９８】
また、固定長圧縮回路２０における画像符号化方法について説明する。
【００９９】
図１９は、前記並べ替えられた単位ブロック内のカラー画像信号を固定長圧縮回路２０で符号化する場合のアドレスを示す図である。すなわち、図１９は、縦横４画素分ずつの同一カラーの画像信号の塊である単位ブロックのうち縦方向ｍ行横方向ｎ列（ｍ、ｎは０＜ｍ、ｎ≦４の自然数）のアドレスの画像信号に後述する量子化レベルを付すことを表している。
【０１００】
図２０は、各画素の画像信号の強度を階層化（量子化）する量子化レベルを表す。図２０において、Ｌｍｉｎは図１９に示した１６画素の画像信号強度中の最小値、Ｌｍａｘは同じ１６画素の画像信号強度中の最大値、Ｐ１は最大値Ｌｍａｘと最小値Ｌｍｉｎとの間を８等分した下から８分の１の値、Ｐ２は上から８分の１の値、Ｑ１はＬｍｉｎ以上Ｐ１以下の信号強度を有する画素の平均値、Ｑ８はＬｍａｘ以下Ｐ２より大きい信号強度の画素の平均値である。
【０１０１】
また、ＬＤは単位ブロック内の階調幅指標で、Ｑ８−Ｑ１に等しい。Ｌ１〜Ｌ７は階調幅指標ＬＤを８等分した値を小さいほうから並べたものである。ＬＡは単位ブロック内の画像データ平均値レベルで（Ｑ１＋Ｑ８）／２に等しい。φｉｊｋは画素毎の量子化レベルを表す。
【０１０２】
図２１及び図２２は、この実施の形態による符号化手順を示すフローチャートである。以下、このフローチャートを参照しながら符号化手順を説明する。
【０１０３】
まず、固定長圧縮回路２０は、図１８の（ｂ）のように並べ替えられた単位ブロック内の画像データを読み込む（ステップＳＴ１）。次に、読み込んだ４×４画素分の画像データの信号強度を演算し、以下の各式に従って順次Ｐ１、Ｐ２、Ｑ１、Ｑ８、ＬＡ、ＬＤ、Ｌ１〜Ｌ７の値を求める（ステップＳＴ２〜ステップＳＴ１３）。
【０１０４】
Ｐ１＝（Ｌｍａｘ＋７Ｌｍｉｎ）／８
Ｐ２＝（７Ｌｍａｘ＋Ｌｍｉｎ）／８
Ｑ１＝Ａｖｅ（Ｘｍｎ≦Ｐ１）
Ｑ２＝Ａｖｅ（Ｘｍｎ＞Ｐ２）
ＬＡ＝（Ｑ１＋Ｑ８）／２
ＬＤ＝Ｑ８−Ｑ１
Ｌ１＝ＬＡ−３ＬＤ／８
Ｌ２＝ＬＡ−ＬＤ／４
Ｌ３＝ＬＡ−ＬＤ／８
Ｌ５＝ＬＡ＋ＬＤ／８
Ｌ６＝ＬＡ＋ＬＤ／４
Ｌ７＝ＬＡ＋３ＬＤ／８
【０１０５】
なお、Ｑ１の式はＬｍｉｎ以上Ｐ１以下の信号強度を持つ画素の平均値を求めることを意味し、Ｑ８の式はＬｍａｘ以下Ｐ２より大きい信号強度を持つ画素の平均値を求めることを意味する。
【０１０６】
このようにして、順次Ｐ１、Ｐ２、Ｑ１、Ｑ８、ＬＡ、ＬＤ、Ｌ１〜Ｌ７の値を求めた後、固定長圧縮回路２０はｎ＝１、ｍ＝１とおいて（ステップＳＴ１４、１５）、この時のアドレス（ｍ，ｎ）の画素の信号強度（以後、画素値と称する）Ｘｍｎ（すなわち画素値Ｘ１１）がＬ１以下であるか否かを判断する（ステップＳＴ１６）。
【０１０７】
画素値Ｘ１１がＬ１以下である場合には、この画素値の量子化レベルφｉｊｋを２進数の０００と設定する（ステップＳＴ１７）。次に、ｍを１だけインクリメントし（ステップＳＴ３１）、ｍが４以下であるか否かを判別する（ステップＳＴ３２）。ｍが４以下である場合は、その画素の画素値を再びＬ１と比較する（ステップＳＴ１６）。
【０１０８】
ｍが４より大きい場合には、ｎを１だけインクリメントし（ステップＳＴ３３）、インクリメントしたｎが４以下であるか否かを判別する（ステップＳＴ３４）。ｎが４以下である場合は、その画素の画素値を再びＬ１と比較する（ステップＳＴ１６）。
【０１０９】
画素値ＸｍｎがＬ１より大きい場合には、Ｌ２以下であるか否かを判別し（ステップＳＴ１８）、が措置ＸｍｎがＬ２以下である場合は、この画素の量子化レベルφｉｊｋを２進数の００１と設定する（ステップＳＴ１９）。次に、ｍを１だけインクリメントし（ステップＳＴ３１）、ｍが４以下であるか否かを判別する（ステップＳＴ３２）。ｍが４以下である場合は、その画素の画素値を再びＬ１と比較する（ステップＳＴ１６）。ｍが４より大きい場合には、ｎを１だけインクリメントし（ステップＳＴ３３）、インクリメントしたｎが４以下であるか否かを判別する（ステップＳＴ３４）。ｎが４以下である場合には、その画素の画素値を再びＬ１と比較する（ステップＳＴ１６）。
【０１１０】
以下、同様に、画素値がＬ１〜Ｌ２間、Ｌ２〜Ｌ３間、Ｌ３〜ＬＡ間、ＬＡ〜Ｌ５間、Ｌ５〜Ｌ６間、Ｌ６〜Ｌ７間のいずれの値を有するかを判別し（ステップＳＴ１６、ＳＴ１８、ＳＴ２０、ＳＴ２２、ＳＴ２４、ＳＴ２６、ＳＴ２８）、その値に応じてそれぞれ量子化レベルφｉｊｋ＝０００、００１、０１０、０１１、１００、１０１、１１０、１１１を当該画素に割り振る（ステップＳＴ１７、ＳＴ１９、ＳＴ２１、ＳＴ２３、ＳＴ２５、ＳＴ２７、ＳＴ２９）。
【０１１１】
このようにして、同一単位ブロック内の全画素に量子化レベルを割り振って符号化を終了する。単位ブロックの符号化データはＬＡ、ＬＤ、各画素毎のφｉｊｋである。
【０１１２】
これらの処理は、画面全体に対して単位ブロック数分繰り返す。Ａ／Ｄ変換回路４からのデジタル信号が１画素あたり１０ビットである場合（ビット詰めしない場合２バイト相当）、固定長符号化による画像圧縮率は１０／３２である。
このように各撮影画像データは、逐次同色画素の画素のみで構成されたブロック毎に固定長圧縮回路２０において固定長符号化が行われ、画像データ量の低減が施された上でメモリ制御部８を介して圧縮画像メモリ２２に蓄積される。
【０１１３】
こうして２枚の撮影画像データが圧縮画像メモリ２２に蓄積された後に、画像処理部９における画像処理が施される。始めに、固定長圧縮回路２０において符号化されている画像データの復号化処理を行う。固定長伸長処理部２３における画像復号化方法について図について説明する。図２３は、固定長伸長処理部２３の動作を示すフローチャートである。以下、このフローチャートを参照しながら固定長伸長処理部２３の固定長復号化動作を説明する。
【０１１４】
固定長復号化動作が開始されると、まず、縦方向座標値ｎを１に設定し（ステップＳＴ４０）、に横方向座標値ｍを１に設定する（ステップＳＴ４１）。すなわち、ステップＳＴ４０及びステップＳＴ４１の動作によってある単位ブロック中の座標値（１，１）のアドレスが指定される。
【０１１５】
次に、指定したアドレスの量子化レベルΦｉｊｋがいくつであるか判定し（ステップＳＴ４２、ＳＴ４４、ＳＴ４６、ＳＴ４８、ＳＴ５０、ＳＴ５２、ＳＴ５４）、判定した各量子化レベルΦｉｊｋに応じて平均値レベルＬＡと階調幅指標ＬＤとに基づいてその画素の信号強度Ｙｍｎ（座標値（１，１）の画素であるならばＹ１１）を求める（ステップＳＴ４３、ＳＴ４５、ＳＴ４７、ＳＴ４９、ＳＴ５１、ＳＴ５３、ＳＴ５５、ＳＴ５６）。
【０１１６】
各ステップにおいて平均値レベルＬＡと階調幅指標値ＬＤとから信号強度Ｙｍｎを求めるには、それぞれ次の演算式に従う。
【０１１７】
Ｙｍｎ＝ＬＡ−ＬＤ／２ （ステップＳＴ４３）
Ｙｍｎ＝ＬＡ−５ＬＤ／１４ （ステップＳＴ４５）
Ｙｍｎ＝ＬＡ−３ＬＤ／１４ （ステップＳＴ４７）
Ｙｍｎ＝ＬＡ−ＬＤ／１４ （ステップＳＴ４９）
Ｙｍｎ＝ＬＡ＋ＬＤ／１４ （ステップＳＴ５１）
Ｙｍｎ＝ＬＡ＋３ＬＤ／１４ （ステップＳＴ５３）
Ｙｍｎ＝ＬＡ＋５ＬＤ／１４ （ステップＳＴ５５）
Ｙｍｎ＝ＬＡ＋ＬＤ／２ （ステップＳＴ５６）
画素（１，１）の信号強度を求めたら、次に横方向に画素を１つ移動し（ステップＳＴ５７、ＳＴ５８）、同一の手順で画素（２，１）の信号強度を復号化する（ステップＳＴ４２〜ＳＴ５６）。
【０１１８】
このようにして、単位ブロック内の最上段の画素について信号強度を復号化した後（ステップＳＴ５８）、縦方向の座標値を１だけインクリメントし（ステップＳＴ５９）、次の段の画素について同様にして信号強度を復号化する（ステップＳＴ４２〜ＳＴ５８）。
【０１１９】
このようにして単位ブロック内の全画素について信号強度を復号化して（ステップＳＴ４１〜ＳＴ６０）復号化動作を終了する。
【０１２０】
次に、復号化されたデータに、図２４に示すように固定長圧縮回路２０で並べ替えたデータの逆の並べ替え処理を施して画素信号を走査線方向に読み出したときと同一の順序で並べ替え、第一の画像メモリ７に記憶される。
【０１２１】
以上のように、第一の画像メモリ７上には固定長圧縮前の画像データと等価な画像データが蓄積されていることになり、実施の形態１と同様の画像処理が画像処理部９の他のブロックにおいて施され、インタフェース部１４に出力される。
【０１２２】
以上のように、複数枚の画像を撮影する際に、各々の画像データを固定長圧縮回路２０を用いてデータ圧縮することにより、撮像部５のＡ／Ｄ変換手段３から出力される画像信号を第一の画像メモリ７に蓄積する際、より少ないデータ量をデータバスに供給し記憶手段に蓄積することが可能となり、連写間隔を短縮するための高速なメモリ制御回路あるいは最低１フレーム分の高速書込みが可能な半導体メモリを使用する必要がなくなり、装置を低価格に構成することが可能となる。
【０１２３】
これは言い換えれば、少量の高速な圧縮画像メモリ２２を使用することで記憶手段への撮影したデータ蓄積速度を見かけ上高速化することが可能であり、１枚目の撮影シーケンスが短時間に終了することで２枚目の撮影動作を短時間の間合いで開始することができることになる。そのために、２枚の撮影画像間に生じる被写体の時間的な変化あるいは撮影者の手ぶれ等を最小限に抑えることが可能となり、同一被写体を高精度に１／２画素ずらした撮影が実現できるという効果がある。
【０１２４】
実施の形態２では、第一の画像メモリ７と圧縮画像メモリ２１は別個に設ける構成としたが、この限りでなく、共通の物理アドレス領域を有する半導体メモリ上に併設する構成としてもよい。また、これらのメモリと第二の画像メモリ１２７についても同一の半導体メモリ上に併設可能であることは言うまでもない。
【０１２５】
また、実施の形態２では、固定長圧縮回路２０と固定長圧縮回路２０に用いるラインバッファ２２及び圧縮画像メモリ２１を別個に設ける構成としたが、この限りでなく、これらの構成要素の２つ以上をＬＳＩ等の集積回路１石に包含する構成としてもよい。
【０１２６】
実施の形態３．
以下、実施の形態３を図について説明する。画素ずらし手段を含む撮像部５の構成及び動作、画像処理部９の構成に関しては、実施の形態１及び２と同様に、図１あるいは図１７のように構成される。実施の形態３で特徴的である画像処理部９の信号レベル補正処理部１０の動作について詳細な説明を行う。
【０１２７】
撮像部５で撮影された２枚の画像信号は、Ａ／Ｄ変換回路３を経て画像処理部９の第一の画像メモリ７に蓄積される。蓄積された画像データは、各々信号レベル補正処理部１０で白バランス補正処理やガンマ補正処理等の個別に適用可能な信号補正処理を施される。この時、１枚目の画像と２枚目の画像は時系列的に撮影されているため、例えば５０〜６０ＭＨｚの比較的低周波で発光を繰り返す蛍光灯を光源として撮影した場合、光源の発光タイミングとシャッタ時間の関係で２枚の画像の入射光量に差異が生じる場合がある。即ち、撮影した２枚の画像の平均明度に差が生じ、後段の合成処理部１２８で補間画像の合成処理を行う際に画質劣化の原因となりうる。
【０１２８】
図２５（ａ）は１枚目の撮像条件に比して２枚目の撮像条件における照明光量が３０％程度低下した場合の同一被写体を撮影した際の明度特性を示している。定照明下で撮影した際の各信号レベルは、照明条件が変化した場合に平均明度の低下とダイナミックレンジの縮小が発生する。図中、Ｍａｘ及びＭｉｎ値は各画像における最大信号レベル及び最小信号レベルである。実施の形態３における信号レベル補正処理部１０では、各画像内の信号レベル補正処理の他、２枚の画像の信号レベル補正処理を行う。
【０１２９】
即ち、各画像における最大信号レベル値及び最小信号レベル値を検出し、いずれか一方の信号レベル特性に他方を合せ込む処理を行う。通常は、１枚目の画像撮影時に自動露出補正あるいはシャッタ速度最適化を行い最適条件下で撮影を行うため、２枚目の画像を１枚目の明度分布特性に合わせるようにする。
【０１３０】
補正前後の２枚目の画像における各信号レベルをそれぞれＤ（ｊ）（ｊは定照明下での各明度レベル値）及びＤ’（ｊ）とすると、始めに式（４）に従って補正を行う（図２５（ｂ）参照）。
【０１３１】
Ｄ’（ｊ）＝Ｄ（ｊ）＋（Ｍｉｎ１−Ｍｉｎ２） （４）
【０１３２】
続いて、式（５）に従って１枚目の画像及び式（４）にて補正した２枚目の画像のダイナミックレンジを補正し最終的な２枚目の補正信号レベルＤ’’（ｊ）を算出する（図２５（ｃ）参照）。
【数３】

以上の手順で信号レベル補正処理部１０を構成することにより、撮影した２枚の画像の照明条件が異なる場合においても明度特性を良好に一致させることが可能であり、高精度の画素補間・合成処理が実現可能である。
【０１３３】
また、撮像装置を図１７（実施の形態２）に示す構成にし、撮影画像を固定長圧縮回路２０を用いて符号化する場合、各符号におけるＬＡは１６画素からなるブロック内の平均信号レベルを代表する値であることから、上記手順を全てＬＡを対象に実施しても等価的に同等の効果を実現可能であることは容易に説明できる。このように構成した場合、上記全画素を対象に行っていた最大・最小信号レベル検出及び信号レベル補正に関する演算規模を１／１６に抑制することが可能であり、処理の高速化を実現できるという格別の効果がある。
【０１３４】
実施の形態３では、２枚目の撮影画像に対して式（４）及び式（５）を順次適用するように構成したが、この限りでなく、これらを１回の画像データ走査で行えることは言うまでもない。
【０１３５】
実施の形態４．
以下、実施の形態４を図について説明する。実施の形態４における全体構成は実施の形態１における図１あるいは実施の形態２における図１７の構成と同様である。本実施の形態４では、画素ずらし光学系１として、電界の強さに応じて透過光の屈折現象が変化する電気光学素子を用いる点が、実施の形態１及び実施の形態２と異なる。
【０１３６】
図２６及び図２７は、本実施の形態４における撮像部５の構成を示したものである。これら図において、３０は電圧印加により電界を発生する電界発生回路、３１は電気光学素子である。
【０１３７】
次に動作について説明する。図２６の撮像部５について構成及び動作を説明する。図２６の動作原理図である図２７中の電気光学素子３１及び偏光板１８をＡ−Ａ、Ｂ−Ｂ、Ｃ−Ｃの方向から見たものをそれぞれ図２８、図２９、図３０に示す。
【０１３８】
図２７において、画像を結像するためのレンズ６１から電気光学素子３１に入射する自然光は、図２８に示すように、互いに直交する２つの偏光成分Ｌ_X、Ｌ_Yで表すことができる。電気光学素子３１に電界が印加されていない場合は、図２９のＬ_X0、Ｌ_Y0に示す偏光成分が前記Ｌ_X、Ｌ_Yが直進した位置に観測される。電気光学素子３１に所定の電界Ｅ１が印加されると、該電気光学素子３１は複屈折現象を呈し、図２９のＬ_X1、Ｌ_Y1に示す偏光成分が観測される。
【０１３９】
ここで、Ｌ_X1は複屈折現象における常光線としてＬXが直進した位置に、Ｌ_Y1は複屈折現象の異常光線としてＬ_Yが距離Ｐだけずれ対置に観測される。ただし、この場合、図３０に示すようなＰ＝ＰＨ／２（ＰＨは撮像素子２の水平画素ピッチを示す）となるような電界Ｅ１を電界発生回路３０により電気光学素子３１に印加する。
【０１４０】
上記撮像装置における動作タイミングを図８により説明すると、図８（ａ）はフィールドシフトパルス、図８（ｂ）は電気光学素子３１に印加される電界Ｅ１の強さを示す図、図８（ｃ）は信号読み出しパルス、図８（ｄ）は撮像素子２の出力信号を示す図、図８（ｅ）はＡ、Ｂフィールドを１フレームとして見た場合の図である。電界Ｅ１の強さの変化の位相を図８（ａ）のフィールドシフトパルスに一致させる。
【０１４１】
上述の動作により、この実施の形態４の撮像装置は、 Ａ、Ｂフィールドでの信号電荷蓄積を入射画像と撮像素子２の画素との相対的な位置に関してＰＨ／２だけ離れた位置で行うことができる。これに対応して図８（ｃ）に示す信号読み出しパルスのタイミングもＰＨ／２に相当する時間Ｔだけずらしてある。その結果、図８（ｅ）に示すように、この撮像装置はＡ、Ｂフィールドを１フレームとした１周期で高画素密度化した画像を得ることができる。
【０１４２】
実施の形態４における画像処理部９の構成及び動作はこれまでの実施の形態で説明したいずれを用いても構成できる。
【０１４３】
実施の形態５．
以下、実施の形態５を図について説明する。実施の形態５における全体構成は実施の形態１における図１あるいは実施の形態２における図１９と同様である。本実施の形態５では、画素ずらし光学系１として直線偏光を作る偏光板及び入射光の方位角を変更する液晶板を用いた場合を例にとって説明する。
【０１４４】
図３１は、本実施の形態５における撮像部５の構成を示したものである。
【０１４５】
図３１において、１８は入射光に対して直線偏光を生成する偏光板、９２は偏光板１８で生成された直線偏光を電圧の印加条件で方位角が変更可能な液晶板、９１は液晶板９２への印加電圧を制御する駆動電圧制御回路である。
【０１４６】
次に、動作について説明する。画素ずらし光学系１および撮像素子２の拡大図を図３２に示す。図３２において、Ａ−Ａは偏光板１８の入射側、Ｂ−Ｂは偏光板１８の出射側、Ｃ−Ｃは液晶板９２の出射側、Ｄ−Ｄは複屈折板１９の出射側の垂直方向の位置を示している。図３３〜図３６には図３２に示した前記Ａ−ＡからＤ−Ｄまでの各位置における光の偏光方向を示す。
【０１４７】
被写体を撮像する場合、光学系に入射される光は無偏光であるため、図３３のように、垂直方向の偏光成分と水平方向の偏光成分とで示すことができる。偏光板１８は入射された光から直線偏光した光を出射する。また、その方位角は、図３４に示すように、撮像素子２の水平方向の画素配列を基準角とすると、基準角に対して４５°の角度となるように、偏光板１８の偏光方向の偏光軸を傾けて配置する。
【０１４８】
液晶板９２は、内部に液晶層を有し、２枚の電極基板で液晶を挟んだ板であり、外部から電圧すなわち電界が加わることにより、内部の液晶の分子配列変化に伴う電気光学効果を引き起こす。この発明にて用いる液晶板９２は、入射される直線偏光の光の偏光方向を、液晶板９２に電圧を加える、または加えている電圧を切ることにより、変化させるものである。上記液晶板９２を実現させるための一例として前記電気光学効果の内、代表的な電界効果型である、ねじれネマチック（ＴＮ）型液晶による旋光効果などが挙げられる。その動作を図３７および図３８に示す。
【０１４９】
ＴＮモードを示すネマチック液晶は正の誘電異方性をもち、素子の分子配列を図３７、３８に示す。図３７，３８中、９３が液晶分子である。基板に電圧が印加されたときは、図３７に示すように分子長軸方向が両基板面で直行しているように配列するため、偏光板１９を介して入射された直線偏光はそのまま出射される。電圧が無印加の時は素子内の液晶分子の配列は連続的に９０°ねじれており、光学的には９０°旋光効果を起こすため、偏光板１９から入射された直線偏光は９０°ねじれて、出射光は入射光に対して９０°偏光方向が異なるようになる。
【０１５０】
液晶板９２から出射された光の偏光を図３５に示す。図３５において、ｃ１は前記液晶板９２において電圧が印加された状態であり、電圧無印加時にはｃ２のように基準角度に対して１３５°の方位角を有した直線偏光となる。今、ｃ１の直線偏光を第１の偏位方向、ｃ２の直線偏光を第２の偏位方向とする。液晶板９２から出射された光は次に複屈折板１９に入射される。
【０１５１】
複屈折板１９は屈折率が偏光方向によって均一でない、いわゆる複屈折を有する材質であり、撮像装置では被写体の空間周波数が撮像素子２の画素ピッチから求められるサンプリング周波数を超えている場合、追い返しノイズ（エイリアジングノイズ）として画像に現れるため、光学的ローパスフィルタとして水晶板などが良く用いられている。水晶板すなわち複屈折板１９は入射する光を常光線と異常光線とに分離し、その分離の距離間は複屈折板１９の厚みによって調整することが可能である。
【０１５２】
第１の偏位方向の入射光を常光線、第２の偏位方向の入射光を異常光線とするように複屈折板を設けると、図４に示すように、異常光線は点線で示されるように、その光軸は常光線と異とし、それぞれの偏位方向を有する入射光は図３６中ｄ１，ｄ２に示すように撮像素子２上の異なった位置に結像する。また、前記複屈折板１９の厚みを図３６に示すように水平方向の撮像素子の画素ピッチｗｘの半分であるｗｘ／２、垂直方向の撮像素子の画素ピッチｗｙの半分であるｗｙ／２だけ異常光線が常光線に対して異なるように設ける。
【０１５３】
よって、前記第１の偏位方向の時に撮像素子２上に結像した像を撮像した第１の画像と、第２の偏位方向の時に撮像した第２の画像の２枚の画像は互いに水平垂直とも１／２画素分、相対的にずれている。
【０１５４】
以上のことから、撮像装置は、撮像部制御回路４からの制御により撮像素子２を駆動させる際に駆動電圧制御回路９１から液晶板９２へ駆動電圧を出力する。液晶板９２は電圧が加わることにより図３９に示す液晶分子配列となり、第１の偏位方向の入射光は撮像素子２上に結像される。撮像素子２は撮像部制御回路４の制御により駆動電圧制御回路９１からの駆動信号により撮像を行い第１の画像を得る。
【０１５５】
次に、第１の画像を撮像した後、駆動電圧制御回路９１は液晶板９２への印加電圧を０とする。液晶板９２の液晶分子配列は、図３８に示したようになり、第２の偏位方向の入射光は撮像素子２上に結像される。撮像素子２は駆動電圧制御回路９１からの駆動信号により撮像を行い第２の画像を得る。
【０１５６】
液晶板９２への印加電圧パルスを図３９に示す。撮像部制御回路４は図３９における状態１の間に第１の画像の撮像を終了させ、同図状態２の間に第２の画像の撮像を終了させる。
【０１５７】
上記のように動作させることにより、水平垂直とも１／２画素ずれた第１の画像および第２の画像を得ることができる。
【０１５８】
以上のように得られた１枚目の撮影画像と１／２画素ずらして撮影された２枚目の画像は、逐次Ａ／Ｄ変換手段３においてデジタル信号に変換された後、画像処理部９に送信される。
【０１５９】
実施の形態５では、１枚目の画像に対して２枚目の撮影画像を１／２画素ずらす方向として４５度方向にずらすものを例に説明したがこの限りでなく、液晶９２及び複屈折板１９の配置あるいは液晶９２への印加電圧を適宜組み合わせることにより１３５度方向、−４５度方向、−１３５度方向など実現可能である。
【０１６０】
また、実施の形態５では、入射する直線偏光の偏光方向を変化させる液晶板９２として、ＴＮ型液晶を一例として挙げたが、光電効果により旋光効果を有する液晶板９２であれば、同様の効果を有することができる。
【０１６１】
なお、実施の形態１において、図３に示した画素ずらし光学系１では、レンズ６１は、偏光板１８、磁気光学素子１７、複屈折板１９からなる画素ずらしを行うための手段よりも被写体寄りの位置に配置されているが、この限りでない。即ち、レンズ６１をこれらの手段の後段に配置することで同様の画素ずらし効果が得られると共に、画素ずらしのための手段を着脱可能な構成にすることにより、通常の撮像装置でありながら画素ずらしのための手段を装着することで高精細な画像を撮像可能な撮像装置として実現できるという格別な効果が生じる。
【０１６２】
また、実施の形態４では、図２６に示した画素ずらし光学系１では、レンズ６１は、偏光板１８と電気光学素子３１からなる画素ずらしを行うための手段よりも被写体寄りの位置に配置されているが、この限りでない。即ち、レンズ６１をこれらの手段の後段に配置することで同様の画素ずらし効果が得られると共に、画素ずらしのための手段を着脱可能な構成にすることにより、通常の撮像装置でありながら画素ずらしのための手段を装着することで高精細な画像を撮像可能な撮像装置として実現できるという格別な効果が生じる。
【０１６３】
また、以上の全ての実施の形態においては、静止画像を連続的に取り込み可能なディジタルスチルカメラでの構成例を示したが、動画像を取り込み可能なディジタルビデオレコーダにおける高精細静止画撮影モードなどとして構成可能であることは言うまでもない。
【０１６４】
また、以上の全ての実施の形態においては、撮像装置の内部で色補間処理及び合成処理を実行可能な構成例を示したがこの限りでなく、パーソナルコンピュータやカラープリンタ等、撮像装置に直接あるいは記憶媒体を間接的に経由して接続可能な機器上で構成してもよい。
【０１６５】
【発明の効果】
以上のように、この発明によれば、撮像素子に対する入射光の位置を半画素分だけ光学的に変位させる画素ずらし手段と、上記画素ずらし手段による変位前後の撮像素子の撮像画像を記憶する第一の画像記憶手段と、上記第一の画像記憶手段に記憶されている撮像画像を構成する撮像色信号のうち、参照色である撮像色信号について類似性が高いと判断された 1 次元方向に配列する３点の画素座標を横軸にこれら３点の信号レベル値を縦軸にした平面上に形成される三角形と、補間色である非撮像色信号について参照色と同一方向である１次元方向に配列する２点と補間生成しようとする画素の画素座標を横軸にこれら３点の信号レベル値を縦軸にした平面上に形成される三角形の相似比を考慮して色補間を行う補間手段と、変位前後の撮像画像を上記補間手段で各々色補間した結果を記憶する第二の画像記憶手段と、変位前後の撮像素子の撮像画像を合成する合成手段とを備えたので、画素数の増加量に対して十分に解像度が向上した画像を実現することが可能な撮像装置を得ることができる。
【０１６６】
また、上記合成手段は、撮像画素が存在しない位置の空白画素を画素補間する場合、画像中のエッジに当たる部分が空白画素を通過するか否かを判定し、その空白画素を通過する時にはエッジ成分の稜線方向に当該空白画素を画素補間するようにしたので、エッジ線分の稜線方向に画素補間を実行することでシャープなエッジ再現が可能となる。
【０１６７】
また、撮像素子の撮像画像を圧縮して、その圧縮画像を圧縮画像メモリに格納し、その圧縮画像を伸張して上記第一の画像記憶手段に出力する圧縮・伸張手段を設けたので、複数枚の画像を撮影する際に、各々の画像データを圧縮することにより、撮像部の画像信号を第一の画像メモリに蓄積する際、より少ないデータ量をデータバスに供給し記憶手段に蓄積することが可能となり、連写間隔を短縮するための高速なメモリ制御回路あるいは最低１フレーム分の高速書込みが可能な半導体メモリを使用する必要がなくなり、装置を低価格に構成することが可能となる。
【０１６８】
また、上記第一の画像記憶手段に記憶されている変位前後の撮像素子の撮像画像を構成する撮像色信号の信号レベルを補正する信号レベル補正手段を設けたので、ＲＧＢ各色に対応して色フィルタの感度特性を良好なものにすることができる。
【０１６９】
また、上記信号レベル補正手段は、変位前後の撮像素子の撮像画像における所定領域の撮像色信号の平均信号レベルを計算し、双方の平均信号レベルが一致するように撮像色信号の信号レベルを補正するようにしたので、撮像した２枚の画素の照明条件が異なる場合においても明度特性を良好に一致させることが可能であり、高精度の画素補間・合成処理が実現できる。
【０１７０】
また、上記画素ずらし手段は、入射光を撮像素子に導く光路に配置された磁気光学素子を備え、当該磁気光学素子に与える磁界の強度を制御して、撮像素子に対する入射光の位置を変位させるので、高画素密度化画像を得ることができる。
【０１７１】
また、上記画素ずらし手段は、入射光を撮像素子に導く光路に配置された電気光学素子を備え、当該電気光学素子に与える電界の強度を制御して、撮像素子に対する入射光の位置を変位させるので、高画素密度化画像を得ることができる。
【０１７２】
また、上記画素ずらし手段は、入射光を撮像素子に導く光路に配置された液晶板を備え、当該液晶板に与える電圧の強度を制御して、撮像素子に対する入射光の位置を変位させるので、高画素密度化画像を得ることができる。
【０１７３】
また、撮像光学系と撮像素子の間に上記画素ずらし手段を配置することで、画素ずらし手段を着脱可能な構成にすることにより通常の撮像装置でありながら画素ずらし手段を装着することで高精細度な画像を撮像可能な撮像装置として実現できる。
を特徴とするものである。
【０１７４】
さらに、撮像光学系の前段に上記画素ずらし手段を配置することで、画素ずらし手段を着脱可能な構成にすることにより通常の撮像装置でありながら画素ずらし手段を装着することで高精細度な画像を撮像可能な撮像装置として実現できる。
【図面の簡単な説明】
【図１】 この発明の実施の形態１を示す全体ブロック図である。
【図２】 Bayer配列をした撮像素子の説明図である。
【図３】 図１における撮像部５の詳細構成を示した図である。
【図４】 図１における撮像部５の構成及び動作説明のための図である。
【図５】 偏光板１８、ファラデー素子１７、複屈折板１９を、図４のＡ−Ａ線から見た図である。
【図６】 偏光板１８、ファラデー素子１７、複屈折板１９を、図４のＢ−Ｂ線から見た図である。
【図７】 偏光板１８、ファラデー素子１７、複屈折板１９を、図４のＣ−Ｃ線から見た図である。
【図８】 撮像装置における動作タイミングの説明図である。
【図９】 撮像原理により、図２のBayer型単板カラー撮像素子を用いて、画素ずらし画像を撮影する説明図である。
【図１０】 補間処理部１１及び合成処理部１２８の処理手順を示したフローチャートである。
【図１１】 画素数が縦横とも２倍である１枚の合成画像を構成する画素の配置説明図である。
【図１２】 図１１で重心の位置関係に対応した、補間処理部１１の処理実行前の既存画像信号成分とその位置関係についての説明図である。
【図１３】 撮像色信号がＲ色である画素の位置（以下、Ｒ画素位置と称す）の非撮像色信号のうちＧ色信号を補間生成する場合の説明図である。
【図１４】 図１３の場合で水平方向の類似性が高いと判断されて水平方向成分の参照色信号を用いて補間色信号の色補間を行う場合の説明図である。
【図１５】 画像中に存在するＧ成分の位置関係を示ス説明図である。
【図１６】 線分角度情報に基づいて線分の稜線方向に存在する複数の画素値を線形補間して注目画素のＧ成分が算出される場合の説明図である。
【図１７】 この発明の実施の形態２の全体構成を表すブロック図である。
【図１８】 固定長圧縮回路２０により図２の画素配列が並べ替えられる様子を示した図である。
【図１９】 並べ替えられた単位ブロック内のカラー画像信号を固定長圧縮回路２０で符号化する場合のアドレスを示す図である。
【図２０】 各画素の画像信号の強度を階層化（量子化）する量子化レベルを表す図である。
【図２１】 実施の形態２による符号化手順を示すフローチャートである。
【図２２】 実施の形態２による符号化手順を示すフローチャートである。
【図２３】 固定長伸長処理部２３の動作を示すフローチャートである。
【図２４】 固定長圧縮回路２０で並べ替えたデータの逆の並べ替え処理を施して画素信号を走査線方向に読み出したときと同一の順序で並べ替え、第一の画像メモリ７に記憶される場合の説明図である。
【図２５】 定照明下での被写体明度の説明図である。
【図２６】 この発明の実施の形態４における撮像部５の構成を示した図である。
【図２７】 図２６の動作原理図である。
【図２８】 図２７中の電気光学素子３１及び偏光板１８をＡ−Ａの方向から見た図である。
【図２９】 図２７中の電気光学素子３１及び偏光板１８をＢ−Ｂの方向から見た図である。
【図３０】 図２７中の電気光学素子３１及び偏光板１８をＣ−Ｃの方向から見た図である。
【図３１】 この発明の本実施の形態５における撮像部５の構成を示した図である。
【図３２】 画素ずらし光学系１および撮像素子２の拡大図である。
【図３３】 図３２に示したＡ−Ａの各位置における光の偏光方向を示す図である。
【図３４】 図３２に示したＢ−Ｂの各位置における光の偏光方向を示す図である。
【図３５】 図３２に示したＣ−Ｃの各位置における光の偏光方向を示す図である。
【図３６】 図３２に示したＤ−Ｄの各位置における光の偏光方向を示す図である。
【図３７】 液晶板９２を実現させるための一例として前記電気光学効果の内、代表的な電界効果型である、ねじれネマチック（ＴＮ）型液晶による旋光効果の説明図である。
【図３８】 図３７と関連する旋光効果の説明図である。
【図３９】 図液晶分子配列の説明図である。
【図４０】 特開平７−２３６０８６号公報に示された光学的に画素数を増加させる従来の画像入力装置のブロック図である。
【図４１】 撮像部の機構を示す斜視図である。
【図４２】 撮像によって得られた前後左右の信号の平均値からＧ１’，Ｇ２’を補間する際の説明図である。
【図４３】 補間して得られたＧ信号からさらに内挿することにより全画素分のＧ信号を得る場合の説明図である。
【図４４】 左右のＢ１信号からＢ１’信号を内挿し、さらに、補間したＢ１’信号からＢ１”信号を内挿する場合の説明図である。
【図４５】 補間したＢ１’，Ｂ１”，Ｂ２’，Ｂ２”から残りの画素の内挿を行う説明図である。
【符号の説明】
１ 画素ずらし光学系、２ 撮像素子、３ Ａ／Ｄ変換回路、４ 撮像部制御回路、５ 撮像部、６ カメラ制御部、７ 第一の画像メモリ、８ メモリ制御部、９ 画像処理部、１０ 信号レベル補正処理部、１１ 補間処理部、１２ 画像補正処理部、１３ 画像圧縮処理部、１４ インタフェース部、１５ 磁界発生回路、１６ コイル、１７ ファラデー素子、１８ 偏光板、１９ 複屈折板、２０ 固定長圧縮回路、２１ ラインバッファ、２２ 圧縮画像メモリ、２３ 固定長伸張処理部、３０ 電界発生回路、３１ 電気光学素子、６１ レンズ、９１ 駆動電圧発生回路、９２ 液晶板、１２７ 第二の画像メモリ、１２８ 合成処理部。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus for realizing an image with high resolution.
[0002]
[Prior art]
In an image input device using a two-dimensional image sensor such as a CCD, each image is synthesized after each change of the relative position between the image and the image sensor and changed to obtain an apparent number of pixels. One that increases the resolution by increasing it is known. In order to improve the resolution by such a method, a flat transparent member is disposed in parallel between the photographing optical system and the two-dimensional image sensor as described in Japanese Patent Application Laid-Open No. 7-236086. An image pickup device is formed by displacing incident light from a photographic optical system using one of the three tilting means not on a straight line arranged above as a support portion and the other two points as an operation portion for operating the transparent member. The image above is slightly displaced.
[0003]
FIG. 40 is a block diagram of a conventional image input device that optically increases the number of pixels disclosed in, for example, Japanese Patent Application Laid-Open No. 7-236060, and FIG. 41 is a perspective view showing a mechanism of an imaging unit. is there.
[0004]
40 and 41, 61 is a lens that is an imaging optical system for forming an image, 2 is an imaging element such as a CCD that is two-dimensionally arranged for photoelectrically converting the image, and 103 is an imaging lens 61. It is a transparent flat plate member that is arranged substantially in parallel between the image pickup devices 2 and causes a slight displacement in the incident angle of incident light from the image pickup lens 61 to the image pickup device 2.
[0005]
Reference numeral 104 denotes a base unit that supports the lens 61 and the transparent flat plate member 103. Reference numerals 105a, 105b, and 105c fix the transparent flat plate member 103 to the base unit 104 and selectively actuate two points to tilt the transparent flat plate unit. Compression springs 106a, 106b and 106c are spring pressing plates for respectively pressing the corresponding compression springs 105, 107a and 107b are provided together with the transparent flat plate member 103 and screws passing through the compression springs 105a and 105b, and are driven in the vicinity of the transparent flat plate member 103. This is a motor that causes the transparent flat plate member 103 to be inclined by displacing the part in the optical axis direction.
[0006]
Further, 108a and 108b are first and second actuating portions including the transparent flat plate member 103, compression springs 105a and 105b, and spring pressing plates 106a and 106b. Reference numeral 109 denotes a support portion including the transparent flat plate member 103, the compression spring 104c, and the spring pressing plate 106c, and supports the transparent flat plate member 103 when the operation portions 106a and 106b are operated. These are integrally fixed to a housing (not shown), and as shown in FIG. 40, a predetermined image processing circuit and image composition memory 110 for processing the photoelectrically converted image signal and an image buffer are provided in the subsequent stage. A memory 7 or the like is connected.
[0007]
Next, the operation will be described.
[0008]
First, imaging is performed in a state where neither of the two operation units 106a and 106b is operated, and an image is stored in the image memory 7 at the subsequent stage. Next, when one 106a of the two actuating parts is actuated, the transparent flat plate member 103 rotates with the line connecting the other actuating part 106b and the support part as the rotation axis. Therefore, the image transmitted through the transparent flat plate member 103 is moved by the inclination of the transparent flat plate member 103 to form an image on the image sensor 2, and an image slightly shifted is stored in the image buffer memory.
[0009]
Furthermore, if the same one operation part 106a is operated, an image will move sequentially in the same direction, and an image will be imaged and memorize | stored in the image pick-up element 2 sequentially. Further, when the operation unit 106b is driven, the transparent flat plate member 103 is inclined with a line connecting the operation unit 106a and the support unit 109 as a rotation axis, and the image is moved in a direction different from the above.
[0010]
By appropriately combining these two-direction movements, a two-dimensional pixel shift to an arbitrary position is carried out, and then a plurality of photographed images stored in the image buffer memory 7 are considered in consideration of the pixel shift execution direction. An image in which the number of pixels is optically increased by interpolation for each pixel is obtained in the image synthesis memory 109.
[0011]
Next, a conventional signal processing method for generating a high-definition image from an image obtained by conventional pixel shifting will be described.
[0012]
The signal array shown in FIG. 2 is an image sensor having a Bayer array that is most often used in a single-plate image storage, and the R, G, or B signal in the figure is sampled at the pixel position on the arrangement of the image sensor. Color signal.
[0013]
FIG. 12 is a diagram in which the color signal of the image captured by shifting the image sensor shown in FIG.
[0014]
In FIG. 12, R1, G1, and B1 are color signals of the first photographed image, and R2, G2, and B3 are color signals obtained from the second photographed image after pixel shifting.
[0015]
When the image shown in FIG. 12 is generated as a high-resolution image, it is necessary to generate a full-color signal having four times the number of pixels because it is doubled both horizontally and vertically compared to FIG. It is necessary to generate a non-photographing color signal in the pixel from the peripheral pixel signal and to interpolate all the color signals of blank pixels in the same manner.
[0016]
For example, now focusing on only the G signal, there are G signals G1, G2 obtained by imaging only at the positions shown in FIG. In the conventional technique, G1 ′ and G2 ′ are interpolated from the average value of the front and rear, left and right signals, and then further interpolated from the G signal obtained by interpolation as shown in FIG. A G signal can be obtained.
[0017]
Further, focusing on the B signal, as shown in FIG. 44, the B1 ′ signal is interpolated from the left and right B1 signals, and further the B1 ″ signal is interpolated from the interpolated B1 ′ signal. The same applies to the B2 signal. 45, the remaining pixels are interpolated from the interpolated B1 ′, B1 ″, B2 ′, B2 ″ as shown in FIG. By the above method, R, G, B signals for all pixels can be obtained.
[0018]
[Problems to be solved by the invention]
However, the technique disclosed in Japanese Patent Application Laid-Open No. 7-236086 described above has a highly accurate pixel shift amount because the transparent flat plate member is mechanically driven and controlled by two motors in order to change the optical path two-dimensionally. It was difficult to realize. In particular, the pixel pitch of recent solid-state imaging devices is mainly several microns, and a complicated control system is required to obtain a mechanical accuracy that is a fraction of that. In addition, since control is performed using mechanical vibration, a design that fully considers vibration and repeated life is required.
[0019]
Furthermore, in the conventional signal processing that generates a high-resolution image from an image captured by the above method, since it is a simple linear interpolation, it is not possible to obtain a resolution corresponding to the number of pixels obtained by performing pixel shifting. First, since the image shown in FIG. 2 in the single-plate image pickup device has one color filter disposed on each pixel, R, G, and B signals corresponding to the number of pixels cannot be obtained, and the number of pixels is the same. The first cause is that the resolution is not obtained. Next, there is a problem that the linear interpolation method cannot improve the resolution when obtaining a high resolution image from two images.
[0020]
In order to solve such a problem, the present applicant has applied for a method of shifting a minute pixel without mechanically vibrating the imaging optical system in Japanese Patent Application Laid-Open No. Sho 62-26984.
[0021]
The present invention has been made in view of the above-described points, and an object of the present invention is to obtain an imaging device capable of realizing an image with sufficiently improved resolution with respect to an increase in the number of pixels.
[0022]
[Means for Solving the Problems]
  An image pickup apparatus according to the present invention includes a pixel shift unit that optically displaces the position of incident light with respect to the image sensor by a half pixel, and a first image that stores captured images of the image sensor before and after the displacement by the pixel shift unit. Of the captured color signals constituting the captured image stored in the storage unit and the first image storage unit, the captured color signal which is a reference colorIt was judged that the similarity was high 1 It is formed on a plane with the pixel coordinates of three points arranged in the dimension direction as the horizontal axis and the signal level values of these three points as the vertical axis.Triangle and,Non-imaging color signal that is an interpolation colorAre formed on a plane in which two points arranged in a one-dimensional direction that is the same direction as the reference color and the pixel coordinates of the pixel to be interpolated are set on the horizontal axis and the signal level values of these three points are set on the vertical axis.Interpolation means for performing color interpolation in consideration of the similarity ratio of triangles, second image storage means for storing the results of color interpolation of the captured images before and after displacement by the interpolation means, and captured images of the image sensor before and after displacement And a synthesizing means for synthesizing.
[0023]
Further, the above-mentioned combining means determines whether or not the portion corresponding to the edge in the image passes through the blank pixel when performing pixel interpolation on the blank pixel at the position where the imaging pixel does not exist, and when passing through the blank pixel, the edge component The blank pixels are interpolated in the ridge line direction.
[0024]
In addition, compression / expansion means is provided for compressing a captured image of the image sensor, storing the compressed image in a compressed image memory, decompressing the compressed image, and outputting the decompressed image to the first image storage means. It is what.
[0025]
Further, the present invention is characterized in that there is provided signal level correction means for correcting the signal level of the image pickup color signal constituting the picked-up image of the image pickup element before and after displacement stored in the first image storage means.
[0026]
In addition, the signal level correction means calculates the average signal level of the color signal in the predetermined area in the captured image of the image sensor before and after the displacement, and corrects the signal level of the color signal so that both average signal levels match. It is characterized by doing.
[0027]
The pixel shifting means includes a magneto-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of the magnetic field applied to the magneto-optical element to displace the position of the incident light with respect to the image sensor. It is characterized by this.
[0028]
The pixel shifting means includes an electro-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of an electric field applied to the electro-optical element to displace the position of the incident light with respect to the image sensor. It is characterized by this.
[0029]
The pixel shifting means includes a liquid crystal plate disposed in an optical path that guides incident light to the image sensor, and controls the intensity of voltage applied to the liquid crystal plate to displace the position of the incident light with respect to the image sensor. It is a feature.
[0030]
Further, the pixel shifting means is arranged between the imaging optical system and the imaging element.
[0031]
Further, the pixel shifting means is arranged in the preceding stage of the imaging optical system.
[0032]
DETAILED DESCRIPTION OF THE INVENTION
Embodiment 1 FIG.
Embodiment 1 of the present invention will be described below with reference to the drawings.
[0033]
FIG. 1 is an overall block diagram showing Embodiment 1 of the present invention, and describes a case where it is realized as a digital still camera capable of taking a still image.
[0034]
FIG. 3 shows a detailed configuration of the image pickup unit 5 in FIG. 1. As a pixel shifting means, for example, as shown in Japanese Patent Application Laid-Open No. 61-267462 filed by the present applicant, it differs depending on the magnetic field. A case where a magneto-optical element exhibiting directivity is used will be described as an example.
[0035]
1 and 3, reference numeral 61 denotes a lens, 1 denotes a pixel shifting optical system that modulates the optical path of a subject image incident on the lens 61, and 2 denotes a surface in which RGB primary color filters form a Bayer type array as shown in FIG. An image pickup device for photoelectrically converting an optical image having a square charge-coupled device (hereinafter referred to as a CCD) arranged on the surface thereof, and an analog / digital conversion circuit for digitally converting an analog signal output from the image pickup device 2 (Hereinafter referred to as an A / D conversion circuit) 4 is an image pickup unit control circuit for controlling the pixel shift means 1 and the image pickup device 2, and 5 is a lens 61, a pixel shift optical system 1, an image pickup device 2, and an A / D conversion circuit. 3 is an imaging unit including the imaging unit control circuit 4.
[0036]
Reference numeral 6 denotes a camera control unit such as a CPU that controls the entire imaging apparatus while communicating with the imaging unit 5, 7 denotes a first image memory that receives and temporarily stores a digital image signal, and 8 denotes an A / D conversion circuit 3. A memory control unit such as a direct memory access controller (hereinafter referred to as a DMA controller) for writing a digital image signal from the image memory 7 to the image memory 7, 9 performs digital image processing on the image data stored in the first image memory 7 Reference numeral 127 denotes an image processing unit configured by software or an electronic circuit for performing image processing, and a second image memory used for image processing in the image processing unit 9.
[0037]
Further, 10 is a signal level correction processing unit that performs white balance correction, black level correction, gamma correction, and the like on the image signal, 11 is an interpolation processing unit that performs color interpolation on the captured image by digital image processing, and 128 is color-interpolated. A synthesizing processing unit that synthesizes two images to generate a single high-resolution image, and 12 is a noise generated in the optical system or circuit system for one high-resolution image obtained by the synthesizing processing unit 128. An image correction processing unit including a noise removal filter that performs removal, and an image compression processing unit 13 that encodes a finally obtained full-color image by an image compression method such as a JPEG (Joint Photographic Experts Group) method that is an international standardization method. It is.
[0038]
Furthermore, 14 stores an image finally obtained through the imaging unit 5 and the image processing unit 9 in a liquid crystal screen display or a secondary storage means such as a flash memory, or a data interface with a transmission path such as a serial interface or infrared communication. It is an interface unit that performs a man-machine interface such as a shutter switch.
[0039]
Next, the operation will be described.
[0040]
First, a schematic operation of the entire imaging apparatus will be described, and detailed operations of each unit will be described later.
[0041]
Here, a procedure for obtaining two high-definition images having a pixel density 2 × 2 times the actual number of pixels of the CCD by taking two images shifted by a half pixel in the 45 ° direction using the pixel shifting optical system 1 Will be described.
[0042]
When shooting in the high-definition mode is set by the photographer from the interface unit 14 and a depression of a release switch (not shown) is transmitted from the man-machine interface to the camera control unit 6, the imaging unit 5 in the imaging unit control circuit 4 An imaging operation is performed under one magnetic field application condition (for example, no applied magnetic field). As a result, the captured image is digitized and transmitted from the imaging unit 5 to the memory control unit 8.
[0043]
In the memory control unit 8, the input image signal is stored in the first image memory 7 according to the rules described later, and the signal level correction processing unit 10 multiplies the signal level value of each color of RGB of the image by the intensity correction coefficient and performs white balance correction processing. The color is interpolated by the interpolation processing unit 11 and stored in the second image memory 127.
[0044]
Next, an image photographed with a shift of ½ pixel in the −45 degree direction compared to the first photographed image due to the second applied magnetic field condition in the image sensing unit control circuit 4 is transmitted from the image sensing unit 5 through the transmission path. Similarly, it is transmitted to the image processing unit 9, stored in the first image memory 7, color-interpolated by the interpolation processing unit 11 through the signal level correction processing unit 10, and accumulated in the second image memory 127. Next, the image processing unit 9 reads the first and second interpolated images stored on the second image memory 127 with the end of the processing of the second captured image interpolation processing unit 11.
[0045]
Next, the synthesizing unit 128 performs high resolution processing on the first and second interpolated images, and synthesizes two photographed images into one high resolution image. Furthermore, the image data is transferred to the interface unit 14 for display on a liquid crystal display (not shown) or storage in a secondary storage medium such as a compact flash memory card through the image correction processing unit 12 and the image compression processing unit 13 in the subsequent stage.
[0046]
Hereinafter, the pixel shift optical system 1 in the imaging unit 5, the image storage in the first image memory 7 by the memory control unit 8, and the interpolation processing unit 11 and the synthesis processing unit 128 will be described in detail.
[0047]
The configuration and operation of the imaging unit 5 will be described in detail with reference to FIGS.
[0048]
3 and 4, 15 is a magnetic field generation circuit, 16 is a coil that generates a magnetic field in accordance with a voltage applied to the magnetic field generation circuit 15, and 17 is a magnetic field generated by the magnetic field generated in the coil 16 by the magnetic field generation circuit 15. A Faraday element as a magneto-optical element that produces an optical effect, 18 is a polarizing plate, and 19 is a birefringent plate.
[0049]
FIGS. 5, 6, and 7 show the respective polarizing plates 18, the Faraday element 17, and the birefringent plate 19 viewed from the AA, BB, and CC lines in FIG. 4, respectively. Yes. In FIG. 4, when light from a lens 61 for forming an image is incident on the polarizing plate 18, linearly polarized light in the amplitude direction Wa shown in FIG. 5 is obtained. 4 indicates the direction of the magnetic field applied to the Faraday element 17. When the linearly polarized light in the vibration direction Wa shown in FIG. 5 enters the Faraday element 17 such as lead glass, the polarization plane of the linearly polarized light is rotated by the applied magnetic field in the magnetic field direction Ha.
[0050]
When linearly polarized light traveling in the magnetic field direction Ha is incident on the Faraday element 17, the plane of polarization of the transmitted light rotates, and the rotation angle θ is obtained by the following equation (1).
[0051]
θ = R × l × H (1)
Here, l is the thickness of the Faraday element, H is the strength of the magnetic field, and R is a Verdet constant. The above formula (1) is described in, for example, “Optical Measurement Handbook” issued by Asakura Shoten Co., Ltd.
[0052]
In FIG. 4, the strength of the magnetic field at which the rotation angle θ is 0 degree is H_{θ 0}Then, H = H_θIn this case, linearly polarized light in the vibration direction Wa shown in FIG. 5 is obtained, and H = H_{θ 0}At the vibration direction W shown in FIG._{θ 0}Linearly polarized light is obtained. When linearly polarized light in the vibration direction Wa enters the birefringent plate 19, an ordinary ray Lo shown in FIG. 7 is obtained. Note that Q shown in FIG. 7 is the optical axis of the birefringent plate 19. Also, the vibration direction W_{θ 0}Is incident on the birefringent plate 19, the extraordinary ray L shown in FIG._EIs obtained. Ordinary ray Lo and extraordinary ray L_EAnd the distance P in the birefringent plate 19 shown in FIGS. 3 and 4 is selected as P = PH / 2 (PH indicates the horizontal pixel pitch of the image sensor 2). The phase of the change in the magnetic field strength H applied to the Faraday element 17 is matched with the field shift pulse shown in FIG. 8A (see FIGS. 8A and 8B).
[0053]
With the above-described operation, in the imaging apparatus according to the present invention, signal charge accumulation in each of the A and B fields can be performed at a position separated by PH / 2 with respect to the relative position between the incident image and the pixel of the imaging device 2. it can. That is, the spatial sampling region can be increased by temporally changing the strength H of the magnetic field applied to the Faraday element 17 and temporally changing the relative positional relationship between the incident optical image and the imaging element 2. . Correspondingly, the timing of the signal readout pulse shown in FIG. 8C is also shifted by a time T corresponding to PH / 2 (see FIGS. 8C and 8D). As a result, as shown in FIG. 8E, the imaging apparatus according to the present invention can obtain a high pixel density image in one cycle with each A and B field as one frame.
[0054]
In the first embodiment, for simplification of description, the horizontal direction, that is, the coordinates with respect to the one-dimensional space has been described as the pixel shift direction. However, the pixel shift position can be two-dimensionally performed based on the same principle. In that case, n sets of optical element groups each including a polarizing plate 18, a first optical element having a magneto-optical effect, and a birefringent plate 19 are arranged in an imaging system, and a linear line, for example, is provided between the optical element groups. What is necessary is just to insert n-1 quarter wavelength plates for converting polarized light into circularly polarized light. By setting binary voltages to be applied to n optical elements having n magneto-optical effects in the n sets of optical element groups, it is possible to obtain an image shifted by 2 n pixels.
[0055]
Based on the imaging principle described above, the Bayer type single-plate color imaging device of FIG. 2 is used, and the pixel position having the center of gravity by the first applied magnetic field condition represented by the black circle shown in FIG. A pixel-shifted image can be taken under the second applied magnetic field condition represented by a white circle that is shifted by two pixels. Hereinafter, the high resolution processing in the interpolation processing unit 11 and the synthesis processing unit 128 in this case will be described in detail with reference to the drawings.
[0056]
FIG. 10 is a flowchart illustrating processing procedures of the interpolation processing unit 11 and the synthesis processing unit 128. Among them, the interpolation processing unit 11 performs color interpolation on the color components (for example, B and R components if the photographing component is G) that is missing in each pixel in the captured image having the arrangement shown in FIG. A full-color image having the same number of pixels is generated (ST101, ST102), and the synthesis processing unit 128 inserts the two full-color images generated by the interpolation processing unit 11 at positions corresponding to the pixel shift direction in FIG. Thus, by interpolating all color components of pixels (hereinafter referred to as blank pixels) for which no image signal exists (ST103), a single sheet having twice as many pixels as the number of pixels of the image sensor 2 in both length and width Generate an image.
[0057]
In each processing, a target pixel (hereinafter referred to as a target pixel) on which color interpolation or pixel interpolation is performed is sequentially applied to the image stored in the first image memory 7 or the second image memory 127 in the x direction (main scanning). When one line processing is completed, the raster scan advanced by one pixel in the y direction is repeated.
[0058]
Next, specific operations in each step will be described.
[0059]
A captured image under the first applied magnetic field condition is stored in the first image memory 7 in a pixel array as shown in FIG. 2 and subjected to correction processing by the signal level correction processing unit 10. Correction processing includes, for example, white balance correction processing for correcting the sensitivity characteristics of color filters corresponding to RGB colors in the image sensor 2 to correct white to the correct signal level, and correcting the dynamic range of signals in the low or high sensitivity range. The gamma correction processing to be performed is selected as necessary.
[0060]
The first captured image that has been processed by the signal level correction processing unit 10 is temporarily stored in the second image memory 127 and then subjected to processing by the interpolation processing unit 11 so as to be a full color having all RGB color components. The image is stored again in the second image memory 127 as an image.
[0061]
Next, imaging is performed under the second applied magnetic field condition, and the same number of imaging elements 2 as the captured image under the first applied magnetic field condition is processed by the signal level correction processing unit 10 and the interpolation processing unit 11. The image is stored in the second image memory 127 as a full-color image having all RGB color components having the number of pixels. At this time, in the second image memory 127, an image interpolated after shooting under the first applied magnetic field condition and an image interpolated after shooting under the second applied magnetic field condition have both the vertical and horizontal pixel numbers as shown in FIG. They are arranged as part of the pixels constituting one composite image that is doubled. In FIG. 11, each circle indicates the barycentric position of the pixel in the combined image corresponding to the pixel shift position at the time of imaging the first and second images in FIG. 9, and the black circle is the first circle An image captured under the applied magnetic field condition and color-interpolated, a white circle indicates an image captured under the second applied magnetic field condition and color-interpolated, and a hatched circle indicates a blank pixel that does not correspond to any of these ing.
[0062]
FIG. 12 shows the existing image signal components and the positional relationship before the processing of the interpolation processing unit 11 corresponding to the positional relationship of the center of gravity in FIG. From FIG. 12, the color signal with 1 subscript is obtained from the first photographed image, and the color signal with 2 subscript is obtained from the second photographed image. By the processing of the interpolation processing unit 11, all RGB color signals other than the color components shown in FIG. 12 are generated by interpolation.
[0063]
When the interpolation processing unit 11 performs color interpolation of the first photographed image (ST101 in FIG. 10) and color interpolation of the second photographed image (ST102 in FIG. 10), try to interpolate in the vicinity of the target pixel to be interpolated. The color signal level at the target pixel is similar by using a signal of the same color as the color to be used (hereinafter referred to as an interpolated color) and a color signal used for reference in a color other than the interpolated color (hereinafter referred to as a reference color). Is calculated.
[0064]
For example, when the G color signal is generated by interpolation among the non-imaging color signals at the pixel position where the color signal is R (hereinafter referred to as the R pixel position), the R pixel position shown in FIG. Among the pixels existing in the pixel window, a pair of two R color signals (R1, R2) or (R3, R4) existing in the vertical direction or the horizontal direction in addition to the R color signal of interest, and the R color signal Or a combination of two G color signals (G1, G2) or (G3, G4) in the same direction in the same direction.
[0065]
As a method for selecting either the vertical direction or the horizontal direction, a calculation method is known by obtaining an absolute value of a difference between signal levels of two pixels in each direction of the reference color or the interpolation color. . That is, it can be determined that the smaller the difference value is, the higher the similarity of the local image in that direction is. FIG. 14A shows a case where the color interpolation of the interpolated color signal is performed using the reference color signal of the horizontal direction component, for example, when the similarity in the horizontal direction is determined to be high in the case of FIG. It shows the way of thinking. In FIG. 14A, the shape of the triangle formed by these points in a two-dimensional space including the reference direction for each color signal and the axis of the signal level value is indicated by a broken line.
[0066]
In the first embodiment, the interpolation processing unit 11 uses the similarity ratio when the triangle formed by the imaging color signal R as the reference color and the triangle formed by the non-imaging color signal G as the interpolation color are changed to similar figures. An interpolation signal gR for the G color signal is calculated.
[0067]
Here, the triangle (R · R1 · R2H) and the triangle (R · R1H · R2H) formed by the points R1H, R2H and R obtained by replacing R1 and R2 with the average value RAVE are geometrically identical. It is clear to have an area of In the image signal, the integral value (that is, the area of the geometric figure) obtained from the signal distribution corresponds to the image energy, and if it has the same area, that is, the integral value is the same, the average luminance in the region is stored and there is a visual problem. Therefore, the triangle by the broken line in FIG. 14A can be replaced by the isosceles triangle (R, R1H, R2H) by the solid line.
[0068]
Similarly, the triangles (gR, G1, and G2) formed by the G color signal as the interpolation color are replaced with isosceles triangles (gR, G1H, and G2H), and isosceles triangles (R, R1H, and R2H) and isosceles triangles. By determining the interpolation signal value gR so that (gR · G1H · G2H) becomes a similar figure, color interpolation according to the ratio of image energy in a minute region is possible. When color interpolation by the conventional linear interpolation method is performed, the reproduced gR is a point of (G1 + G2) / 2, and the G color component shows a flat transition despite the convex shape of the R color component. The resolution of the G color component is not sufficient, and a false color is generated due to the protrusion of the R color signal. On the other hand, when this method is used, since the transition of the G color signal is reproduced well according to the local image energy, color interpolation with high resolution and few false colors can be realized.
[0069]
Further, the calculation in this case is performed by using Expression (2).
[0070]
Further, the calculation in this case is performed by using Expression (2).
[Expression 1]

However, in equation (2), the (x, y) coordinate of the R pixel position to be interpolated is (m, n), and G_AVE(M, n) is the average value of G1 and G2, R_AVE(M, n) represents an average value of R1 and R2, and is multiplied by 1/2 as a line segment similarity ratio of two isosceles triangles.
[0071]
In the above example, the G color signal component is interpolated at the R pixel position. However, by replacing R with B, the G color signal component at the B pixel position can be similarly interpolated.
[0072]
In the above example, the G component at the R pixel position is color-interpolated. However, the same concept can be applied to the K color interpolation at an arbitrary J pixel position. FIGS. 13B and 13C show the relationship between the reference pixel position of the reference color and the reference pixel position of the interpolation color when performing other color component interpolation at other pixel positions. FIG. 13B shows a case of R component and B component interpolation at the G pixel position, and the above model is applied to the triangle shown in FIG. 14B for each color component.
[0073]
That is, since these three points form a triangle, an isosceles triangle is created by replacing the two points other than the pixel of interest with their average value, and a line segment is formed so that this and the interpolation color form a similar isosceles triangle. Multiply by the similarity ratio of 1/2. Further, the interpolation of the R component and the B component at the G pixel position where R and B are interchanged can be performed in the same manner.
[0074]
FIG. 13C shows the state of the reference pixel position at the time of R component interpolation at the B pixel position. In this case, the imaging G component at the G pixel position shown in FIG. 14C and the interpolated G component at the B pixel position of the target pixel are used as points for forming the reference color triangle, and the generated R component at the G pixel position is used. Is used as a point for forming an interpolation color triangle, and the line segment similarity ratio of each isosceles triangle is set to 1/1, so that the same treatment as in the previous examples can be performed. Further, by interchanging R and B, B component interpolation at the R pixel position is possible.
[0075]
Based on the above description, generalizing equation (2) can be described as equation (3).
[Expression 2]

In Equation (3), K (m, n) is a non-imaging color signal to be interpolated at coordinates (m, n), J is an imaging color signal as a reference color at coordinates (m, n), and Cd is The line segment similarity ratio of the isosceles triangle formed by the K color with respect to the isosceles triangle formed by the J color is shown.
[0076]
In this way, it is possible to execute the color interpolation processing for the photographic pixels for one screen by repeatedly performing all color component interpolation in each pixel while sequentially scanning the pixel position to be color-interpolated in the horizontal and vertical directions. .
[0077]
In this manner, the interpolation processing unit 11 performs color interpolation on the first photographed image photographed under the first applied magnetic field condition, and temporarily stores it in the second image memory 127. Subsequently, the second captured image captured under the second applied magnetic field condition is similarly subjected to the signal level correction processing unit 10, and then subjected to color interpolation processing in the interpolation processing unit 11, and is stored in the second image memory 127. Memorize temporarily.
[0078]
Here, the first interpolation image and the second interpolation image stored in the second image memory 127 are 1 as shown in FIG. 11 for pixel interpolation processing with blank pixels in the synthesis processing unit 128 described later. Although it may be stored in accordance with a physical address indicating a predetermined position in one composite image, it is physically stored in an individual image storage area, and in the next composite processing unit 128, each pixel in FIG. Reading may be performed as appropriate with the positional relationship in mind.
[0079]
Next, a process for synthesizing two interpolated images into one image will be described.
[0080]
First, the G component generation at the blank pixel position will be described in detail.
[0081]
The shaded portion in FIG. 15 shows the positional relationship of the G component present in the image at this stage. When interpolating the G component at the blank pixel position, it is determined whether or not the portion corresponding to the edge in the image passes through the target pixel with reference to the surrounding pixels, and in that case, the pixel interpolation is performed in the ridge line direction of the edge line segment. By doing so, sharp edge reproduction is possible. A direction indicated by an arrow in FIG. 15 indicates a line segment direction that can be detected based on a reference pixel window including 7 × 7 pixels centered on the target pixel.
[0082]
The line segment angle detection is executed according to the following procedure.
[0083]
The signal level average value Dav is calculated for the G component existing in the target pixel and the eight neighboring pixels. Next, the signal value of each G pixel in the 7 × 7 pixel window is binarized to 1 or 0 using the calculated signal level average value Dav as a threshold value. The G pixel in the binarized window is compared with a plurality of predetermined patterns, and the line segment angle passing through the central pixel and the line segment on which side the line segment is bright or dark due to pattern matching Angle information is recognized. Based on the line segment angle information, the G component of the target pixel is calculated by linearly interpolating a plurality of pixel values existing in the ridge line direction of the line segment.
[0084]
For example, when the pixels in the 7 × 7 pixel window are binarized into the hatched pixel (binarized data: 0) and the diagonal line pixel (binarized data: 1) in FIG. It is detected that the edge of the image passes in the direction of the arrow, and the pixel of interest is obtained by simple averaging or weighted averaging with respect to the distance from the pixel of interest, etc. Is interpolated.
[0085]
By detecting the angle of the line segment and interpolating in the ridge line direction, it is possible to interpolate the edge sharply and the line segment more smoothly than in the case of component interpolation using a simple average value of eight neighboring pixels. Thereby, it is possible to give a high resolution to the observer of the image.
[0086]
With the above method, the G components of all the pixels are generated with high resolution, and the R and B components can be interpolated and generated with high resolution by referring to this result.
[0087]
As for R and B component interpolation at blank pixel positions, interpolation is performed by detecting edge line vectors for line segments using pattern matching similar to G component interpolation at the blank pixel positions. At this time, in order to prevent erroneous interpolation caused by differences in individual pattern matching results of RGB components, R and B component interpolation may be performed based on the line segment angle results detected during G component interpolation.
[0088]
As described above, the high-resolution image for four pixels having the number of pixels 2 × 2 times that of the image sensor is generated by the synthesis processing unit 128, and the generated full-color image is converted into an optical system and an imaging system by the image correction processing unit 12. Processing such as a noise removal filter that corrects image distortion is performed, and the image compression processing unit 13 encodes the image in the JPEG format and sends the image to the interface unit 14. In this way, a high-resolution image exceeding the Nyquist frequency of the image sensor 2 is formed from the two pixel-shifted images.
[0089]
In the first embodiment, the example in which the high resolution processing is performed using the two images captured by shifting the pixels by ½ pixel in the −45 degrees direction using the pixel shifting unit has been described. In other words, a finite number of light receiving elements of the image sensor 2 are arranged in a plane, and the incident light on the light receiving element surface is output as an integrated value, so that the spatial frequency of the captured image is restricted (Nyquist frequency). This can be explained by the sampling theorem. On the other hand, according to "Ultra high-definition image acquisition method using two cameras" (Komatsu, Aizawa, Saito: Television Society Journal Vol. 45, No. 10, pp. 1256 to 1262), the aperture ratio is 100%. It is clearly shown that when the pixel-shifted images are integrated using this image sensor, the resolution can be improved to within twice. By using this method and taking into account the aperture ratio of the actual image sensor, the shape of the light receiving element, and the like, and optimizing the shift direction, shift amount, and number of shots, the resolution in the composite image can be optimized.
[0090]
In the first embodiment, eight line segments are detected using a window of 7 × 7 pixels at the time of pattern matching in the synthesis processing unit 128. However, the present invention is not limited to this. In order to increase the speed or reduce the circuit scale, the window size may be reduced and the correlation between the edge line segments between the pixels in four directions of 45 degrees or only in the two directions of horizontal and vertical may be used.
[0091]
In the first embodiment, the first image obtained by changing the imaging condition of the pixel shifting optical system 1 is taken and stored in the first image memory 7 to be stored in the signal level correction processing unit 10 and the interpolation processing unit 11. After finishing the above processing, the second image capturing under the second applied magnetic field condition is started. However, this is not restrictive. That is, the first image memory 7 has a capacity for storing two shot images, and the image processing unit 9 starts image processing when the two shot images are ready. Effects can be obtained. This can be flexibly changed according to the design concept of the actual apparatus.
[0092]
In the first embodiment, the example in which the second captured image is shifted in the −45 degree direction with respect to the first captured image has been described. It goes without saying that the same processing can be applied even if the direction or the direction of −135 degrees is shifted.
[0093]
Embodiment 2. FIG.
The second embodiment will be described below with reference to the drawings.
[0094]
FIG. 17 is a block diagram illustrating the overall configuration of the second embodiment. In FIG. 2, the same parts as those of the first embodiment shown in FIG. As a new code, 20 is a fixed-length compression circuit that encodes each captured image data by a fixed-length compression method, and 21 is a compressed digital signal output from the imaging unit 5 during compression by the fixed-length compression circuit 20. A line buffer that delays and accumulates the number of lines to be used, 22 is a compressed image memory that stores image data compressed by the fixed-length compression circuit 20, and 23 is a fixed-length expansion processing unit.
[0095]
Next, the operation will be described. The operation of the imaging unit 5 including the pixel shifting mechanism is the same as in the first embodiment. A captured image input from the imaging unit 5 using the pixel shifting optical system 1 is temporarily stored in the line buffer 21 for image compression by the fixed length compression circuit 20. Hereinafter, the compression operation in the fixed length compression circuit 20 will be described in detail.
[0096]
As the fixed-length encoding algorithm in the fixed-length encoding means 20, for example, an algorithm that performs an encoding method that eliminates redundancy of adjacent pixel information for each block having 4 × 4 pixels as one unit is used. In this case, since the same color pixel does not exist in the adjacent pixel position in the pixel array of FIG. 2, the same color pixel is blocked in the fixed-length compression circuit 20.
[0097]
FIG. 18 is a diagram illustrating a state in which the pixel array in FIG. 2 is rearranged by the fixed-length compression circuit 20. The captured image data is sequentially stored in the line buffer 21 from the imaging unit 5, and the encoding in the fixed-length compression circuit 20 is performed at the stage where rearrangement of the pixel arrangement is started when the unprocessed data is arranged for 8 lines. Pixel rearrangement is performed in units of 8 × 8 pixels, and each color 4 × 4 pixel is divided into unit blocks. The rearranged image signals are arranged by collecting G components at addresses in the upper left direction and addresses in the lower right direction of the block, collecting B components at addresses in the lower left direction, and R components at addresses in the upper right direction. Is done.
[0098]
An image encoding method in the fixed length compression circuit 20 will be described.
[0099]
FIG. 19 is a diagram showing addresses when the fixed-length compression circuit 20 encodes the color image signals in the rearranged unit blocks. That is, FIG. 19 shows addresses of m blocks in the vertical direction and n columns in the horizontal direction (m and n are natural numbers of 0 <m and n ≦ 4) in the unit block which is a block of image signals of the same color for every 4 pixels in the vertical and horizontal directions. This indicates that a quantization level described later is attached to the image signal.
[0100]
FIG. 20 shows a quantization level at which the intensity of the image signal of each pixel is hierarchized (quantized). 20, Lmin is the minimum value in the image signal intensity of 16 pixels shown in FIG. 19, Lmax is the maximum value in the image signal intensity of the same 16 pixels, and P1 is 8 between the maximum value Lmax and the minimum value Lmin. Equally divided 1/8 value from the bottom, P2 is 1/8 value from the top, Q1 is an average value of pixels having a signal intensity of Lmin or more and P1 or less, and Q8 is a pixel having a signal intensity greater than Lmax or less than P2. Is the average value.
[0101]
LD is a gradation width index in the unit block, and is equal to Q8-Q1. L1 to L7 are obtained by arranging the values obtained by dividing the gradation width index LD into eight equal parts from the smallest. LA is equal to (Q1 + Q8) / 2 at the average level of image data in the unit block. φijk represents the quantization level for each pixel.
[0102]
21 and 22 are flowcharts showing the encoding procedure according to this embodiment. The encoding procedure will be described below with reference to this flowchart.
[0103]
First, the fixed length compression circuit 20 reads the image data in the unit block rearranged as shown in FIG. 18B (step ST1). Next, the signal intensity of the read image data for 4 × 4 pixels is calculated, and the values of P1, P2, Q1, Q8, LA, LD, L1 to L7 are sequentially obtained according to the following formulas (step ST2 to step ST2). ST13).
[0104]
P1 = (Lmax + 7Lmin) / 8
P2 = (7Lmax + Lmin) / 8
Q1 = Ave (Xmn ≦ P1)
Q2 = Ave (Xmn> P2)
LA = (Q1 + Q8) / 2
LD = Q8-Q1
L1 = LA-3LD / 8
L2 = LA-LD / 4
L3 = LA-LD / 8
L5 = LA + LD / 8
L6 = LA + LD / 4
L7 = LA + 3LD / 8
[0105]
The expression Q1 means obtaining an average value of pixels having a signal intensity not less than Lmin and not more than P1, and the expression Q8 means obtaining an average value of pixels having a signal intensity greater than Lmax and less than P2.
[0106]
In this way, after sequentially obtaining the values of P1, P2, Q1, Q8, LA, LD, and L1 to L7, the fixed length compression circuit 20 sets n = 1 and m = 1 (steps ST14 and 15). It is determined whether or not the signal intensity (hereinafter referred to as pixel value) Xmn (that is, the pixel value X11) of the pixel at the address (m, n) at this time is equal to or less than L1 (step ST16).
[0107]
If the pixel value X11 is equal to or less than L1, the quantization level φijk of this pixel value is set to binary 000 (step ST17). Next, m is incremented by 1 (step ST31), and it is determined whether m is 4 or less (step ST32). When m is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0108]
When m is larger than 4, n is incremented by 1 (step ST33), and it is determined whether or not the incremented n is 4 or less (step ST34). When n is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0109]
If the pixel value Xmn is larger than L1, it is determined whether or not it is L2 or less (step ST18). If the measure Xmn is L2 or less, the quantization level φijk of this pixel is set to binary 001. Set (step ST19). Next, m is incremented by 1 (step ST31), and it is determined whether m is 4 or less (step ST32). When m is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16). When m is larger than 4, n is incremented by 1 (step ST33), and it is determined whether or not the incremented n is 4 or less (step ST34). When n is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0110]
Similarly, it is determined whether the pixel value has a value between L1 and L2, between L2 and L3, between L3 and LA, between LA and L5, between L5 and L6, and between L6 and L7 (step ST16). , ST18, ST20, ST22, ST24, ST26, ST28), and according to the values, quantization levels φijk = 000, 001, 010, 011, 100, 101, 110, 111 are allocated to the pixels (steps ST17, ST19). , ST21, ST23, ST25, ST27, ST29).
[0111]
In this way, the quantization level is assigned to all the pixels in the same unit block, and the encoding is completed. The encoded data of the unit block is LA, LD, and φijk for each pixel.
[0112]
These processes are repeated for the number of unit blocks for the entire screen. When the digital signal from the A / D conversion circuit 4 is 10 bits per pixel (corresponding to 2 bytes when not bit-packed), the image compression rate by fixed-length encoding is 10/32.
As described above, each captured image data is subjected to fixed-length encoding in the fixed-length compression circuit 20 for each block composed of pixels of the same color sequentially, and after the amount of image data is reduced, the memory control unit 8 is stored in the compressed image memory 22.
[0113]
After the two photographed image data are accumulated in the compressed image memory 22 in this way, image processing in the image processing unit 9 is performed. First, the image data encoded in the fixed length compression circuit 20 is decoded. An image decoding method in the fixed length expansion processing unit 23 will be described with reference to the drawings. FIG. 23 is a flowchart showing the operation of the fixed length extension processing unit 23. Hereinafter, the fixed-length decoding operation of the fixed-length decompression processing unit 23 will be described with reference to this flowchart.
[0114]
When the fixed-length decoding operation is started, first, the vertical coordinate value n is set to 1 (step ST40), and the horizontal coordinate value m is set to 1 (step ST41). That is, the address of the coordinate value (1, 1) in a certain unit block is designated by the operations of step ST40 and step ST41.
[0115]
Next, it is determined how many quantization levels Φijk of the designated address are (steps ST42, ST44, ST46, ST48, ST50, ST52, ST54), and the average value level LA and the respective quantization levels Φijk are determined. Based on the gradation width index LD, the signal intensity Ymn of the pixel (Y11 if the pixel has the coordinate value (1, 1)) is obtained (steps ST43, ST45, ST47, ST49, ST51, ST53, ST55, ST56). .
[0116]
In order to obtain the signal intensity Ymn from the average value level LA and the gradation width index value LD in each step, the following arithmetic expressions are used, respectively.
[0117]
Ymn = LA-LD / 2 (Step ST43)
Ymn = LA-5LD / 14 (Step ST45)
Ymn = LA-3LD / 14 (Step ST47)
Ymn = LA-LD / 14 (step ST49)
Ymn = LA + LD / 14 (Step ST51)
Ymn = LA + 3LD / 14 (Step ST53)
Ymn = LA + 5LD / 14 (Step ST55)
Ymn = LA + LD / 2 (Step ST56)
After obtaining the signal strength of the pixel (1, 1), the pixel is moved one by one in the horizontal direction (steps ST57 and ST58), and the signal strength of the pixel (2, 1) is decoded by the same procedure (step ST57). ST42 to ST56).
[0118]
In this way, after decoding the signal intensity for the uppermost pixel in the unit block (step ST58), the vertical coordinate value is incremented by 1 (step ST59), and the same processing is performed for the next pixel. The signal strength is decoded (steps ST42 to ST58).
[0119]
In this way, the signal strength is decoded for all the pixels in the unit block (steps ST41 to ST60), and the decoding operation is terminated.
[0120]
Next, the decoded data is subjected to the reverse rearrangement process of the data rearranged by the fixed length compression circuit 20 as shown in FIG. 24, and the pixel signals are read out in the scanning line direction in the same order. Rearrangement is stored in the first image memory 7.
[0121]
As described above, the image data equivalent to the image data before the fixed length compression is stored in the first image memory 7, and the image processing similar to that in the first embodiment is performed by the image processing unit 9. It is applied in other blocks and output to the interface unit 14.
[0122]
As described above, when a plurality of images are taken, each image data is compressed using the fixed length compression circuit 20, thereby outputting an image signal output from the A / D conversion unit 3 of the imaging unit 5. Is stored in the first image memory 7, a smaller amount of data can be supplied to the data bus and stored in the storage means, and a high-speed memory control circuit for shortening the continuous shooting interval or at least one frame can be stored. It is not necessary to use a semiconductor memory capable of high-speed writing, and the apparatus can be configured at a low price.
[0123]
In other words, by using a small amount of the high-speed compressed image memory 22, it is possible to increase the apparent data storage speed in the storage means, and the first image capturing sequence is completed in a short time. By doing so, the second image capturing operation can be started in a short time. Therefore, it is possible to minimize the temporal change of the subject or the camera shake of the photographer that occurs between the two photographed images, and the same subject can be photographed with a ½ pixel shift with high accuracy. effective.
[0124]
In the second embodiment, the first image memory 7 and the compressed image memory 21 are provided separately. However, the present invention is not limited to this, and the first image memory 7 and the compressed image memory 21 may be provided on a semiconductor memory having a common physical address area. Needless to say, these memories and the second image memory 127 can also be provided on the same semiconductor memory.
[0125]
In the second embodiment, the fixed-length compression circuit 20 and the line buffer 22 and the compressed image memory 21 used in the fixed-length compression circuit 20 are separately provided. However, the present invention is not limited to this, and two of these components are used. The above may be included in one integrated circuit such as an LSI.
[0126]
Embodiment 3 FIG.
The third embodiment will be described below with reference to the drawings. The configuration and operation of the imaging unit 5 including the pixel shifting means and the configuration of the image processing unit 9 are configured as shown in FIG. 1 or FIG. 17 as in the first and second embodiments. The operation of the signal level correction processing unit 10 of the image processing unit 9 which is characteristic in the third embodiment will be described in detail.
[0127]
Two image signals captured by the imaging unit 5 are stored in the first image memory 7 of the image processing unit 9 via the A / D conversion circuit 3. The accumulated image data is subjected to individually applicable signal correction processing such as white balance correction processing and gamma correction processing in the signal level correction processing section 10. At this time, since the first image and the second image are taken in time series, for example, when a fluorescent lamp that emits light at a relatively low frequency of 50 to 60 MHz is taken as a light source, the light emission of the light source There may be a difference in the amount of incident light between the two images due to the relationship between the timing and the shutter time. That is, a difference occurs in the average brightness of the two captured images, which may cause image quality degradation when the synthesis processing unit 128 in the subsequent stage performs the interpolation image synthesis processing.
[0128]
FIG. 25A shows the lightness characteristics when the same subject is imaged when the illumination light quantity in the second image capturing condition is reduced by about 30% compared to the first image capturing condition. Each signal level when photographing under constant illumination causes a decrease in average brightness and a reduction in dynamic range when the illumination conditions change. In the figure, the Max and Min values are the maximum signal level and the minimum signal level in each image. The signal level correction processing unit 10 according to the third embodiment performs signal level correction processing of two images in addition to signal level correction processing in each image.
[0129]
That is, the maximum signal level value and the minimum signal level value in each image are detected, and processing for adjusting the other to one of the signal level characteristics is performed. Normally, automatic exposure correction or shutter speed optimization is performed at the time of photographing the first image, and photographing is performed under the optimum conditions. Therefore, the second image is adjusted to the brightness distribution characteristic of the first image.
[0130]
If each signal level in the second image before and after correction is D (j) (j is each lightness level value under constant illumination) and D ′ (j), correction is first performed according to equation (4). (See FIG. 25 (b)).
[0131]
D '(j) = D (j) + (Min1-Min2) (4)
[0132]
Subsequently, the dynamic range of the first image and the second image corrected by equation (4) is corrected according to equation (5), and the final correction signal level D ″ (j) of the second image is corrected. Calculate (see FIG. 25C).
[Equation 3]

By configuring the signal level correction processing unit 10 according to the above procedure, the brightness characteristics can be matched well even when the illumination conditions of the two captured images are different, and high-accuracy pixel interpolation / synthesis is possible. Processing is feasible.
[0133]
Also, when the imaging apparatus is configured as shown in FIG. 17 (Embodiment 2) and the captured image is encoded using the fixed-length compression circuit 20, LA in each code is the average signal level in a block of 16 pixels. Since it is a representative value, it can be easily explained that an equivalent effect can be realized equivalently even if all the above procedures are performed on LA. When configured in this manner, it is possible to reduce the computation scale related to the maximum / minimum signal level detection and signal level correction, which has been performed for all the pixels, to 1/16, and to realize high-speed processing. There is a special effect.
[0134]
In the third embodiment, the expressions (4) and (5) are sequentially applied to the second photographed image. However, the present invention is not limited to this, and these can be performed by a single image data scan. Needless to say.
[0135]
Embodiment 4 FIG.
Hereinafter, the fourth embodiment will be described with reference to the drawings. The overall configuration in the fourth embodiment is the same as the configuration in FIG. 1 in the first embodiment or the configuration in FIG. 17 in the second embodiment. The fourth embodiment is different from the first and second embodiments in that an electro-optical element that changes the refraction phenomenon of transmitted light according to the strength of the electric field is used as the pixel shifting optical system 1.
[0136]
26 and 27 show a configuration of the imaging unit 5 in the fourth embodiment. In these figures, 30 is an electric field generating circuit for generating an electric field by applying a voltage, and 31 is an electro-optical element.
[0137]
Next, the operation will be described. The configuration and operation of the imaging unit 5 in FIG. 26 will be described. FIG. 28, FIG. 29, and FIG. 30 show the electro-optic element 31 and the polarizing plate 18 in FIG. 27 as viewed from the directions of AA, BB, and CC, respectively. .
[0138]
In FIG. 27, natural light incident on the electro-optic element 31 from the lens 61 for forming an image is two polarized components L orthogonal to each other as shown in FIG._X, L_YCan be expressed as When no electric field is applied to the electro-optic element 31, L in FIG._X0, L_Y0The polarization component shown in FIG._X, L_YIs observed at a straight line. When a predetermined electric field E1 is applied to the electro-optic element 31, the electro-optic element 31 exhibits a birefringence phenomenon, and L in FIG._X1, L_Y1The polarization component shown in FIG.
[0139]
Where L_X1Is the position where LX goes straight as an ordinary ray in the birefringence phenomenon._Y1Is L as an extraordinary ray of birefringence phenomenon._YIs observed in the opposite position by a distance P. However, in this case, an electric field E1 such that P = PH / 2 (PH indicates the horizontal pixel pitch of the image sensor 2) as shown in FIG. 30 is applied to the electro-optic element 31 by the electric field generation circuit 30.
[0140]
The operation timing in the imaging apparatus will be described with reference to FIG. 8. FIG. 8A shows a field shift pulse, FIG. 8B shows the strength of the electric field E1 applied to the electro-optic element 31, and FIG. ) Is a signal readout pulse, FIG. 8D is a diagram showing an output signal of the image sensor 2, and FIG. 8E is a diagram when the A and B fields are viewed as one frame. The phase of the change in the intensity of the electric field E1 is made to coincide with the field shift pulse in FIG.
[0141]
With the above-described operation, the imaging apparatus according to the fourth embodiment performs signal charge accumulation in the A and B fields at a position separated by PH / 2 with respect to the relative position between the incident image and the pixel of the imaging element 2. Can do. Correspondingly, the timing of the signal readout pulse shown in FIG. 8C is also shifted by a time T corresponding to PH / 2. As a result, as shown in FIG. 8E, this imaging apparatus can obtain an image with a high pixel density in one cycle with the A and B fields as one frame.
[0142]
The configuration and operation of the image processing unit 9 in the fourth embodiment can be configured using any of those described in the previous embodiments.
[0143]
Embodiment 5 FIG.
The fifth embodiment will be described below with reference to the drawings. The overall configuration in the fifth embodiment is the same as that in FIG. 1 in the first embodiment or FIG. 19 in the second embodiment. In the fifth embodiment, a case where a polarizing plate that produces linearly polarized light and a liquid crystal plate that changes the azimuth angle of incident light are used as the pixel shifting optical system 1 will be described as an example.
[0144]
FIG. 31 shows a configuration of the imaging unit 5 in the fifth embodiment.
[0145]
In FIG. 31, 18 is a polarizing plate that generates linearly polarized light with respect to incident light, 92 is a liquid crystal plate that can change the azimuth angle of the linearly polarized light generated by the polarizing plate 18 under voltage application conditions, and 91 is a liquid crystal plate 92. It is the drive voltage control circuit which controls the applied voltage to.
[0146]
Next, the operation will be described. An enlarged view of the pixel shifting optical system 1 and the image sensor 2 is shown in FIG. 32, AA is the incident side of the polarizing plate 18, BB is the outgoing side of the polarizing plate 18, CC is the outgoing side of the liquid crystal plate 92, and DD is the vertical side of the outgoing side of the birefringent plate 19. In FIG. The position of the direction is shown. 33 to 36 show the polarization directions of light at the positions AA to DD shown in FIG.
[0147]
When an object is imaged, the light incident on the optical system is non-polarized light and can be represented by a vertical polarization component and a horizontal polarization component as shown in FIG. The polarizing plate 18 emits light linearly polarized from the incident light. Further, as shown in FIG. 34, the azimuth angle of the polarization direction of the polarizing plate 18 is 45 ° with respect to the reference angle when the horizontal pixel arrangement of the image sensor 2 is used as the reference angle. Arrange the polarization axis at an angle.
[0148]
The liquid crystal plate 92 is a plate having a liquid crystal layer inside and sandwiching the liquid crystal between two electrode substrates. When a voltage, that is, an electric field is applied from the outside, an electro-optic effect associated with a change in the molecular arrangement of the internal liquid crystal is exhibited. cause. The liquid crystal plate 92 used in the present invention changes the polarization direction of incident linearly polarized light by applying a voltage to the liquid crystal plate 92 or turning off the applied voltage. As an example for realizing the liquid crystal plate 92, there is an optical rotation effect by a twisted nematic (TN) type liquid crystal, which is a typical field effect type among the electro-optic effects. The operation is shown in FIG. 37 and FIG.
[0149]
The nematic liquid crystal exhibiting the TN mode has positive dielectric anisotropy, and the molecular arrangement of the element is shown in FIGS. 37 and 38, 93 is a liquid crystal molecule. When a voltage is applied to the substrates, as shown in FIG. 37, the molecular long axis direction is arranged so as to be perpendicular to both substrate surfaces, so that the linearly polarized light incident through the polarizing plate 19 is emitted as it is. The When no voltage is applied, the alignment of the liquid crystal molecules in the device is continuously twisted by 90 °, and optically causes a 90 ° optical rotation effect. Therefore, the linearly polarized light incident from the polarizing plate 19 is twisted by 90 °. The outgoing light has a 90 ° polarization direction different from that of the incident light.
[0150]
The polarization of the light emitted from the liquid crystal plate 92 is shown in FIG. In FIG. 35, c1 is a state in which a voltage is applied to the liquid crystal plate 92, and when no voltage is applied, it becomes linearly polarized light having an azimuth angle of 135 ° with respect to the reference angle as in c2. Now, let c1 linearly polarized light be the first deviation direction and c2 linearly polarized light be the second deviation direction. The light emitted from the liquid crystal plate 92 is then incident on the birefringent plate 19.
[0151]
The birefringent plate 19 is a material having a so-called birefringence in which the refractive index is not uniform depending on the polarization direction. In the imaging apparatus, when the spatial frequency of the subject exceeds the sampling frequency obtained from the pixel pitch of the imaging device 2, a repetitive noise is generated. Since it appears in the image as (aliasing noise), a quartz plate or the like is often used as an optical low-pass filter. The quartz plate, that is, the birefringent plate 19 separates incident light into ordinary rays and extraordinary rays, and the separation distance can be adjusted by the thickness of the birefringent plate 19.
[0152]
When a birefringent plate is provided so that incident light in the first deflection direction is an ordinary ray and incident light in the second deviation direction is an extraordinary ray, the extraordinary ray is indicated by a dotted line as shown in FIG. As described above, the optical axis is different from that of the ordinary ray, and the incident light having the respective deflection directions forms images at different positions on the image sensor 2 as indicated by d1 and d2 in FIG. Further, as shown in FIG. 36, the thickness of the birefringent plate 19 is wx / 2 which is half of the pixel pitch wx of the horizontal image sensor, and wy / 2 which is half of the pixel pitch wy of the vertical image sensor. An extraordinary ray is provided to be different from an ordinary ray.
[0153]
Therefore, the two images, the first image obtained by imaging the image formed on the image sensor 2 in the first deviation direction and the second image taken in the second deviation direction, are mutually Both horizontal and vertical are relatively shifted by 1/2 pixel.
[0154]
From the above, the imaging apparatus outputs a driving voltage from the driving voltage control circuit 91 to the liquid crystal plate 92 when driving the imaging device 2 under the control of the imaging unit control circuit 4. When the voltage is applied to the liquid crystal plate 92, the liquid crystal molecular arrangement shown in FIG. 39 is formed, and incident light in the first deviation direction is imaged on the image sensor 2. The image sensor 2 captures an image with a drive signal from the drive voltage control circuit 91 under the control of the image capturing unit control circuit 4 to obtain a first image.
[0155]
Next, after taking the first image, the drive voltage control circuit 91 sets the voltage applied to the liquid crystal plate 92 to zero. The liquid crystal molecular arrangement of the liquid crystal plate 92 is as shown in FIG. 38, and incident light in the second deflection direction is imaged on the image sensor 2. The image sensor 2 captures an image with a drive signal from the drive voltage control circuit 91 to obtain a second image.
[0156]
FIG. 39 shows voltage pulses applied to the liquid crystal plate 92. The imaging unit control circuit 4 ends the imaging of the first image during the state 1 in FIG. 39 and ends the imaging of the second image during the state 2 in FIG.
[0157]
By operating as described above, it is possible to obtain a first image and a second image that are shifted by 1/2 pixel both horizontally and vertically.
[0158]
The second image captured by shifting the first captured image by 1/2 pixel from the first captured image obtained as described above is sequentially converted into a digital signal by the A / D conversion unit 3 and then the image processing unit 9. Sent to.
[0159]
In the fifth embodiment, an example has been described in which the second captured image is shifted by 45 degrees as the direction in which the second captured image is shifted by 1/2 pixel. However, the present invention is not limited to this, and the liquid crystal 92 and birefringence are not limited thereto. A 135 degree direction, a −45 degree direction, a −135 degree direction, and the like can be realized by appropriately combining the arrangement of the plate 19 or the voltage applied to the liquid crystal 92.
[0160]
In the fifth embodiment, the TN liquid crystal is given as an example of the liquid crystal plate 92 that changes the polarization direction of the incident linearly polarized light. However, the same effect can be obtained if the liquid crystal plate 92 has an optical rotation effect by the photoelectric effect. Can have.
[0161]
In the first embodiment, in the pixel shifting optical system 1 shown in FIG. 3, the lens 61 is closer to the subject than the pixel shifting unit including the polarizing plate 18, the magneto-optical element 17, and the birefringent plate 19. However, this is not the case. In other words, the same pixel shifting effect can be obtained by disposing the lens 61 in the subsequent stage of these means, and the pixel shifting means can be attached and detached, so that the pixel shifting can be performed even in a normal imaging apparatus. By mounting the means for the above, a special effect that it can be realized as an imaging device capable of capturing a high-definition image occurs.
[0162]
In the fourth embodiment, in the pixel shifting optical system 1 shown in FIG. 26, the lens 61 is disposed at a position closer to the subject than the pixel shifting unit including the polarizing plate 18 and the electro-optic element 31. However, this is not the case. In other words, the same pixel shifting effect can be obtained by disposing the lens 61 in the subsequent stage of these means, and the pixel shifting means can be attached and detached, so that the pixel shifting can be performed even in a normal imaging apparatus. By mounting the means for the above, a special effect that it can be realized as an imaging device capable of capturing a high-definition image occurs.
[0163]
In all the embodiments described above, the configuration example of the digital still camera capable of continuously capturing still images has been described. However, a high-definition still image shooting mode in a digital video recorder capable of capturing moving images, etc. Needless to say, it can be configured as.
[0164]
Further, in all the embodiments described above, the configuration examples capable of executing the color interpolation process and the synthesis process inside the imaging apparatus have been shown. However, the present invention is not limited to this, and the configuration is not limited to the imaging apparatus such as a personal computer or a color printer. You may comprise on the apparatus which can be connected via a storage medium indirectly.
[0165]
【The invention's effect】
  As described above, according to the present invention, the pixel shifting unit that optically displaces the position of the incident light with respect to the image sensor by a half pixel and the captured images of the image sensor before and after the displacement by the pixel shifting unit are stored. One image storage means and a captured color signal which is a reference color among the captured color signals constituting the captured image stored in the first image storage meansIt was judged that the similarity was high 1 It is formed on a plane with the pixel coordinates of three points arranged in the dimension direction as the horizontal axis and the signal level values of these three points as the vertical axis.Triangle and,Non-imaging color signal that is an interpolation colorAre formed on a plane in which two points arranged in a one-dimensional direction that is the same direction as the reference color and the pixel coordinates of the pixel to be interpolated are set on the horizontal axis and the signal level values of these three points are set on the vertical axis.Interpolation means for performing color interpolation in consideration of the similarity ratio of triangles, second image storage means for storing the results of color interpolation of the captured images before and after displacement by the interpolation means, and captured images of the image sensor before and after displacement Therefore, it is possible to obtain an imaging device capable of realizing an image with sufficiently improved resolution with respect to an increase in the number of pixels.
[0166]
Further, the above-mentioned combining means determines whether or not the portion corresponding to the edge in the image passes through the blank pixel when performing pixel interpolation on the blank pixel at the position where the imaging pixel does not exist, and when passing through the blank pixel, the edge component Since the blank pixel is subjected to pixel interpolation in the ridge line direction, sharp edge reproduction can be performed by performing pixel interpolation in the ridge line direction of the edge line segment.
[0167]
In addition, since the image picked up by the image pickup device is compressed, the compressed image is stored in the compressed image memory, the compressed image is decompressed and output to the first image storage means, the compression / expansion means is provided. When capturing a single image, each image data is compressed to store an image signal of the imaging unit in the first image memory, and a smaller amount of data is supplied to the data bus and stored in the storage means. Therefore, it is not necessary to use a high-speed memory control circuit for shortening the continuous shooting interval or a semiconductor memory capable of high-speed writing for at least one frame, and the apparatus can be configured at a low price. .
[0168]
In addition, since signal level correction means for correcting the signal level of the image pickup color signal constituting the picked-up image of the image pickup element before and after displacement stored in the first image storage means is provided, color corresponding to each RGB color is provided. The sensitivity characteristic of the filter can be improved.
[0169]
In addition, the signal level correction means calculates the average signal level of the color signal in the predetermined area in the captured image of the image sensor before and after the displacement, and corrects the signal level of the color signal so that both average signal levels match. Thus, even when the illumination conditions of the two captured pixels are different, the brightness characteristics can be matched well, and highly accurate pixel interpolation / synthesis processing can be realized.
[0170]
The pixel shifting means includes a magneto-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of the magnetic field applied to the magneto-optical element to displace the position of the incident light with respect to the image sensor. Therefore, a high pixel density image can be obtained.
[0171]
The pixel shifting means includes an electro-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of an electric field applied to the electro-optical element to displace the position of the incident light with respect to the image sensor. Therefore, a high pixel density image can be obtained.
[0172]
Further, the pixel shifting means includes a liquid crystal plate disposed in an optical path that guides incident light to the image sensor, and controls the intensity of voltage applied to the liquid crystal plate to displace the position of the incident light with respect to the image sensor. A high pixel density image can be obtained.
[0173]
In addition, by arranging the pixel shifting means between the image pickup optical system and the image pickup device so that the pixel shifting means can be attached and detached, a high-definition can be achieved by mounting the pixel shifting means while being a normal imaging device. It can be realized as an image pickup apparatus capable of taking frequent images.
It is characterized by.
[0174]
Furthermore, by arranging the pixel shifting means in the preceding stage of the imaging optical system, the pixel shifting means can be attached and detached, so that a high-definition image can be obtained by mounting the pixel shifting means while being a normal imaging device. Can be realized as an image pickup apparatus capable of picking up images.
[Brief description of the drawings]
FIG. 1 is an overall block diagram showing a first embodiment of the present invention.
FIG. 2 is an explanatory diagram of an image sensor having a Bayer arrangement.
3 is a diagram illustrating a detailed configuration of an imaging unit 5 in FIG. 1. FIG.
4 is a diagram for explaining the configuration and operation of an imaging unit 5 in FIG. 1. FIG.
5 is a view of the polarizing plate 18, the Faraday element 17, and the birefringent plate 19 as seen from the line AA in FIG.
6 is a view of the polarizing plate 18, the Faraday element 17, and the birefringent plate 19 as seen from the line BB in FIG.
7 is a view of the polarizing plate 18, the Faraday element 17, and the birefringent plate 19 as seen from the line CC in FIG.
FIG. 8 is an explanatory diagram of operation timing in the imaging apparatus.
FIG. 9 is an explanatory diagram for capturing a pixel-shifted image using the Bayer type single-plate color image sensor of FIG. 2 based on the imaging principle.
10 is a flowchart showing processing procedures of an interpolation processing unit 11 and a synthesis processing unit 128. FIG.
FIG. 11 is an explanatory diagram of the arrangement of pixels constituting one composite image in which the number of pixels is double in both length and width.
FIG. 12 is an explanatory diagram of an existing image signal component and its positional relationship before the processing of the interpolation processing unit 11 corresponding to the positional relationship of the center of gravity in FIG.
FIG. 13 is an explanatory diagram in a case where a G color signal is generated by interpolation among non-imaging color signals at a pixel position (hereinafter referred to as an R pixel position) where the captured color signal is an R color.
14 is an explanatory diagram of a case where it is determined that the similarity in the horizontal direction is high in the case of FIG. 13 and the color interpolation of the interpolation color signal is performed using the reference color signal of the horizontal direction component.
FIG. 15 is an explanatory diagram showing the positional relationship of G components present in an image.
FIG. 16 is an explanatory diagram when the G component of the target pixel is calculated by linearly interpolating a plurality of pixel values existing in the ridge line direction of the line segment based on the line segment angle information.
FIG. 17 is a block diagram showing an overall configuration of a second embodiment of the present invention.
18 is a diagram showing a state in which the pixel array in FIG. 2 is rearranged by the fixed length compression circuit 20. FIG.
FIG. 19 is a diagram illustrating addresses when color image signals in rearranged unit blocks are encoded by the fixed-length compression circuit 20;
FIG. 20 is a diagram illustrating a quantization level for hierarchizing (quantizing) the intensity of an image signal of each pixel.
FIG. 21 is a flowchart showing an encoding procedure according to the second embodiment.
FIG. 22 is a flowchart showing an encoding procedure according to the second embodiment.
23 is a flowchart showing the operation of the fixed length extension processing unit 23. FIG.
FIG. 24 shows the reverse order of the data rearranged by the fixed-length compression circuit 20 and rearranges the pixel signals in the same order as when the pixel signals are read out in the scanning line direction, and stores them in the first image memory 7. FIG.
FIG. 25 is an explanatory diagram of subject brightness under constant illumination.
FIG. 26 is a diagram showing a configuration of an imaging unit 5 according to Embodiment 4 of the present invention.
27 is an operation principle diagram of FIG. 26. FIG.
28 is a view of the electro-optic element 31 and the polarizing plate 18 in FIG. 27 as viewed from the direction AA.
29 is a view of the electro-optic element 31 and the polarizing plate 18 in FIG. 27 as seen from the direction BB.
30 is a view of the electro-optic element 31 and the polarizing plate 18 in FIG. 27 as viewed from the CC direction.
FIG. 31 is a diagram showing a configuration of an imaging unit 5 according to the fifth embodiment of the present invention.
FIG. 32 is an enlarged view of the pixel shifting optical system 1 and the image sensor 2;
33 is a diagram showing the polarization direction of light at each position AA shown in FIG. 32;
34 is a diagram showing the polarization direction of light at each position BB shown in FIG. 32;
FIG. 35 is a diagram showing the polarization direction of light at each position of CC shown in FIG. 32;
36 is a diagram showing the polarization direction of light at each position of DD shown in FIG. 32;
FIG. 37 is an explanatory view of an optical rotation effect by a twisted nematic (TN) type liquid crystal, which is a representative field effect type, of the electro-optic effect as an example for realizing the liquid crystal plate 92.
FIG. 38 is an explanatory diagram of an optical rotation effect related to FIG.
FIG. 39 is an explanatory diagram of a liquid crystal molecular arrangement.
FIG. 40 is a block diagram of a conventional image input apparatus for optically increasing the number of pixels disclosed in Japanese Patent Laid-Open No. 7-236086.
FIG. 41 is a perspective view illustrating a mechanism of an imaging unit.
FIG. 42 is an explanatory diagram when G1 ′ and G2 ′ are interpolated from the average values of front, rear, left and right signals obtained by imaging.
FIG. 43 is an explanatory diagram in the case of obtaining G signals for all the pixels by further interpolation from the G signals obtained by interpolation.
FIG. 44 is an explanatory diagram in the case of interpolating the B1 ′ signal from the left and right B1 signals and further interpolating the B1 ″ signal from the interpolated B1 ′ signal.
FIG. 45 is an explanatory diagram for interpolating the remaining pixels from the interpolated B1 ′, B1 ″, B2 ′, B2 ″.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Pixel shift optical system, 2 Imaging element, 3 A / D conversion circuit, 4 Imaging part control circuit, 5 Imaging part, 6 Camera control part, 7 First image memory, 8 Memory control part, 9 Image processing part, 10 Signal level correction processing unit, 11 interpolation processing unit, 12 image correction processing unit, 13 image compression processing unit, 14 interface unit, 15 magnetic field generation circuit, 16 coil, 17 Faraday element, 18 polarizing plate, 19 birefringent plate, 20 fixed Long compression circuit, 21 line buffer, 22 compressed image memory, 23 fixed length expansion processing unit, 30 electric field generation circuit, 31 electro-optic element, 61 lens, 91 drive voltage generation circuit, 92 liquid crystal plate, 127 second image memory, 128 Synthesis processing unit.

Claims (10)

Pixel shifting means for optically displacing the position of incident light with respect to the image sensor by half a pixel;
First image storage means for storing captured images of the image sensor before and after displacement by the pixel shifting means;
Three pixel coordinates arranged in a one- dimensional direction that are determined to have high similarity with respect to the captured color signal that is the reference color among the captured color signals that constitute the captured image stored in the first image storage unit And a triangle formed on a plane with the signal level values of these three points on the horizontal axis and two points arranged in a one-dimensional direction that is the same direction as the reference color for the non-imaging color signal that is the interpolation color, Interpolation means for performing color interpolation in consideration of a similarity ratio of triangles formed on a plane in which the pixel coordinate of a pixel to be interpolated is abscissa and the signal level values of these three points are ordinate ;
Second image storage means for storing the results of color interpolation of the captured images before and after displacement by the interpolation means;
An image pickup apparatus comprising: combining means for combining the picked-up images of the image pickup element before and after displacement. The imaging device according to claim 1,
The above synthesis means determines whether or not a portion corresponding to an edge in an image passes through a blank pixel when performing pixel interpolation on a blank pixel at a position where no image pickup pixel exists, and when passing through the blank pixel, the edge line of the edge component An imaging apparatus, wherein the blank pixel is interpolated in the direction. The imaging device according to claim 1 or 2,
A compression / expansion unit is provided that compresses a captured image of an image sensor, stores the compressed image in a compressed image memory, decompresses the compressed image, and outputs the decompressed image to the first image storage unit. Imaging device. In the imaging device according to any one of claims 1 to 3,
An image pickup apparatus comprising signal level correction means for correcting a signal level of a picked-up color signal constituting a picked-up image of an image pickup element before and after displacement stored in the first image storage means. The imaging apparatus according to claim 4.
The signal level correction means calculates the average signal level of the color signal in a predetermined area in the captured image of the image sensor before and after the displacement, and corrects the signal level of the color signal so that the average signal level of both is the same. An imaging apparatus characterized by the above. The imaging device according to any one of claims 1 to 5,
The pixel shifting means includes a magneto-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of the magnetic field applied to the magneto-optical element to displace the position of the incident light with respect to the image sensor. An imaging device that is characterized. The imaging device according to any one of claims 1 to 5,
The pixel shifting means includes an electro-optical element disposed in an optical path that guides incident light to the image sensor, and controls the intensity of an electric field applied to the electro-optical element to displace the position of the incident light with respect to the image sensor. An imaging device that is characterized. The imaging device according to any one of claims 1 to 5,
The pixel shifting means includes a liquid crystal plate arranged in an optical path that guides incident light to the image sensor, and controls the intensity of voltage applied to the liquid crystal plate to displace the position of the incident light with respect to the image sensor. An imaging device. The imaging device according to any one of claims 6 to 8,
An image pickup apparatus comprising: a pixel shifting unit disposed between an image pickup optical system and an image pickup element. The imaging device according to any one of claims 6 to 8,
An image pickup apparatus, wherein a pixel shifting unit is arranged in front of an image pickup optical system.

Images