JP4545978B2 - Image processing apparatus and image input apparatus - Google Patents

Description

【００４８】
ここに、(x,y)が画像中の位置、aはスケーリングファクター、θ_nはフィルタの方向成分を表し、Wは基本空間周波数、σ_Xσ_Yはフィルタ関数のｘ方向、ｙ方向の広がりの大きさを与えるパラメータである。但し、図３に示す回路構成上は、（１）式のガウシアン関数形状がラプラシアンで近似されたものとなる。
【００４９】
本実施の形態では方向選択性パラメータとしてのθ_nは６方向で０度、３０度、６０度、９０度、１２０度、１５０度の値をとり、aは２とし、スケール選択性パラメータとしてのｍは１から３までの値をとる整数として与える。なお、実際の回路構成上は、図３の回路要素の特性値（コンダクタンス値及びトランスコンダクタンス増幅器のゲイン）を適切に制御することにより、このような選択性が実現される。
【００５０】
Gaborフィルタカーネル（（１）式のg_mn）のサイズ(広がり)はスケールインデックスmに応じて変わるので、異なるスケールインデックスを有するGaborフィルタは異なるサイズ選択性を有するが、ここでは図３に示すような回路アレイの広がりが無限にあると仮定した場合にその伝達関数が、空間周波数選択性（スケール選択性）を有すること（文献１）を利用している。
【００５１】
先述した駆動タイミング制御信号の入力時刻ｔで、選択されるフィルタの特性パラメータをスケール選択性に関してm(t)と方向選択性に関してn(t)とおく。この選択された空間フィルタ特性を有する各フィルタについてg_m(t)n(t)(x,y)と入力濃淡画像との２次元畳み込み演算を行う（但し実際には、図３に示すような回路要素では、g_m(t)n(t)(x,y)の関数形状をラプラシアンとしたものとの畳み込み演算と等価な変換出力が得られる）ことによりGaborウエーブレット変換が行われる。すなわち、
【００５２】
【数２】

【００５３】
ここに、Iは入力画像、W_m(t)n(t)はGaborウエーブレット変換係数である。W_m(t)n(t) (m(t)=1,2,3; n(t)=1,..., 6)のセットを特徴ベクトルとして各点で求める。「*」は、複素共役をとることを示す。
【００５４】
図３に示す回路要素の空間フィルタアレイの出力、すなわち、h_r(p,q), h_i(p,q)としては、W_m(t)n(t) ≒ h_r(p,q) + j h_i(p,q)のように関係づけられる。
【００５５】
各空間フィルタ要素回路は、分布重み係数と画像データとの積和入力を行って得られるウエーブレット変換係数値出力を行う。すなわち、センサ出力からの全画面分のデータについて、所定のフィルタ特性を与えての空間フィルタ回路要素からの出力が時系列に読み出され、同様にして一連の異なるフィルタ特性を与えてのフィルタ出力がなされた結果、空間フィルタアレイ全体の時系列出力（式（２）のインデクスm, nが取りうる値の範囲全体にわたるWmnに相当するの値のセット）として、（2）式のGabor wavelet変換が行われたことになる。
【００５６】
なお、以下のようにして更新される各フィルタ特性ごとの時系列出力結果は一旦、それぞれ個別に複数の（或いは、一括して単一の）一時記憶手段に２次元アレイデータとして格納されて後続の処理（例えば、画像認識処理など）が行われる。例えば、空間フィルタ出力を受ける後続の処理モジュールが複数の演算素子が並列階層的に結合してなる神経回路網である場合には、上述した２次元アレイデータは、（例えば電圧信号に変換されて）通常は一括して同時に入力層の神経素子に出力される。
【００５７】
〈センサ回路からの画像データの転送制御〉
以下、再び図１を参照して説明する。制御回路２は、Gaborタイプ空間フィルタアレイ回路１の特性パラメータ(m,n)を更新するたびに、空間フィルタ回路要素中の対応する各抵抗値、及び増幅器ゲインの制御信号を出力する。
【００５８】
具体的には、制御回路２は、フィルタ特性制御信号として各抵抗を構成するＣＭＯＳトランスミッションゲート回路を構成するトランジスタのゲート電圧制御信号（V_G1x, V_G1y,V_G0）、及びトランスコンダクタンス増幅器を構成するトランジスタへのゲイン制御信号（V_G2x,V_G2y）を各空間フィルタ回路要素の特性が一括して更新されるように出力する。
【００５９】
以下、図４を参照して、制御回路の構成等について説明する。制御回路２は、内部の記憶手段４３に、Gaborタイプフィルタのスケール及び方向選択性インデクスの組(m,n)に対して、それぞれ固有のコンダクタンス値および増幅器ゲイン値のセット(G₀, G_1x, G_1y, G_2x, G_2y)を与える制御信号値の組のデータをテーブルとして記憶しており、各インデクスの組に対応する制御信号としてV_con,(mn)が方向・スケール選択信号発生器４１より入力されると、ゲート制御信号発生回路４２からは以下に示すコンダクタンス値および増幅器ゲイン値を与える制御信号V_G1x, V_G1y, V_G2x,V_G2y, V_G0が、図３に示す各空間フィルタ回路要素の各抵抗及び増幅器を構成するトランジスタへ出力される。
【００６０】
すなわち、V_con,(mn)に対応するG_1x,(mn), G_2x,(mn),G_1y,(mn),G_2y,(mn), G_0,(mn)は、それぞれ以下のように与えられるとすると、メモリでは、それぞれに対応する制御信号レベルV_G1x, V_G1y, V_G2x,V_G2y, V_G0を上記インデクス組(m,n)に対応して記憶している。なお、G_1x,(mn), G_2x,(mn),G_1y,(mn),G_2y,(mn), G_0,(mn)の値は上述した定義式により、中心方向θ_n、中心空間周波数Ω_mを用いて、
G_1x,(mn)=α_m ²cos(Ω_mcosθ_n)、G_2x,(mn)=α_m ²sin(Ω_mcosθ_n)、
G_1y,(mn)=α_m ²cos(Ω_msinθ_n)、G_2y,(mn)=α_m ²sin(Ω_msinθ_n)、
G_0,(mn)=α_m ²[(Δ_mΩ_m)²＋４―２cos（Ω_mcosθ_n）―２cos（Ω_msinθ_n）]
のように表される。ここに、
α_m ²=G_2(max)[1+cot²Ω_m]^0.5, Δ_m＝λ/（α_mΩ_m）
である。
【００６１】
なお、各抵抗および増幅器への制御信号線は図３の各回路要素の所定のトランジスタに配線されているが、煩雑さを避けるため図示していない。
【００６２】
制御回路２は、上述した方向・スケール選択信号発生器からの制御信号の出力タイミングに同期するようにセンサ回路３中の水平選択回路３ｈ及び垂直選択回路３ｖに対して、各水平・垂直選択線のうち順次”Ｈ”にすべき線の選択線信号を出力する。
【００６３】
具体的には、更新された上記選択信号V_con,(mn)が出力されると、これに同期して、センサ側に水平選択線信号と垂直選択線信号のアクティベート信号が出力され、センサ回路では、このアクティベート信号が入力されると順次”Ｈ”にすべき線の選択線信号が出力され、水平選択線と垂直選択線の両方がアクティブとなった画素の画素信号がセンサから上述した空間フィルタ回路へ出力される。
【００６４】
このようにして時系列信号として読み出された画素信号は、デマルチプレクサ回路５により選択された各空間フィルタ回路要素（図３）の入力部に以下のように転送される。デマルチプレクサ回路５としては、通常用いられる回路構成でよく（例えば、スイッチアレイとそのアクティベート回路等から構成される）、前処理された画素信号と選択線信号を入力して、選択線信号で決まる入力画像中の読み出し画素位置に対応するGaborタイプ空間フィルタ回路要素へ対応する画素信号の出力を行う。
【００６５】
次に、Gaborタイプ空間フィルタアレイ回路１からの出力の読み出しについて説明する。 読み出しは、制御回路２の方向・スケール選択信号発生器により設定されるべきスケールパラメータｍと方向成分パラメータｎが更新され、各フィルタ回路要素に入力されるべき画素信号が全て転送された後に行われる。
【００６６】
例えば、Gaborタイプフィルタの上記パラメータ(m,n)に応じて読み出された画像データのフィルタ回路要素への転送終了タイミングを、最後の選択線信号を入力することにより検出し、その後、デバイス特性により決まる所定の遅延時間を経て空間フィルタ出力が読み出され、一時記憶手段に複数の２次元アレイデータ（同一特性ごとの空間フィルタアレイ出力）として格納されるか、或いはそのまま時系列データとして後続の処理モジュールに出力してもよい。
【００６７】
この空間フィルタ出力の読み出しは、制御回路２からのセンサ選択線信号と同期したフィルタ選択線信号出力から所定の遅延時間後に行われるので、センサからの画像信号出力と常に一定の時間的対応を保持しながら空間フィルタ出力を取り出すことができる。
【００６８】
また、センサからのデータの読み出しと空間フィルタ特性の制御の適切なタイミング制御により、回路規模を小さくしながら上述した空間フィルタ出力とセンサ出力の対応付けを行うことができる。
【００６９】
後続の処理モジュールが神経回路網などの並列演算装置である場合には、読み出された時系列フィルタ出力を入力段の各素子にデマルチプレクサ回路等により分岐出力し、各素子の入力部には、入力データの値を保持するサンプルホールド回路または一時記憶メモリ素子を導入しておくか、或いは、所定のブロックメモリ素子に２次元アレイデータとして書き込み、その後、並列演算装置の入力素子アレイに一括出力してもよい。
【００７０】
以上、本実施の形態では、空間フィルタ回路の制御タイミングに基づいて、センサ回路へのアドレス制御や転送制御を行うものであったが、上述した空間フィルタ以外の他の処理回路、例えば、画像認識回路の中間処理出力タイミングなどに基づいて同様な制御を行う構成としてもよい。
【００７１】
次に、上述した空間フィルタリングを行う撮像信号処理回路を搭載した撮像装置（画像入力装置）の構成について説明する。
本実施の形態の構成に係る撮像信号処理回路を用いたパターン認識(検出)装置を撮影手段に搭載させることにより、特定被写体へのフォーカシングや特定被写体の色補正、露出制御を行う場合について、図５を参照して説明する。
【００７２】
図５の撮像装置９は、撮影レンズおよびズーム撮影用駆動制御機構を含む結像光学系９２、ＣＭＯＳイメージセンサ９３、撮像パラメータの計測手段９４、本実施の形態に係る空間フィルタ動作などを行う撮像信号処理回路９５、記憶手段９６、撮像動作の制御、撮像条件の制御などの制御用信号を発生する制御信号発生手段９７、EVFなどファインダーを兼ねた表示ディスプレイ９８、ストロボ発光手段９９、記録媒体９１０、更に被写体認識手段９１１などを具備する。被写体認識手段９１１は、撮像信号処理回路からのGaborタイプウエーブレット変換出力を用いて画像中から特定カテゴリのパターンの検出等を行う機能を有する。
【００７３】
この撮像装置は、例えば撮影された映像中から予め登録された人物の顔画像の検出(存在位置、サイズの検出)を被写体認識手段９１１により行う。その人物の位置、サイズ情報は制御信号発生手段に９７に入力されると、同制御信号発生手段９７は、撮像パラメータ計測手段９４からの出力に基づき、その人物に対するピント制御、露出条件制御、ホワイトバランス制御などを最適に行う制御信号を発生する。
【００７４】
本実施の形態に係る撮像信号処理回路を用いたパターン検出(認識)装置を、以上のように撮像装置に用いることにより、上記被写体の検出(認識)機能を小回路規模で低消費電力かつ高速（リアルタイム）に実現することができる。またその結果に基づいて、人物等の検出とそれに基づく撮影の最適制御（ＡＦ、ＡＥなど）を行うことができた。
【００７５】
本実施の形態は、図１に示されているセンサ回路３、制御回路２、前処理回路４、及び空間フィルタ回路１をそれぞれ、別々の半導体チップに形成する場合や同一の半導体チップに形成する場合とに適用される。
【００７６】
しかしながら、図１に示されているそれぞれの回路を、ＣＭＯＳプロセス等で同一の半導体チップ上に形成することによって、本実施の形態はより効果を発揮する。
【００７７】
本実施の形態では、光電変換素子アレイの一画面分の画像データに対して複数の異なる空間フィルタ特性における複数の空間フィルタ演算を、複数の空間フィルタ回路を設けて、並列的に行うのではなく、一つの空間フィルタによって時系列的に行うようにしている。
【００７８】
また、センサ回路３と空間フィルタ回路１とを同一半導体チップに形成した場合には、センサ回路３と空間フィルタ１のどちらに欠陥があると、そのチップは使用することはできなくなり、歩留まりが悪くなる。この場合に、空間フィルタを複数設ける場合には、チップ面積が大きくなり、非常に歩留まりが悪くなる不都合が発生する。それに対して、上述した本実施の形態のように、一つの空間フィルタ回路で時系列的に処理を行うようにした場合には、チップ面積が小さくなり、歩留まりを良くすることができる。
【００７９】
このように、複数の異なる処理を時系列的に行う一つの空間フィルタ回路を、センサ回路３等と一緒に同一半導体チップに形成することによって、複数の回路を同一の半導体チップ上に形成することによる歩留まりの悪化を防ぎ、かつ同一半導体チップに形成することによるメリット、すなわち、少面積化、低電力化、低価格化等のメリットを得ることが可能となる。
【００８０】
（第２の実施の形態）
図６に本実施の形態の要部構成ブロック図を示す。ここでは、図３の空間フィルタ回路要素の入力部にあるメモリ素子Ｃ_mは不要である。センサ回路出力のうち、時系列的にアクセスされる所定の空間フィルタ回路要素に対応する必要部分をブロックメモリに一旦記録し、その後ブロックメモリの各画素データを電圧信号として一括に（図３の回路でメモリ素子Ｃ_mを用いる場合は、時系列に）読み出して、上述したような空間フィルタ処理を行う。
【００８１】
本実施の形態では制御回路２は、第１の実施の形態と同様に空間フィルタ特性パラメータ(m,n)のうち、スケールパラメータｍに対応するスケール制御信号をアドレス選択信号とともにセンサ回路側に送出し、センサ回路の水平・垂直選択回路は、このスケール制御信号に応じた所定領域サイズについて各サンプリング画素を中心とする局所領域の画像データ（間引きによりサンプリングされた局所領域）をブロックメモリ６に記憶させる。このようにすることにより、センサ回路への読み出しタイミング制御信号を出力する制御回路２の負荷を軽減させることができる。
【００８２】
このブロックメモリ６は一次記憶素子としての機能を果たし、制御回路２からのタイミング信号に従って、各サンプリング位置を中心とした局所的な領域（サイズはスケールパラメータ依存）の画像データは、そのサンプリング点位置に対応する各空間フィルタ回路要素へ送出される。
【００８３】
この時、ブロックメモリが記憶する画像データは、制御回路２からの制御信号により間引きのサンプリング点位置が更新されるたびに、そのサンプリング点位置に対応する画像信号データ（前処理後）が後段の空間フィルタアレイ回路に転送され、また上記位置（アドレス）でのセンサからの画像信号が入力され、そのブロックメモリ上の対応するアドレスの画像データが書き換えられる結果、一定範囲で重複した画像データとして更新される。
【００８４】
その後、制御回路２の制御信号により空間フィルタ回路要素に時系列的に画像信号として出力される。なお、空間フィルタ回路からの出力の読み出しその他は、第１の実施の形態と同様である。
【００８５】
この場合も、図６に示されているセンサ回路３、制御回路２、前処理回路４、ブロックメモリ６、及び空間フィルタ回路１をそれぞれ、別々の半導体チップに形成する場合や同一の半導体チップに形成する場合とに適用される。
【００８６】
しかしながら、図６に示されているそれぞれの回路を、ＣＭＯＳプロセス等で同一の半導体チップ上に形成することによって、本実施の形態はより効果を発揮することができる。
【００８７】
本実施の形態では、光電変換素子アレイの一画面分の画像データに対して複数の異なる空間フィルタ特性における複数の空間フィルタ演算を、複数の空間フィルタ回路を設けて、並列的に行うのではなく、一つの空間フィルタによって時系列的に行うようにしている。
【００８８】
また、センサ回路３と空間フィルタ回路１とを同一半導体チップに形成した場合には、センサ回路３と空間フィルタ１のどちらに欠陥があると、そのチップは使用することはできなくなり、歩留まりが悪くなる。この場合に、空間フィルタを複数設ける場合には、チップ面積が大きくなり、非常に歩留まりが悪くなる不都合が発生する。それに対して、上述した本実施の形態のように、一つの空間フィルタ回路で時系列的に処理を行うようにした場合には、チップ面積が小さくなり、歩留まりを良くすることができる。
【００８９】
このように、複数の異なる処理を時系列的に行う一つの空間フィルタ回路１を、センサ回路３等と一緒に同一半導体チップに形成することによって、複数の回路を同一の半導体チップ上に形成することによる歩留まりの悪化を防ぎ、かつ同一半導体チップに形成することによるメリット、すなわち、少面積化、低電力化、低価格化等のメリットを得ることが可能となる。
【００９０】
（第３の実施の形態）
本実施の形態の要部構成を図７に示す。ここでは、センサ回路のタイミング制御特性、すなわち、空間フィルタ回路側の変調タイミング制御を通常のＣＭＯＳセンサで用いる読み出し制御信号を用いて行う。ただし、第１の実施の形態に示したような間引き読み出し制御が行われるものとする。
【００９１】
このようにすると、ＣＭＯＳセンサ回路についての読み出しタイミング制御、或いは転送回路等の内部の配線に変更を加えずに済み、既存のＣＭＯＳセンサチップ等を用いて空間フィルタリング動作を安定して行うことができる。
【００９２】
制御回路２は、センサ読み出し時の選択線信号を入力し、画像信号のアドレスの情報を得て、対応する位置の空間フィルタ回路要素に画像信号の転送制御を行う。
【００９３】
このとき、第１の実施の形態に示したGaborタイプのウエーブレット変換の特性パラメータ(m,n)に対応させて図３に示す各回路要素（抵抗、トランスコンダクタンス増幅器）の特性制御信号の送出を行うが、その更新はセンサからの読み出し動作が一通り終了してから行われる。例えば、入力画像について行うべき間引き読み出しが終了するたびに更新すればよい。また、その更新後に再び、（間引き）読み出し走査がセンサ回路において行われる。
【００９４】
画素信号は、第１の実施の形態と同様に入力部のトランスコンダクタンス増幅器Ｇｓに結合するメモリ素子Ｃ_mに電圧信号として一時的に保持され、空間フィルタ回路からの出力が読み出されると、そのメモリ素子の内容の書き換え動作を行う。その他の動作及び構成は、第１の実施の形態と同様である。
【００９５】
この第３の実施の形態においても、図７に示されているセンサ回路３、制御回路２、前処理回路４、及び空間フィルタ回路１をそれぞれ、別々の半導体チップに形成する場合や同一の半導体チップに形成する場合とに適用される。
【００９６】
しかしながら、図７に示されているそれぞれの回路を、ＣＭＯＳプロセス等で同一の半導体チップ上に形成することによって、本実施の形態はより効果を発揮することができる。
【００９７】
本実施の形態では、光電変換素子アレイの一画面分の画像データに対して複数の異なる空間フィルタ特性における複数の空間フィルタ演算を、複数の空間フィルタ回路を設けて、並列的に行うのではなく、一つの空間フィルタによって時系列的に行うようにしている。
【００９８】
また、センサ回路３と空間フィルタ回路１とを同一半導体チップに形成した場合には、センサ回路３と空間フィルタ１のどちらに欠陥があると、そのチップは使用することはできなくなり、歩留まりが悪くなる。この場合に、空間フィルタを複数設ける場合には、チップ面積が大きくなり、非常に歩留まりが悪くなる不都合が発生する。それに対して、上述した本実施の形態のように、一つの空間フィルタ回路１で時系列的に処理を行うようにした場合には、チップ面積が小さくなり、歩留まりを良くすることができる。
【００９９】
このように、複数の異なる処理を時系列的に行う一つの空間フィルタ回路１を、センサ回路３等と一緒に同一半導体チップに形成することによって、複数の回路を同一の半導体チップ上に形成することによる歩留まりの悪化を防ぎ、かつ同一半導体チップに形成することによるメリット、すなわち、少面積化、低電力化、低価格化等のメリットを得ることが可能となる。
【０１００】
（本発明の他の実施の形態）
本発明は複数の機器から構成されるシステムに適用しても１つの機器からなる装置に適用しても良い。
【０１０１】
また、上述した実施の形態の機能を実現するように各種のデバイスを動作させるように、上記各種デバイスと接続された装置あるいはシステム内のコンピュータに対し、記憶媒体から、またはインターネット等の伝送媒体を介して上記実施の形態の機能を実現するためのソフトウェアのプログラムコードを供給し、そのシステムあるいは装置のコンピュータ（ＣＰＵあるいはＭＰＵ）に格納されたプログラムに従って上記各種デバイスを動作させることによって実施したものも、本発明の範疇に含まれる。
【０１０２】
また、この場合、上記ソフトウェアのプログラムコード自体が上述した実施の形態の機能を実現することになり、そのプログラムコード自体、およびそのプログラムコードをコンピュータに供給するための手段、例えばかかるプログラムコードを格納した記憶媒体は本発明を構成する。かかるプログラムコードを記憶する記憶媒体としては、例えばフロッピーディスク、ハードディスク、光ディスク、光磁気ディスク、ＣＤ−ＲＯＭ、磁気テープ、不揮発性のメモリカード、ＲＯＭ等を用いることができる。
【０１０３】
また、コンピュータが供給されたプログラムコードを実行することにより、上述の実施の形態で説明した機能が実現されるだけでなく、そのプログラムコードがコンピュータにおいて稼働しているＯＳ（オペレーティングシステム）あるいは他のアプリケーションソフト等の共同して上述の実施の形態で示した機能が実現される場合にもかかるプログラムコードは本発明の実施の形態に含まれることは言うまでもない。
【０１０４】
さらに、供給されたプログラムコードがコンピュータの機能拡張ボードやコンピュータに接続された機能拡張ユニットに備わるメモリに格納された後、そのプログラムコードの指示に基づいてその機能拡張ボードや機能拡張ユニットに備わるＣＰＵ等が実際の処理の一部または全部を行い、その処理によって上述した実施の形態の機能が実現される場合にも本発明に含まれる。
【０１０５】
【発明の効果】
本発明は上述したように、本発明によれば、画像入力手段から入力される同一の画像データに対して、複数の異なる空間フィルタ特性に対応させて複数の空間フィルタ演算を時系列的に行うようにしたので、時系列で入力された画像データに対して、予め定められた複数の異なる幾何学的な特徴を単一の回路で効率よく逐次抽出することができる。また、時間的に変調された空間フィルタ出力をセンサ出力タイミングと連動させることができ、撮像された画像について異なる空間フィルタ処理を効率よく、かつ安定して行うようにすることができる。
【０１０６】
また、本発明のその他の特徴によれば、光電変換特性を有する画素アレイと、上記画素アレイから画像データを読み出すための読み出し手段とを含むセンサ領域と、上記センサ領域から読み出された同一の画像データに対して、複数の異なる空間フィルタ特性に対応させて複数の空間フィルタ演算を時系列的に行う画像処理手段とを同一半導体チップに形成したので、複数の回路を同一の半導体チップ上に形成することによる歩留まりの悪化を防ぎ、かつ同一半導体チップに形成することによるメリットである、少面積化、低電力化、低価格化等のメリットを得ることができる。
【図面の簡単な説明】
【図１】第１の実施の形態に係る撮像信号処理回路の要部構成を示すブロック図である。
【図２】センサ回路と前処理回路の構成例を示す図である。
【図３】空間フィルタ回路要素の構成例を示す図である。
【図４】本発明で用いる制御回路のブロック構成例を示す図である。
【図５】本発明の撮像信号処理回路を搭載した被写体認識を行う撮像装置の構成例を示す図である。
【図６】第２の実施の形態に係る撮像信号処理回路の要部構成を示す図である。
【図７】第３の実施の形態に係る撮像信号処理回路の要部構成を示す図である。
【図８】センサ回路の要部構成例を示す図である。
【図９】センサ回路での間引き読み出し時のタイミング制御を示す図である。
【符号の説明】
１ 空間フィルタ回路
２ 制御回路
３ センサ回路
３ｈ 水平シフトレジスタ
３ｔ 転送回路
３ｖ 垂直シフトレジスタ
４ 前処理回路
５ デマルチプレクサ回路
６ ブロックメモリ[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to an image processing apparatus.as well asThe present invention relates to an image input device, and more particularly to a circuit suitable for use in sensor signal processing for processing image signals from an image sensor such as a CMOS sensor and performing image processing such as recognition, detection, and extraction of a subject.
[0002]
[Prior art]
Conventionally, various image sensors to which an image processing function is added by integrating an analog processing circuit in an image sensor as a solid-state image sensor, for example, a semiconductor chip in which a solid-state image sensor and a spatial filter process are integrated. (Japanese Patent Laid-Open No. 2000-13694, Japanese Patent Laid-Open No. 11-266002) and the like are known.
[0003]
On the other hand, as an attempt to add an initial visual function of a living body to an image sensor, there is a study of a silicon retina by Mead et al. (C. Mead, “Analog VLSI and Neural Systems”, Addison Wesley Pub., 1989, US Pat. In addition, an artificial retina LSI (Volume 53, No.2, pp.178- of the Institute of Image Information and Television Engineers) that integrates a pixel array, a pixel access control circuit, a multiplexer, and a weight control circuit on a single chip to realize a predetermined filtering function. 183, 1999).
[0004]
Hereinafter, the operation of the CMOS sensor circuit will be described as a conventional technique. FIG. 8 is a schematic circuit diagram (CMOS sensor drive circuit diagram) of a sensor circuit according to Japanese Patent Laid-Open No. 11-196332. The sensor circuit outputs a video signal to the spatial filter circuit of the image signal processing circuit described above.
[0005]
In FIG. 8, PN junction photodiodes PD11 to PD22... And transfer MOS transistors ST11 to ST22 connected to the cathodes constitute a photoelectric conversion element. The outputs V1 to V8... Are sequentially output to “H” by the vertical selection circuit VSR to sequentially activate the vertical selection lines HL1, HL2,... And the outputs H1 to H8. Each horizontal transfer switch MOS transistor HT1 to HT8... Is turned on, and image charges accumulated in each pixel are sequentially read out from each vertical output line HV1 to HV8 to the output line HOL in time series, and output via the amplifier AMP. Is done.
[0006]
The reset MOS transistor RES is turned on to reset the output line HOL every time the pixel charge is output. Further, the photodiodes PD11, PD12, PD21, PD22, PD31... Connected to the vertical output lines HV1, HV2 of the photodiodes PD11 to PD22... Middle vertical output lines HV1 to HV8. Shaded.
[0007]
FIG. 9 shows a timing chart for thinning readout of the solid-state imaging device.
The readout pixels are read out by skipping adjacent pixels one by one in a horizontal and vertical arrangement of photoelectric conversion elements in the effective signal range.
[0008]
First, the output V1 is set to “H” by the vertical selection circuit VSR, and the vertical selection line HL1 is set to “H”. Subsequently, the outputs H1, H2, H3, H5, H7... Of the horizontal selection circuit HSR are sequentially set to “H”, and the horizontal transfer switch MOS transistors HT1, HT2, HT3, HT5, HT7. Image charges accumulated in each pixel are sequentially read out from the HV1, HV2, HV3, HV5, HV7... To the output line HOL in time series and output through the amplifier AMP.
[0009]
At that time, the outputs H1, H2, H3, H5, H7,... Of the horizontal selection circuit HSR are sequentially set to “H” and then the reset MOS transistor RES is turned on to reset the output line HOL each time the pixel charges are read. The influence of adjacent pixels is eliminated.
[0010]
In FIG. 9, the output AMP of the amplifier AMP is the “H” signal because the reference signal output photoelectric conversion element group of the photodiodes PD11 and PD12 of the pixel is shielded from light, and the first effective signal of PD13, PD15, and PD17. The low potential side level is output according to the charge of the output photoelectric conversion element group. In addition to such thinning readout, block readout for reading an arbitrary range from the entire image is similarly performed.
[0011]
Further, the thinning interval can be arbitrarily changed. A conventional example in which a spatial filter is applied after reading as described above is disclosed in Japanese Patent Laid-Open No. 2000-13694.
[0012]
[Problems to be solved by the invention]
However, in the above conventional example, there is a problem that a plurality of different geometric features determined in advance cannot be efficiently and sequentially extracted from image data input in time series.
[0013]
In addition, in order to extract a plurality of different geometric features determined in advance by an array circuit in which feature extraction circuits are arranged in parallel spatially, the time-series image signals from the sensors are spatially separated and individually separated. It was necessary to input to the corresponding feature extraction circuit. In addition, it is necessary to associate the sensor output with the address of the feature extraction circuit on the array circuit.
[0014]
In view of the above-described problems, the present invention is to enable a single circuit to efficiently and sequentially extract a plurality of different predetermined geometric features from image data input in time series. The first purpose.
Another object of the present invention is to perform different spatial filter processing efficiently and stably for a captured image by linking the temporally modulated spatial filter output with sensor output timing.
[0015]
[Means for Solving the Problems]
  The image processing apparatus of the present inventionA sensor circuit that reads out an analog pixel signal corresponding to a specified address in an image, and a plurality of filter element circuits that execute analog signal processing corresponding to the set filter characteristics with respect to the pixel signal corresponding to each address And a set of one of a plurality of direction selectivity parameters and one of a plurality of scale selectivity parameters are sequentially selected, and a filter characteristic corresponding to the selected group is selected from the plurality of filter element circuits. For each of the selected sets, the address in the image is sequentially designated to the sensor circuit, and the analog pixel signal read out from the sensor circuit corresponding to the designated address Control circuit for controlling the filter element circuit corresponding to the address to processIt is characterized by having.
  Another feature of the present invention is thatSaidSensorcircuitWhen,The spatial filter circuit and the control circuitAre formed on the same semiconductor chip.
  In addition, other features of the present invention are:SaidThe control circuitSaidPixel signal readout controlsignalTheSaidOutput to sensor circuitTo specify the addresses in the image sequentiallyWithSaidRead controlsignalBased on the spatial filter characteristics control signalSaidOutput to spatial filter circuitTo set the corresponding filter characteristics in the plurality of filter element circuits.It is characterized by that.
  In addition, other features of the present invention are:SaidThe control circuitRead controlSynchronized to the signalDrive timingSignalThe spatial filterIt outputs to a circuit.
  In addition, other features of the present invention are:The spatial filter circuit is:Demultiplexer circuitTheHaveSaidThe demultiplexer circuitSaidFrom the sensor circuitAnalog pixelSignalCorresponds to the specified addressfilterElement circuitoutputDoIt is characterized by that.
  In addition, other features of the present invention are:Read from the sensor circuitBlock memory that temporarily stores pixel group signalsTheHaveSaidThe spatial filter circuitThe block memoryFromtransferPixel signalProcessIt is characterized by that.
[0016]
An image input apparatus according to the present invention includes any one of the image processing apparatuses described above.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
FIG. 1 is a block diagram showing an overall configuration of an imaging signal processing circuit according to the present embodiment.
[0020]
In FIG. 1, a spatial filter circuit 1 performs a spatial filter operation of predetermined different characteristics in a time series at points corresponding to each pixel position on a sensor of a video signal input from a sensor circuit 3. It has a function of extracting different geometric image feature amounts for a predetermined size region centered on each point.
[0021]
The control circuit 2 outputs a predetermined read control pulse signal to the sensor circuit 3 according to a predetermined timing as described later, and also outputs a timing pulse signal synchronized with the read control pulse signal to the spatial filter circuit 1. By outputting, the output from the sensor circuit 2 and the processing of the spatial filter circuit 1 can be performed synchronously. Detailed control of the spatial filter circuit 1 will be described later.
[0022]
Next, an example of image signal reading and its pre-processing will be described in detail with reference to FIG.
In the photoelectric conversion device 10 in which the photoelectric conversion elements are arranged in a matrix, each horizontal line is sequentially activated by the vertical selection circuit VSR11, and a reference signal output photoelectric conversion element group and an effective signal output photoelectric conversion element by the horizontal selection circuit HSR12. Group image signals are sequentially output via an amplifier AMP.
[0023]
The output of the solid-state imaging device configured as described above is input to the reference voltage generator 14 during the output period of the reference signal output photoelectric conversion element group, and generates a reference voltage. In this case, when the reference signal output photoelectric conversion element group has two pixels, for example, the average value is set as the reference voltage.
[0024]
Next, during the output period of the effective signal output photoelectric conversion element group, it is input to the clamp circuit 13 and is clamped to the reference voltage of the reference voltage generator 14 so that the photoelectric charge voltage of the effective signal output photoelectric conversion element group is The reference voltage corresponding to the dark charge on the same horizontal line of the photoelectric conversion device is subtracted to obtain an image signal corresponding to the photocharge.
[0025]
In this case, the reference voltage of the reference voltage generator 14 is reset after reading out one horizontal line, and the reference voltage is generated again by the reference signal output photoelectric conversion element group for reading out the horizontal line. A difference is taken from the photoelectric charge voltage of the photoelectric conversion element group for use, and a substantial image signal is output. The output of the clamp circuit 13 is converted to a digital signal by the A / D converter 15 and subjected to a spatial filter process described below.
[0026]
<Configuration and operation of spatial filter circuit>
FIG. 3 is a diagram illustrating a simplified configuration example of the spatial filter element used in the present embodiment. Here, the output at the nearest neighboring pixel position (p ± 1, q ± 1) of the pixel position (p, q) (h_r(p ± 1, q ± 1), h_i(p ± 1, q ± 1)) are displayed, but in reality, these circuit elements are two-dimensionally arranged.
[0027]
In FIG. 3, h_r(p, q), h_i(p, q) represents a real part and an imaginary part of the filter output (described below) corresponding to the pixel position (p, q) on the sensor, respectively. The input / output control of the spatial filter element circuit will be described later.
[0028]
The spatial filter used here is a so-called Gabor type filter that performs output similar to the Gabor filter or Gabor filter output. Spatial frequency selectivity in a predetermined direction, that is, direction selectivity (sensitivity to edge direction components in a predetermined range in the image) And scale selectivity (sensitivity to a predetermined range of size or spatial frequency geometric features in the image).
[0029]
In the present embodiment, Gabor wavelet conversion is performed on an input image by multiscale processing using a Gabor type filter. As for the method and circuit for performing Gabor wavelet conversion, the documents IEEE Trans. On Acoustics, Speech, and Signal Processing, vol. 36, pp. 1169-1179 (1988), and IEEE Trans. On Circuits and Systems-I: Please refer to Fundamental Theory and Applications, vol.45, pp.121-132. (1998).
[0030]
The spatial filter used in this embodiment can control the selectivity by the characteristics of circuit elements (conductance, amplifier gain, etc.). In FIG. 3, a point (p, q) in the input image is centered. Reference 1 (BEShi, "2D Focal Plane Steerabale and Scalable Cortical Filters", Proc. Of Seventh International Conference on Microelectronics for Neural, Fuzzy, and Bio -Inspired Systems, pp.232-239. 1999).
[0031]
In FIG. 3, the elements indicated by trapezoids are typically transconductance amplifiers using five MOS transistors (for example, a configuration in which the gain is variable is disclosed in Japanese Patent Laid-Open Nos. 7-235839 and 11-68477). Described in the Gazette and the like).
[0032]
Each resistive element is composed of a CMOS transmission gate (for example, composed of two CMOS transistors; Reference 2: BEShi "Subthreshold Current Mode design of Gabor-type CNN Image Filters, Proc. European Conference on Circuit Theory and Design , pp.1163-1169, 1999), the label G of the resistance element represents conductance, and the label G of each transconductance type amplifier represents gain.The filter circuit may operate each transistor above a threshold value. The operation may be performed below a threshold value.
[0033]
Next, input of image data to the filter circuit will be described. The pixel signal is sent from the sensor circuit driven by the readout control pulse signal output from the control circuit 2 at a predetermined timing to the memory element C in FIG._mSent to.
[0034]
This memory element and transconductance amplifier G_sIs an input part of the spatial filter element circuit whose address (pixel position) is (p, q), and exists for each address (each filter element circuit). The filter output is read out from the input section (memory element C) of each filter element circuit._m) Is performed after all the image signals have been transferred to.
[0035]
The pixel signal is a memory element C coupled to the transconductance amplifier Gs of the input unit._mWhen the output from the spatial filter circuit is read by a predetermined filter output read signal from the control circuit, the contents of the memory element are rewritten. The data held in the memory element is voltage-current converted by the transconductance amplifier Gs and input as u (p, q).
[0036]
Spatial frequency (w_x0, w_y0) And a direction selectivity and a scale selectivity with a predetermined width. W_x0 = atan (G_2x/ G_1x), w_y0= atan (G_2y/ G_1y). The central direction θ of direction selectivity is θ = atan (w_y0/ w_x0).
[0037]
<Spatial filter circuit control overview>
Next, a description will be given of a process of extracting a geometric feature in a local region centered on a predetermined sampling point of an input image by the filter array circuit in which the spatial filter elements described above are two-dimensionally arranged. The filter characteristics and drive timing control signal of the spatial filter array circuit are collectively output from the control circuit of FIG. 1 to each filter element circuit (FIG. 3) of the spatial filter array circuit.
[0038]
That is, the filter characteristics and the drive timing control signal collectively control (update) the filter characteristics of each filter element circuit. The sensor circuit described above performs thinning-out scanning for input image data at a predetermined interval, and outputs image signal data.
[0039]
As the drive timing control signal for updating the characteristics of the spatial filter circuit, a control signal synchronized with the output timing from the spatial filter circuit array for one screen is output from the control circuit 2.
[0040]
When a readout control signal (sensor selection line signal in FIG. 1) synchronized with this control signal is output to the vertical / horizontal selection circuit of the sensor circuit, image signal data at a corresponding address is read out from the sensor.
[0041]
The timing control signal (filter selection line signal in FIG. 1) is output to a predetermined spatial filter element circuit in the filter array circuit corresponding to the sensor selection line signal (address in the input image) via the demultiplexer. The image signal data at the address specified by the sensor selection line signal is transferred to the corresponding spatial filter element circuit.
[0042]
Each spatial filter element circuit is configured so that each time a filter selection line signal is input, the spatial filter characteristic becomes one of a set of parameter values to which the direction selectivity and scale selectivity belong. Each G₀, G_1x, G_1y, G_2x, G_2yIs set (by the filter characteristic control signal from the control circuit 2), and image data from the sensor is input.
[0043]
Specifically, as each parameter value, the scale parameter is set to Ω using the above spatial frequency selectivity parameter.^-1= [W_x0 ²+ w_y0 ²]^-0.5When each selectivity parameter Ω, θ is given, the characteristic value that each resistor and amplifier should set is α²= G_{2 (max)}[1 + cot²Ω]^0.5, Δ = λ / (αΩ),
G_1x= α²cos (Ωcosθ), G_2x= α²sin (Ωcosθ), G_1y= α²cos (Ωsinθ),
G_2y= α²sin (Ωsinθ), G₀= α²[(ΔΩ)²+ 4-2 cos (Ωcosθ) -2cos (Ωsinθ)]
(Reference 1).
[0044]
After the transfer of the image data to all the spatial filter circuit elements for the predetermined filter characteristics is completed, the spatial filter processing result having the set characteristics is read out.
[0045]
<About Gabor filter and Gabor wavelet conversion>
Next, Gabor (type) filtering performed by each spatial filter will be described. The Gabor function has a shape obtained by modulating a sine wave having a constant directional component and a spatial frequency with a Gaussian function (non-Gaussian function in the case of a Gabor type filter) as given by Equation (1), and has a scaling level described below. The index m and the direction component index n set the characteristics.
[0046]
This set of filter functions as wavelets have similar function shapes and differ in main direction and magnitude. This wavelet must be localized in the spatial frequency domain and the real space domain, with minimal simultaneous uncertainty regarding position and spatial frequency, and be the most localized function in both real and frequency space. (J, G. Daugman (1985), Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters, Journal of Optical Society of America A, vol.2, pp. 1160 -1169).
[0047]
[Expression 1]

[0048]
Where (x, y) is the position in the image, a is the scaling factor, θ_nRepresents the directional component of the filter, W is the fundamental spatial frequency, σ_Xσ_YIs a parameter that gives the magnitude of the spread of the filter function in the x and y directions. However, on the circuit configuration shown in FIG. 3, the shape of the Gaussian function of equation (1) is approximated by Laplacian.
[0049]
In this embodiment, θ as the direction selectivity parameter_nTakes values of 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees and 150 degrees in 6 directions, a is 2, and m as a scale selectivity parameter is given as an integer having a value from 1 to 3 . Note that in the actual circuit configuration, such selectivity is realized by appropriately controlling the characteristic values (conductance value and gain of the transconductance amplifier) of the circuit elements in FIG.
[0050]
Gabor filter kernel (g in equation (1))_mn) Varies depending on the scale index m, so Gabor filters having different scale indexes have different size selectivity, but here it is assumed that the circuit array has an infinite extent as shown in FIG. In this case, it is used that the transfer function has spatial frequency selectivity (scale selectivity) (Reference 1).
[0051]
At the input time t of the drive timing control signal described above, the characteristic parameter of the selected filter is set to m (t) with respect to scale selectivity and n (t) with respect to direction selectivity. G for each filter with this selected spatial filter characteristic_{m (t) n (t)}2D convolution operation of (x, y) and input grayscale image is performed (in practice, however, in the circuit element as shown in FIG._{m (t) n (t)}Gabor wavelet transform is performed by obtaining a conversion output equivalent to a convolution operation with a function shape of (x, y) as Laplacian. That is,
[0052]
[Expression 2]

[0053]
Where I is the input image, W_{m (t) n (t)}Is a Gabor wavelet transform coefficient. W_{m (t) n (t)} A set of (m (t) = 1,2,3; n (t) = 1, ..., 6) is obtained at each point as a feature vector. “*” Indicates that a complex conjugate is taken.
[0054]
The output of the spatial filter array of circuit elements shown in FIG._r(p, q), h_i(p, q) is W_{m (t) n (t)} ≒ h_r(p, q) + j h_iIt is related as (p, q).
[0055]
Each spatial filter element circuit outputs a wavelet transform coefficient value obtained by performing a product-sum input of a distribution weight coefficient and image data. That is, for all screen data from the sensor output, the output from the spatial filter circuit element giving a predetermined filter characteristic is read in time series, and the filter output giving a series of different filter characteristics in the same manner As a result, the Gabor wavelet transform of equation (2) is used as the time series output of the entire spatial filter array (a set of values corresponding to Wmn over the range of possible values of the index m, n of equation (2)) Has been done.
[0056]
The time-series output result for each filter characteristic updated as follows is once stored individually as a two-dimensional array data in a plurality of (or a single batch) temporary storage means. (For example, image recognition processing) is performed. For example, when the subsequent processing module that receives the spatial filter output is a neural network in which a plurality of arithmetic elements are connected in parallel and hierarchically, the above-described two-dimensional array data (for example, converted into a voltage signal) ) Normally, it is output to the neural elements in the input layer at the same time.
[0057]
<Transfer control of image data from sensor circuit>
Hereinafter, description will be given with reference to FIG. 1 again. Each time the control circuit 2 updates the characteristic parameter (m, n) of the Gabor type spatial filter array circuit 1, the control circuit 2 outputs a corresponding resistance value in the spatial filter circuit element and a control signal for the amplifier gain.
[0058]
Specifically, the control circuit 2 uses a gate voltage control signal (V) of a transistor constituting a CMOS transmission gate circuit constituting each resistor as a filter characteristic control signal._G1x, V_G1y, V_G0), And a gain control signal (V_G2x, V_G2y) Is output so that the characteristics of each spatial filter circuit element are updated collectively.
[0059]
Hereinafter, the configuration and the like of the control circuit will be described with reference to FIG. The control circuit 2 stores a set of unique conductance values and amplifier gain values (G, G) for the Gabor type filter scale and direction selection index set (m, n) in the internal storage means 43, respectively.₀, G_1x, G_1y, G_2x, G_2y) Is stored as a table, and the control signal corresponding to each index set is V_{con, (mn)}Is inputted from the direction / scale selection signal generator 41, the gate control signal generation circuit 42 gives a control signal V giving a conductance value and an amplifier gain value shown below._G1x, V_G1y, V_G2x, V_G2y, V_G0Is output to each resistor of each spatial filter circuit element shown in FIG.
[0060]
That is, V_{con, (mn)}G corresponding to_{1x, (mn)}, G_{2x, (mn)}, G_{1y, (mn)}, G_{2y, (mn)}, G_{0, (mn)}Are given as follows, in the memory, the control signal level V corresponding to each_G1x, V_G1y, V_G2x, V_G2y, V_G0Are stored corresponding to the index set (m, n). G_{1x, (mn)}, G_{2x, (mn)}, G_{1y, (mn)}, G_{2y, (mn)}, G_{0, (mn)}The value of the center direction θ_n, Center spatial frequency Ω_mUsing,
G_{1x, (mn)}= α_m ²cos (Ω_mcosθ_n), G_{2x, (mn)}= α_m ²sin (Ω_mcosθ_n),
G_{1y, (mn)}= α_m ²cos (Ω_msinθ_n), G_{2y, (mn)}= α_m ²sin (Ω_msinθ_n),
G_{0, (mn)}= α_m ²[(Δ_mΩ_m)²+ 4-2 cos (Ω_mcosθ_n) -2cos (Ω_msinθ_n)]
It is expressed as here,
α_m ²= G_{2 (max)}[1 + cot²Ω_m]^0.5, Δ_m= Λ / (α_mΩ_m)
It is.
[0061]
The control signal lines to the resistors and the amplifier are wired to predetermined transistors of the circuit elements in FIG. 3, but are not shown in order to avoid complication.
[0062]
The control circuit 2 applies each horizontal / vertical selection line to the horizontal selection circuit 3h and the vertical selection circuit 3v in the sensor circuit 3 so as to be synchronized with the output timing of the control signal from the direction / scale selection signal generator. The selection line signal of the line to be sequentially set to “H” is output.
[0063]
Specifically, the updated selection signal V_{con, (mn)}In synchronization with this, the activation signal of the horizontal selection line signal and the vertical selection line signal is output to the sensor side, and the sensor circuit should be sequentially set to “H” when this activation signal is input. The line selection line signal is output, and the pixel signal of the pixel in which both the horizontal selection line and the vertical selection line are active is output from the sensor to the above-described spatial filter circuit.
[0064]
The pixel signal read out as a time series signal in this way is transferred to the input section of each spatial filter circuit element (FIG. 3) selected by the demultiplexer circuit 5 as follows. The demultiplexer circuit 5 may have a circuit configuration that is normally used (for example, a switch array and its activation circuit), and is input by a preprocessed pixel signal and a selection line signal and determined by the selection line signal. The pixel signal corresponding to the Gabor type spatial filter circuit element corresponding to the read pixel position in the input image is output.
[0065]
Next, reading of the output from the Gabor type spatial filter array circuit 1 will be described. Reading is performed after the scale parameter m and the direction component parameter n to be set by the direction / scale selection signal generator of the control circuit 2 are updated and all the pixel signals to be input to the respective filter circuit elements are transferred. .
[0066]
For example, the transfer end timing to the filter circuit element of the image data read according to the above parameters (m, n) of the Gabor type filter is detected by inputting the last selection line signal, and then the device characteristics The spatial filter output is read out after a predetermined delay time determined by, and stored in the temporary storage means as a plurality of two-dimensional array data (spatial filter array output for each same characteristic), or as time series data as it is You may output to a processing module.
[0067]
Since the spatial filter output is read out after a predetermined delay time from the filter selection line signal output synchronized with the sensor selection line signal from the control circuit 2, the image signal output from the sensor always maintains a constant temporal correspondence. The spatial filter output can be taken out.
[0068]
Further, the above-described spatial filter output and sensor output can be associated with each other while reducing the circuit scale by appropriate timing control for reading data from the sensor and controlling the spatial filter characteristics.
[0069]
When the subsequent processing module is a parallel processing device such as a neural network, the read time-series filter output is branched and output to each element of the input stage by a demultiplexer circuit, etc. Introduce a sample hold circuit or temporary storage memory element that holds the value of the input data, or write it as a two-dimensional array data in a predetermined block memory element, and then output it collectively to the input element array of the parallel processing unit May be.
[0070]
As described above, in the present embodiment, address control and transfer control to the sensor circuit are performed based on the control timing of the spatial filter circuit, but other processing circuits other than the spatial filter described above, for example, image recognition A similar control may be performed based on the intermediate processing output timing of the circuit.
[0071]
Next, the configuration of an imaging apparatus (image input apparatus) equipped with the imaging signal processing circuit that performs the spatial filtering described above will be described.
A case in which focusing on a specific subject, color correction of a specific subject, and exposure control are performed by mounting a pattern recognition (detection) device using the imaging signal processing circuit according to the configuration of the present embodiment on an imaging unit. This will be described with reference to FIG.
[0072]
The imaging device 9 in FIG. 5 includes an imaging optical system 92 including a photographing lens and a zoom photographing drive control mechanism, a CMOS image sensor 93, an imaging parameter measuring unit 94, and an imaging that performs a spatial filter operation according to the present embodiment. A signal processing circuit 95, a storage means 96, a control signal generating means 97 for generating a control signal for controlling an imaging operation, an imaging condition control, a display display 98 that also serves as a finder such as an EVF, a strobe light emitting means 99, and a recording medium 910 Furthermore, subject recognition means 911 and the like are further provided. The subject recognition unit 911 has a function of detecting a pattern of a specific category from an image using a Gabor type wavelet conversion output from the imaging signal processing circuit.
[0073]
In this imaging apparatus, for example, the subject recognition unit 911 detects a face image of a person registered in advance (detection of an existing position and size) from a captured video. When the position and size information of the person is input to the control signal generation means 97, the control signal generation means 97, based on the output from the imaging parameter measurement means 94, focuses control, exposure condition control, white Generates control signals for optimal balance control.
[0074]
By using the pattern detection (recognition) device using the imaging signal processing circuit according to the present embodiment in the imaging device as described above, the subject detection (recognition) function can be achieved with a small circuit scale, low power consumption and high speed. (Real time). Further, based on the result, it was possible to perform the optimum control (AF, AE, etc.) of detection of a person or the like and photographing based on the detection.
[0075]
In the present embodiment, the sensor circuit 3, the control circuit 2, the preprocessing circuit 4, and the spatial filter circuit 1 shown in FIG. 1 are formed on separate semiconductor chips or on the same semiconductor chip. Applies to the case.
[0076]
However, the present embodiment is more effective when the respective circuits shown in FIG. 1 are formed on the same semiconductor chip by a CMOS process or the like.
[0077]
In the present embodiment, a plurality of spatial filter operations in a plurality of different spatial filter characteristics are not performed in parallel with a plurality of spatial filter circuits provided for image data for one screen of the photoelectric conversion element array. This is performed in time series by one spatial filter.
[0078]
When the sensor circuit 3 and the spatial filter circuit 1 are formed on the same semiconductor chip, if either the sensor circuit 3 or the spatial filter 1 is defective, the chip cannot be used and the yield is poor. Become. In this case, when a plurality of spatial filters are provided, the chip area becomes large, resulting in a disadvantage that the yield is very poor. On the other hand, when processing is performed in time series with one spatial filter circuit as in the present embodiment described above, the chip area can be reduced and the yield can be improved.
[0079]
In this way, a single spatial filter circuit that performs a plurality of different processes in time series is formed on the same semiconductor chip together with the sensor circuit 3 and the like, thereby forming a plurality of circuits on the same semiconductor chip. It is possible to prevent the deterioration of the yield due to the above, and to obtain the merits of forming the same semiconductor chip, that is, the merits such as the reduction in area, the reduction in power consumption, the reduction in price, and the like.
[0080]
(Second Embodiment)
FIG. 6 is a block diagram showing a main configuration of the present embodiment. Here, the memory element C at the input of the spatial filter circuit element of FIG._mIs unnecessary. Of the sensor circuit output, a necessary portion corresponding to a predetermined spatial filter circuit element accessed in time series is temporarily recorded in the block memory, and then each pixel data of the block memory is collectively displayed as a voltage signal (the circuit of FIG. 3). In memory element C_mIs used (in time series), the spatial filter processing as described above is performed.
[0081]
In the present embodiment, the control circuit 2 sends the scale control signal corresponding to the scale parameter m out of the spatial filter characteristic parameters (m, n) to the sensor circuit side together with the address selection signal, as in the first embodiment. Then, the horizontal / vertical selection circuit of the sensor circuit stores, in the block memory 6, image data of a local area (local area sampled by thinning) centered on each sampling pixel for a predetermined area size according to the scale control signal. Let By doing so, it is possible to reduce the load on the control circuit 2 that outputs the read timing control signal to the sensor circuit.
[0082]
The block memory 6 functions as a primary storage element. According to the timing signal from the control circuit 2, image data in a local region (size depends on the scale parameter) centered on each sampling position is the position of the sampling point. Is sent to each spatial filter circuit element corresponding to.
[0083]
At this time, the image data stored in the block memory has the image signal data (after pre-processing) corresponding to the sampling point position in the subsequent stage every time the sampling point position for thinning is updated by the control signal from the control circuit 2. The image data is transferred to the spatial filter array circuit, and the image signal from the sensor at the above position (address) is input, and the image data at the corresponding address on the block memory is rewritten. Is done.
[0084]
Thereafter, it is output as an image signal in time series to the spatial filter circuit element by the control signal of the control circuit 2. Note that the output reading from the spatial filter circuit and others are the same as those in the first embodiment.
[0085]
Also in this case, the sensor circuit 3, the control circuit 2, the preprocessing circuit 4, the block memory 6, and the spatial filter circuit 1 shown in FIG. 6 are formed on separate semiconductor chips or on the same semiconductor chip. Applies to when forming.
[0086]
However, by forming each circuit shown in FIG. 6 on the same semiconductor chip by a CMOS process or the like, this embodiment can exhibit more effects.
[0087]
In the present embodiment, a plurality of spatial filter operations in a plurality of different spatial filter characteristics are not performed in parallel with a plurality of spatial filter circuits provided for image data for one screen of the photoelectric conversion element array. This is performed in time series by one spatial filter.
[0088]
When the sensor circuit 3 and the spatial filter circuit 1 are formed on the same semiconductor chip, if either the sensor circuit 3 or the spatial filter 1 is defective, the chip cannot be used and the yield is poor. Become. In this case, when a plurality of spatial filters are provided, the chip area becomes large, resulting in a disadvantage that the yield is very poor. On the other hand, when the processing is performed in time series with one spatial filter circuit as in the present embodiment described above, the chip area can be reduced and the yield can be improved.
[0089]
In this manner, by forming one spatial filter circuit 1 that performs a plurality of different processes in time series on the same semiconductor chip together with the sensor circuit 3 and the like, a plurality of circuits are formed on the same semiconductor chip. Therefore, it is possible to prevent the deterioration of the yield due to this, and to obtain the merits of forming on the same semiconductor chip, that is, the merits such as the reduction in area, the reduction in power consumption, and the reduction in price.
[0090]
(Third embodiment)
FIG. 7 shows the main configuration of the present embodiment. Here, timing control characteristics of the sensor circuit, that is, modulation timing control on the spatial filter circuit side is performed using a read control signal used in a normal CMOS sensor. However, it is assumed that the thinning-out reading control as shown in the first embodiment is performed.
[0091]
In this way, it is not necessary to change the readout timing control for the CMOS sensor circuit or the internal wiring of the transfer circuit or the like, and the spatial filtering operation can be stably performed using the existing CMOS sensor chip or the like. .
[0092]
The control circuit 2 receives a selection line signal at the time of sensor reading, obtains address information of the image signal, and controls transfer of the image signal to the spatial filter circuit element at the corresponding position.
[0093]
At this time, the characteristic control signal of each circuit element (resistor, transconductance amplifier) shown in FIG. However, the update is performed after the reading operation from the sensor is completed. For example, it may be updated each time thinning out to be performed on the input image is completed. Further, after the update, (thinning-out) readout scanning is performed again in the sensor circuit.
[0094]
The pixel signal is a memory element C coupled to the transconductance amplifier Gs of the input unit as in the first embodiment._mWhen the output from the spatial filter circuit is read out temporarily as a voltage signal, the contents of the memory element are rewritten. Other operations and configurations are the same as those in the first embodiment.
[0095]
Also in the third embodiment, the sensor circuit 3, the control circuit 2, the preprocessing circuit 4, and the spatial filter circuit 1 shown in FIG. 7 are formed on separate semiconductor chips or the same semiconductor. It is applied to the case of forming on a chip.
[0096]
However, by forming each circuit shown in FIG. 7 on the same semiconductor chip by a CMOS process or the like, this embodiment can exhibit more effects.
[0097]
In the present embodiment, a plurality of spatial filter operations in a plurality of different spatial filter characteristics are not performed in parallel with a plurality of spatial filter circuits provided for image data for one screen of the photoelectric conversion element array. This is performed in time series by one spatial filter.
[0098]
When the sensor circuit 3 and the spatial filter circuit 1 are formed on the same semiconductor chip, if either the sensor circuit 3 or the spatial filter 1 is defective, the chip cannot be used and the yield is poor. Become. In this case, when a plurality of spatial filters are provided, the chip area becomes large, resulting in a disadvantage that the yield is very poor. On the other hand, when processing is performed in time series with one spatial filter circuit 1 as in the present embodiment described above, the chip area can be reduced and the yield can be improved.
[0099]
In this manner, by forming one spatial filter circuit 1 that performs a plurality of different processes in time series on the same semiconductor chip together with the sensor circuit 3 and the like, a plurality of circuits are formed on the same semiconductor chip. Therefore, it is possible to prevent the deterioration of the yield due to this, and to obtain the merits of forming on the same semiconductor chip, that is, the merits such as the reduction in area, the reduction in power consumption, and the reduction in price.
[0100]
(Another embodiment of the present invention)
The present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device.
[0101]
Further, in order to operate various devices so as to realize the functions of the above-described embodiments, a transmission medium such as the Internet is transmitted from a storage medium to an apparatus connected to the various devices or a computer in the system. The program implemented by operating the various devices according to the program stored in the computer (CPU or MPU) of the system or apparatus is supplied via the software program code for realizing the functions of the above-described embodiments Are included in the scope of the present invention.
[0102]
In this case, the program code of the software itself realizes the functions of the above-described embodiments, and the program code itself and means for supplying the program code to the computer, for example, the program code are stored. This storage medium constitutes the present invention. As a storage medium for storing the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
[0103]
Further, by executing the program code supplied by the computer, not only the functions described in the above-described embodiments are realized, but also the OS (operating system) or other operating system in which the program code is running on the computer. It goes without saying that the program code is also included in the embodiment of the present invention even when the functions described in the above-described embodiment are realized in cooperation with application software or the like.
[0104]
Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, the CPU provided in the function expansion board or function expansion unit based on the instruction of the program code The present invention also includes a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing.
[0105]
【The invention's effect】
As described above, according to the present invention, according to the present invention, a plurality of spatial filter operations are performed in time series corresponding to a plurality of different spatial filter characteristics with respect to the same image data input from the image input means. As described above, a plurality of different geometric features determined in advance can be efficiently and sequentially extracted from image data input in time series. In addition, the temporally modulated spatial filter output can be linked to the sensor output timing, so that different spatial filter processing can be efficiently and stably performed on the captured image.
[0106]
According to another aspect of the invention, a sensor region including a pixel array having photoelectric conversion characteristics and a reading unit for reading image data from the pixel array, and the same read from the sensor region Image processing means for performing a plurality of spatial filter operations in time series corresponding to a plurality of different spatial filter characteristics for image data is formed on the same semiconductor chip. It is possible to prevent the deterioration of the yield due to the formation, and to obtain the advantages such as a reduction in area, a reduction in power consumption, and a reduction in price, which are the merits of forming on the same semiconductor chip.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a main configuration of an imaging signal processing circuit according to a first embodiment.
FIG. 2 is a diagram illustrating a configuration example of a sensor circuit and a preprocessing circuit.
FIG. 3 is a diagram illustrating a configuration example of a spatial filter circuit element.
FIG. 4 is a diagram illustrating a block configuration example of a control circuit used in the present invention.
FIG. 5 is a diagram illustrating a configuration example of an imaging apparatus that performs subject recognition equipped with an imaging signal processing circuit of the present invention.
FIG. 6 is a diagram illustrating a main configuration of an imaging signal processing circuit according to a second embodiment.
FIG. 7 is a diagram illustrating a main configuration of an imaging signal processing circuit according to a third embodiment.
FIG. 8 is a diagram illustrating a configuration example of a main part of a sensor circuit.
FIG. 9 is a diagram illustrating timing control at the time of thinning readout in the sensor circuit.
[Explanation of symbols]
1 Spatial filter circuit
2 Control circuit
3 Sensor circuit
3h Horizontal shift register
3t transfer circuit
3v vertical shift register
4 Pre-processing circuit
5 Demultiplexer circuit
6 block memory

Claims (7)

A sensor circuit that reads an analog pixel signal corresponding to a specified address in an image;
A spatial filter circuit including a plurality of filter element circuits that perform analog signal processing corresponding to the set filter characteristics with respect to the pixel signal of the address to which each corresponds,
A set of one of a plurality of direction selectivity parameters and one of a plurality of scale selectivity parameters are sequentially selected, and a filter characteristic corresponding to the selected set is set in the plurality of filter element circuits, and the selected For each of the sets, the filter element corresponding to the address corresponding to the analog pixel signal read from the sensor circuit corresponding to the designated address is sequentially specified for the sensor circuit. An image processing apparatus comprising: a control circuit that controls the circuit to perform processing. The image processing apparatus according to claim 1, wherein the sensor circuit , the spatial filter circuit, and the control circuit are formed on the same semiconductor chip. Wherein the control circuit is configured to sequentially specify the addresses in the image by outputting the read control signal of the pixel signal to the sensor circuit, the spatial filtering characteristic control signal based on the read control signal to the spatial filter circuit the image processing apparatus according to claim 1 or 2, characterized in that to set the corresponding filter characteristic by outputting the plurality of filter element circuits. The image processing apparatus according to claim 3 , wherein the control circuit outputs a drive timing signal synchronized with the readout control signal to the spatial filter circuit. The spatial filter circuit includes a demultiplexer circuit,
The demultiplexer circuit, an image processing apparatus according to an analog pixel signal to any one of claim 1 to 3, wherein the output to the filter element circuits corresponding to the designated address from the sensor circuit . A block memory for temporarily storing the signal of the pixel group read from the sensor circuit ;
The spatial filter circuit, an image processing apparatus according to claim 1 or 2, characterized in that processing the pixel signals transferred from the block memory. An image input apparatus characterized by having an image processing apparatus according to any one of claims 1-6.

Images