JP3630792B2 - Gaze detection device and optical device - Google Patents

Description

センサの読み出し領域が設定されたら、サブルーチンをリターンする（＃Ｅ１５）。
【００９８】
このように、全体の明るさで、瞳孔推定で推定した瞳孔領域を変えることによって、明るいときに瞳孔が小さくなっても無駄な領域を読み出すことがなく、さらに、暗い時も瞳孔がはみ出さない領域を設定することが出来る。
【００９９】
ここで、図６のステップ＃Ｅ０４の説明を行う。
【０１００】
眼鏡のフレームが画像内に入り込むと、黒く細長い形として現れることがある。これは、眼鏡のフレーム部分が全反射してしまうため、センサ上に照明の光が全くこなくなるからである。
【０１０１】
図８（ａ）に示したように、横位置ではセンサの上下方向の幅があまり大きくないので、眼鏡フレームの暗黒部分がセンサ外に出てしまったために問題にならなかったが、図８（ｂ）に示すように、縦位置では眼鏡のフレームの暗黒部分がセンサ内に入り込んでしまうため、瞳孔位置検出に悪影響を及ぼすことがある。そこで、眼鏡のフレームと瞳孔とを識別し、正しい瞳孔のみを検出する必要がある。そこで、眼鏡フレームの形状に着目し、ステップ＃Ｅ０２と＃Ｅ０３で検出した低輝度部分の形で判定を行う。
【０１０２】
具体的には、縦位置の場合、水平方向（センサ上ｘ軸方向）が短く、かつ、垂直方向（センサ上ｙ軸方向）が長い場合を判定する。図８（ｃ）のＡの位置を最低輝度として認識できた場合、これは瞳孔部分が正しく選ばれたので、低輝度部の水平方向の長さｘｄと垂直方向の長さｙｄは同程度の大きさになって、どちらかが大きかったり小さかったりしていない。ところが、もし、Ｂ部分を最低輝度として検出した場合、水平方向の長さｘｆは非常に短く、垂直方向の長さｙｆは極端に長い。この特徴から眼鏡フレームを識別する。
【０１０３】
眼鏡フレームの水平方向の閾値をｘｆｔｈ 、垂直方向の閾値をｙｆｃｈ 、低輝度部分の水平方向の長さをｌｄｋｘ、垂直方向の長さをｌｄｋｙ とすると、
ｌｄｋｘ ＜ｘｆｔｈ かつ ｌｄｋｙ ＞ｙｆｔｈ
をみたす場合、眼鏡フレームと判定する。
【０１０４】
具体的には、ｘfth ＝５，ｙfth ＝１０（縦位置の場合）で検出を行っている。図に合せて縦位置で説明を行ったが、横位置では水平方向と垂直方向が入れかわる。フレームの判定が垂直方向（縦位置）の長さだけで判定できないのは、画面全体が暗い場合、低輝度部分の大きさが大きくなってしまうことからである。
【０１０５】
眼鏡フレームと判定すると、ステップ＃Ｅ１３に分岐する。ここで、瞳孔部分がうまく推定できなかったため、領域制限のパラメータを初期化し、ブロックデータから求めた最低輝度値のデータも眼鏡フレーム部分の間違った値になっているのでデフォルト値に設定し直す（＃Ｅ１４）。
【０１０６】
この例では眼鏡フレームと判定された場合は領域制限をやめてしまうが、反対に眼鏡フレーム部分を取り除いた部分だけを計算するように領域の制限を行ってもよい。又、これとは別に、眼鏡フレーム部分を排除して瞳孔部分を推定するように再計算してもよい。
【０１０７】
処理が終了するとステップ＃Ｅ１５へ進み、サブルーチンをリターンする。
【０１０８】
このように、眼鏡フレームの特徴を判別し、眼鏡フレームを判別したときは所定の領域で視線演算を行うようにしたため、間違った領域を設定することがない。
【０１０９】
フローチャートでは、説明していないが、眼鏡フレームと判定した場合それを除く領域を設定するようにしても良いし、眼鏡フレーム領域を除いた領域で再度瞳孔位置を推定し直してもよい。
【０１１０】
このようにして、瞳孔位置の誤推定を排除できる。
【０１１１】
次に、図４のサブルーチン「読み出し＆逐次処理」について、図９のフローチャートを用いて説明する。
【０１１２】
このサブルーチン「読み出し＆逐次処理」がコールされると（＃ｘ０１）、まず、センサに読み出し開始通信を行う（＃ｘ０２）。次に、センサ出力をＡ／Ｄ変換する為にＡ／Ｄ変換の初期化を行う（＃ｘ０３）。次いで、逐次計算を行うためのパラメータの初期化や、制限された領域だけを読み出すためのパルス数の設定を行う（＃ｘ０４）。
【０１１３】
次に、これから読む水平ラインが読み出す領域を含むかどうかを判定し（＃ｘ０５）、読み出し領域外でならステップ（＃ｘ１９）分岐する。そして、ステップ＃ｘ１９で、水平ラインを１ライン読み飛ばすための、水平ライン読み飛ばしパルスを出力する。このパルスがセンサに送られるとセンサは水平の１ラインの読み出しを行わない。
【０１１４】
上記ステップ＃ｘ０５で、これから読む水平ラインが読み出す領域を含んでいる場合、ステップ＃ｘ０６へ分岐する。そしてここでは、１ラインの読み出しを行う。まず、１画素毎の読み出しパルスを送り、センサからの出力をＡ／Ｄ変換し所定のＲＡＭに一時的に格納する。読み出す水平ラインの中でも指定された領域外の部分はＡ／Ｄ変換を行わず、読み出しパルスを高速に出力することによって、指定された領域のデータだけを短時間で得ることができる。
【０１１５】
これらの動作はソフト的にタイミングを作って動作させる以外にも、ハード的にパルス出力，Ａ／Ｄ変換，データ格納を行う様な構成にしても良い。この場合、Ａ／Ｄ変換やパルス出力にソフトが介在する必要がないためその間、ソフトは特徴点の抽出を行うことができる。
【０１１６】
次に、カメラの位置が縦位置かどうかを判定し（＃ｘ０７）、縦位置なら読み出しライン数をＡレジスタに入れ（＃ｘ０８）、横位置なら（縦位置でないなら）、今まで検出した瞳孔エッジの総数を計算し、Ａレジスタに格納する（＃ｘ０９）。次いで、Ａレジスタが０かどうかを判定し（＃ｘ１１）、０で無いなら（＃ｘ１２）へ分岐し、サブルーチン「Ｐ像検出」を実行する（＃ｘ１２）。
【０１１７】
カメラ位置が縦位置の場合、少なくとも１ライン以上センサの読み出しが終了してから「Ｐ像検出」を行うようにしている。又、横位置の場合は、照明位置の関係から瞳孔位置よりＰ像の方が下に成ることが判っているので、無駄な演算を省くために瞳孔のエッジが少なくとも１つ検出されてからＰ像の検出を行うようにしている。
【０１１８】
次に、瞳孔エッジを検出するのに充分なライン数の読み出しが終了しているかを判定し（＃ｘ１３）、十分ならステップ＃ｘ１４に分岐する。そしてこのステップ＃ｘ１４では、サブルーチン「瞳孔エッジ検出」をコールし瞳孔エッジを検出する。
【０１１９】
画像データはメモリの制約上から数ライン分記憶できるだけで、その分のライン数が記憶されたら充分なライン数があると判定する。水平方向にのみ瞳孔エッジを検出する場合（即ち、カメラが横位置姿勢の時に瞳孔の左右方向のエッジだけを検出する場合）は、１ラインでも十分である。瞳孔のエッジは、それまで読み出したセンサ出力の最低値を基準として計算する。
【０１２０】
次に、１ライン分のデータ読み込みが終了するまで待ち、読み込んだ画像データを一時的に格納したＲＡＭから計算用のＲＡＭに移しかえる（＃ｘ１５）。上記ステップ＃ｘ０６でソフト的に読み込みが終了している場合はこの動作は必要ないが、ハード的にセンサの読み込みを行っている場合はここでセンサ読み込みのチェックを行う。
【０１２１】
次に、サブルーチン「最低輝度検出」を実行する（＃ｘ１６）。既に図４のステップ＃Ｒ０３でブロック読み出しによって最低輝度を検出しているが、ここでは画素毎の値を使って正確な値を求める。最低輝度は、瞳孔を検出するときに使用するが、センサ出力が１ライン読み込まれる毎に検出を行い、１ライン毎に更新していく。
【０１２２】
１ラインの中で領域外の部分の残りの空送りパルスを出力する（＃ｘ１７）。ハード的にセンサ読み出しを行う場合は、領域外空送りパルスを送る必要がある。次に、読み出したライン数をカウントアップする（＃ｘ１８）。ここで、カウントアップするのは制限された領域内の読み出したライン数をカウントする。
【０１２３】
次に、センサに送った駆動パルス数（１ライン分に対応）をカウントアップする（＃ｘ２０）。カウントがセンサの水平ライン数になったかどうか判定し（＃ｘ２１）、なっていなければステップ＃ｘ０５に戻り、センサ読み出しを繰り返す。
【０１２４】
ステップ＃２０では、センサの水平ライン数のカウントアップを行い、次のステップ＃２１では水平ライン数分だけパルスが出されたかを判定する。
【０１２５】
指定された領域のセンサ読み出しが終了しても、センサの水平ライン数分の読み出しパルスを送る必要があるため、ステップ＃ｘ０５に戻り、読み出しを繰り返す。この場合、読み出しの空送りパルスのみをステップ＃ｘ１９にて出力する。
【０１２６】
センサの読み出しが終了するとサブルーチンをリターンする（＃ｘ２２）。
【０１２７】
次に、最低輝度検出について、図１０を使って説明を行う。
【０１２８】
図１０（ａ）は眼鏡（メガネ）ゴーストがある眼球像の図であり、センサの読み出しは画面左上から水平な１ラインずつ行う。最低輝度の初期値はブロックデータから求めた最低輝度の値に所定量加えた値である。ブロックデータと実際の値とのずれが有るため、ブロックデータの最低値をそのまま使わず所定量を加える必要がある。
【０１２９】
図１０（ｂ），（ｃ）は、図１０（ａ）のＥ−Ｅ´の断面であり、図１０（ｂ）は従来方式の最低輝度検出によって得られる最低輝度を示している。画面下側にゴーストが有るため、図１０（ｂ）は全体的に右上りになっている。最低輝度は瞳孔の一番上側の部分で決定され、以後更新されない。その為、最低輝度を基準としてエッジを検出すると、図１０（ｂ）のＦの部分で瞳孔の輝度が最低輝度を基準にした閾値を上回ってしまう。こうなると、図１０（ａ）画像の下側エッジ、すなわち図１０（ｂ）の右側のエッジが検出出来なくなってしまう。
【０１３０】
そこで、図１０（ｃ）のように最低輝度値を部分的な最低輝度（水平１ライン毎）になるようにし、瞳孔の持ち上がりに追従して閾値も持ち上がるように変えれば〔図１０（ｃ）のＧの部分参照〕、従来例では検出できなかったエッジも検出することが出来る。
【０１３１】
次に、図９のステップ＃ｘ１６で実行されるサブルーチン「最低輝度検出」について、図１１のフローチャートにより説明する。
【０１３２】
図９のステップ＃ｘ１６から、サブルーチン「最低輝度検出」（＃Ｎ０１）がコールされると、まず、最低輝度値が新しく検出されたかどうかを示すフラグ「ＭｉｎＣｈａｎｇ」 をクリアし、最低輝度を検出するためのパラメータ設定を行う（＃Ｎ０２）。次に、その水平ラインの中で最低輝度を検出する範囲を設定する（＃Ｎ０３）。最低輝度の検出開始画素位置（ｊ）や、検出終了位置（ｊｅｎｄ）を設定する。
【０１３３】
最低輝度を検出するための閾値を設定する（＃Ｎ０４）。これまでに検出された最低輝度値をＥＹＥＭＩＮ、瞳孔位置を検出するための閾値をＥＹＥＭＩＮＬＶＬ とすると、検出用の閾値を
ｍｉｎｔｈｒ１ ＝ＥＹＥＭＩＮ＋ＥｙｅＭｉｎＲｉｓｅＴｈｒ
ｍｉｎｔｈｒ２ ＝ＥＹＥＭＩＮＬＶＬ
と設定する。ＥｙｅＭｉｎＲｉｓｅＴｈｒ は、最低輝度の持ち上がりを許容するための量で、これから検出する水平ラインの最低輝度がこの許容量以内で持ち上がっていた場合、最低輝度値は新しく更新する。（最低輝度値は大きい値に更新され、真の意味で最低輝度値とは成らないので注意）
ＥＹＥＭＩＮＬＶＬ とＥＹＥＭＩＮの関係は、ＥＹＥＭＩＮＬＶＬ ＞ＥＹＥＭＩＮである。ＥＹＥＭＩＮＬＶＬ とｍｉｎｔｈｒ１ の関係は、ＥＹＥＭＩＮＬＶＬ ＞ｍｉｎｔｈｒ１ である。
【０１３４】
次に、計算範囲内の画素とその近傍の画素について以下のループを繰り返し最低輝度判定を行う。
【０１３５】
まず、センサ出力値ｉｍ（ｊ）と最低輝度閾値ｍｉｎｔｈｒ１ を比較する（＃Ｎ０５）。ｍｉｎｔｈｒ１ より小さければステップ＃Ｎ０６へ分岐し、二つ手前の画素が閾値ｍｉｎｔｈｒ２ より小さいか較べる（＃Ｎ０６）。小さい場合、ステップ＃Ｎ０７へ分岐し、今度は一つ手間の画素が閾値ｍｉｎｔｈｒ２ より小さいか較べる（＃Ｎ０７）。小さければステップ＃Ｎ０８へ分岐し、閾値を更新し、フラグ「ＭｉｎＣｈａｎｇ」をセットする。
【０１３６】これらをまとめると
ｉｍ（ｊ）＜ ｍｉｎｔｈｒ１ かつ ｉｍ（ｊ−２）＜ ｍｉｎｔｈｒ２
かつ ｉｍ（ｊ−１）＜ ｍｉｎｔｈｒ２
の場合、閾値を更新する。
【０１３７】
閾値が更新された場合も、判定で更新されなかった場合も（つまり、ステップ＃Ｎ０５，＃Ｎ０６，＃Ｎ０７の何れかで条件が満たされなかった場合も）、次にステップ＃Ｎ０９へ進む。そして、ここで画素のインデックスｊをカウントアップし、それが指定された計算範囲内であるか判定する。計算範囲内ならステップ＃Ｎ０５へ戻り、判定を繰り返す。
【０１３８】
計算範囲すべてを判定したらステップ＃Ｎ１０へ進み、最低輝度が更新されたかをフラグ「ＭｉｎＣｈａｎｇ」で判定する。フラグがセットされていれば、ステップ＃Ｎ１１へ分岐し、最低輝度閾値を更新する。
【０１３９】
ＥＹＥＭＩＮ ←ｍｉｎｔｈｒ１
ＥＹＥＭＩＮＬＶＬ ←ｍｉｎｔｈｒ２
最後にサブルーチンをリターンする（＃Ｎ１２）。
【０１４０】
このように、１ライン分の最低輝度を求める時に、そこまでの最低輝度ＥＹＥＭＩＮにＥｙｅＭｉｎＲｉｓｅＴｈｒ を加えた量を閾値にして検出を行うため、瞳孔の持ち上がりに対して追従して最低輝度値が変り、エッジを検出するときの閾値も瞳孔の持ち上がりに追従して閾値も持ち上がる。このようにすることで瞳孔部分の輝度の持ち上がりがあっても、正しくエッジ検出が出来る。
【０１４１】
（発明と実施の形態の対応）本実施の形態において、本発明の画像入力手段は、ＣＣＤ−ＥＹＥ１４、視線検出回路１０１から成る。又、特徴領域検出手段、領域設定手段、形状比較手段は、ＣＰＵ１００とＣＰＵ１００内のプログラムで構成されている。
【０１４２】
以上が実施の形態の各構成と本発明の各構成の対応関係であるが、本発明は、これら実施の形態の構成に限定されるものではなく、請求項で示した機能、又は実施の形態がもつ機能が達成できる構成であればどのようなものであってもよいことは言うまでもない。
【０１４３】
（変形例）
本発明は、一眼レフカメラに適用した場合を述べているが、レンズシャッタカメラ，ビデオカメラ等のカメラにも適用可能である。更に、その他の光学機器、例えば顕微鏡やヘッドマウントディスプレイ装置 やゲーム機にも適用することができるものである。
【０１４４】
【発明の効果】
以上説明したように、本発明によれば、特徴領域の形状を所定の形状と比較する形状比較手段を設け、該形状比較手段により特徴領域の形状が所定の形状であると判断された場合、領域設定手段に領域設定を再度実行し直させるようにしている。
【０１４５】
よって、観察者が眼鏡使用している為に瞳孔部分の輝度よりも低い輝度部分の画像情報が得られ、この部分を瞳孔部分として誤って検出してしまうといった事を防止することができる。
【図面の簡単な説明】
【図１】本発明の実施の形態に係る一眼レフカメラの要部の電気的構成を示すブロック図である。
【図２】図１のカメラのファインダ接眼部を示す図である。
【図３】図１のカメラの視線検出時の動作を示すフローチャートである。
【図４】図３のステップ＃Ｍ０７のセンサ読み出し＆特徴抽出動作を示すフローチャートである。
【図５】図４の動作説明を助ける為の図である。
【図６】図４のステップ＃Ｒ０７の瞳孔位置推定時の動作を示すフローチャートである。
【図７】図６の動作説明を助ける為の図である。
【図８】同じく図６の眼鏡フレーム検出時の動作説明を助ける為の図である。
【図９】図４のステップ＃Ｒ０８の読み出し＆逐次処理時の動作を示すフローチャートである。
【図１０】図９の動作説明を助ける為の図である。
【図１１】図９のステップ＃ｘ１６の最低輝度検出時の動作を示すフローチャートである。
【図１２】視線検出の原理を説明する為の図である。
【符号の説明】
１４ イメージセンサ（ＣＣＤ−ＥＹＥ）
１００ ＣＰＵ
１０１ 視線検出回路
１０３ 焦点検出回路
１０６ ＬＥＤ駆動回路
１０７ ＩＲＥＤ駆動回路[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a line-of-sight detection device for detecting the line of sight of an observer and an optical device such as a camera equipped with the device.
[0002]
[Prior art]
Conventionally, various apparatuses (for example, eye cameras) for detecting a so-called line of sight (visual axis) for detecting which position on the observation surface the photographer is observing have been provided. For example, in Japanese Patent Application Laid-Open No. 1-274736, a parallel light beam from a light source is projected onto the anterior segment of the photographer's eyeball, and a gazing point is obtained using the corneal reflection image by the reflected light from the cornea and the imaging position of the pupil. Seeking. Further, the publication discloses an example in which a gazing point detection device is disposed in a single-lens reflex camera and automatic focusing of a photographic lens is performed using photographic point of sight information.
[0003]
FIG. 12 is a diagram for explaining the gaze detection principle. In FIG. 12, 15 is the eyeball of the photographer, 16 is the cornea, and 17 is the iris.
[0004]
The line-of-sight detection method will be described below with reference to FIG.
[0005]
The infrared light projected from the light source 13b irradiates the cornea 16 of the eyeball 15 of the observer. At this time, a cornea reflection image d (virtual image) formed by a part of the infrared light reflected by the surface of the cornea 16 is collected by the light receiving lens 12 and formed at a position d ′ on the image sensor 14. Similarly, the infrared light projected by the light source 13 a illuminates the cornea 16 of the eyeball 15. At this time, the cornea reflection image e formed by a part of the infrared light reflected by the surface of the cornea 16 is collected by the light receiving lens 12 and formed at a position e ′ on the image sensor 14.
[0006]
Further, the light beams from the ends a and b of the iris 17 form images of the ends a and b at positions a ′ and b ′ on the image sensor 14 through the light receiving lens 12. When the rotation angle θ of the optical axis of the eyeball 15 with respect to the optical axis of the light receiving lens 12 is small, assuming that the x coordinates of the ends a and b of the iris 17 are xa and xb, the coordinate xc of the center position c of the pupil 19 is
xc≈ (xa + xb) / 2
It is expressed.
[0007]
Further, the x coordinate of the midpoint of the cornea reflection images d and e and the x coordinate xo of the center of curvature o of the cornea 16 substantially coincide. Therefore, if the x-coordinates of the corneal reflection image generation positions d and e are xd and xe, and the standard distance between the center of curvature o of the cornea 16 and the center c of the pupil 19 is OC, the rotation of the optical axis 15a of the eyeball 15 is rotated. The angle θx is
OC * SINθx≈ (xd + xe) / 2−xc (1)
Is substantially satisfied. Therefore, the rotation angle θ of the optical axis 15a of the eyeball 15 can be obtained by detecting the position of each feature point (corneal reflection image and the center of the pupil) of the eyeball 15 projected on the image sensor 14.
[0008]
The rotation angle of the optical axis 15a of the eyeball 15 is obtained from the equation (1).
β * OC * SINθx≈ {(xpo−δx) −xic} * pitch (2)
β * OC * SINθy≈ {(ypo−δy) −yic} * pitch (3)
Is required. Here, θx is the rotation angle of the eyeball optical axis in the zx plane, and θy is the rotation angle of the eyeball optical axis in the yz plane. (Xpo, ypo) is the coordinate of the midpoint of the two corneal reflection images on the image sensor 14, and (xic, yic) is the coordinate of the pupil center on the image sensor 14. Pitch is the pixel pitch of the image sensor 14. Β is an imaging magnification determined by the position of the eyeball 15 with respect to the light receiving lens 12, and is substantially obtained as a function of the interval between the two corneal reflection images.
[0009]
δx and δy are correction terms for correcting the coordinates of the midpoint of the cornea reflection image, and correction terms for correcting an error caused by illuminating the photographer's eyeball with divergent light instead of parallel light, and δy Includes a correction term for correcting an offset component generated by illuminating the photographer's eyeball with divergent light from the lower eyelid.
[0010]
Next, the gazing point of the photographer is obtained from the relationship between the eyeball rotation angle and the coordinates of the gazing point. When the relationship between the eyeball rotation angle and the coordinates of the gazing point is the rotation angle (θx, θy) of the photographer's eyeball optical axis, the gazing point (x, y) on the photographer's observation plane has a horizontal camera posture. For position,
x = m * (θx + Δ) (4)
y = m * θy (5)
Is required. Here, the x direction indicates the horizontal direction with respect to the photographer when the camera posture is in the horizontal position, and the y direction indicates the vertical direction with respect to the photographer when the camera posture is in the horizontal position. m is a conversion coefficient for converting the rotation angle of the eyeball into coordinates on the focus plate, and Δ is an angle formed by the eyeball optical axis 15a and the visual axis (gaze point).
[0011]
Since there are individual differences in the conversion coefficients m and Δ at this time, it is necessary to obtain in advance personal difference correction data including the conversion coefficients. The personal difference correction data can be calculated by having the photographer gaze at predetermined coordinates and obtaining the rotation angle of the eyeball optical axis at that time.
[0012]
The detection operation for obtaining the individual difference correction data in this manner is called calibration, and the individual difference correction data obtained by the calibration is called calibration data.
[0013]
If the rotation angle of the eyeball when gazing at (x1,0) on the observation surface is (θx1,0), and the rotation angle of the eyeball when gazing at (x2,0) is (θx2,0), The individual difference data can be obtained.
[0014]
m = (x1-x2) / (θx1-θx2) (6)
Δ = (x2 · θx1−x1 · θx2) / (x1−x2) (7)
It is desirable to input individual difference data for each camera posture.
[0015]
In this way, the photographer's gazing point can be obtained from the cornea reflection image and the position of the pupil. A method for obtaining a feature point such as the cornea reflection image and the position of the pupil from the eyeball image on the sensor is disclosed in JP-A-4 -347133 etc. are proposed.
[0016]
Japanese Patent Laid-Open No. 6-254053 proposes a method for shortening the time without reading all the pixels of the sensor. In this proposal, since the brightness of the pupil part of the eyeball image is very low, the low-luminance part on the image is detected, only that part is read (area limited readout), and calculation is performed to shorten the eye-gaze detection time. It has been proposed.
[0017]
Further, it is disclosed that the edge of the pupil is detected from the read sensor image based on the minimum luminance value.
[0018]
[Problems to be solved by the invention]
However, in the disclosed technique, detection is performed based on the assumption that the pupil has the lowest luminance, and thus the gaze detection cannot be performed well if the assumption is broken.
[0019]
There are the following problems as a cause when the pupil does not have the lowest luminance and when the detection is performed based on the lowest luminance.
[0020]
1) Illumination light does not spread around the sensor, and the amount of light falls.
[0021]
2) Since there is a total reflection object such as a metal frame of spectacles (glasses), no light of that portion is guided onto the sensor. That is, only the frame portion is dark.
[0022]
3) Since the overall brightness is low, an appropriate threshold cannot be set.
[0023]
4) Although the size of the pupil changes depending on the brightness, the area size is the same even if the brightness changes during area limited readout.
[0024]
Each problem is described below.
[0025]
1) When the peripheral portion of the sensor has the lowest luminance, the reading area is determined around that area. However, if the reading area does not include all pupils and P images, the line of sight cannot be detected.
[0026]
2) The frame of the spectacles may be dark on the sensor, and the luminance may be lower than the luminance of the pupil portion. If this is mistakenly detected as a pupil, the readout area is determined centering on the frame portion as in 1) above, and the pupil and the corneal reflection image (Purkinje image: P image) are all included in the readout area. Gaze detection is not possible.
[0027]
3) When the brightness is low as a whole, the threshold setting is not successful. Although the pupil may show the lowest luminance, there are many portions where the luminance is almost the same as that of the pupil, and the pupil and other portions cannot be distinguished. For this reason, the reading area cannot be determined.
[0028]
4) The pupil size is small when bright and large when dark. If the area is determined with the same size, it will be useless.
[0029]
In addition to these, there is a phenomenon in which the luminance of the pupil portion increases due to the influence of ghosts and the like. In such a case, if the pupil is detected based on the minimum luminance, the edge of the pupil cannot be detected.
[0032]
(Object of the invention) Object of the invention The observer is wearing glasses use Therefore, it is possible to provide a line-of-sight detection device and an optical device capable of preventing image information of a luminance portion lower than the luminance of the pupil portion from being obtained and preventing this portion from being erroneously detected as the pupil portion. There is.
[0035]
[Means for Solving the Problems]
the above the purpose In order to achieve 5 Each of the present inventions described in 1), an image input means for inputting image information including an eyeball image, and an image information obtained by the image input means. Feature region detecting means for detecting a feature region including a feature point, and the feature region based on the feature region Area setting means for setting the area of the image information, and based on the image information of the set area Perform gaze detection In the line-of-sight detection device, Shape comparison means for comparing the shape of the feature region with a predetermined shape Comprising When the shape comparison means determines that the feature area has a predetermined shape, the area setting means re-executes the area setting. It is characterized by this. Claims 7 The present invention described in claim 1 5 An optical device comprising the line-of-sight detection device according to any one of the above.
[0040]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described in detail based on illustrated embodiments.
[0041]
FIG. 1 is a block diagram of a main part of an electric circuit built in the camera body of the present invention.
[0042]
A central processing unit (hereinafter referred to as a CPU) 100 of a microcomputer, which is a camera control means built in the camera body, includes a line-of-sight detection circuit 101, an attitude detection circuit 102, an automatic focus detection circuit 103, and a signal input circuit 104. , An LCD drive circuit 105, an LED drive circuit 106, and an IRED drive circuit 107 are connected. Although not shown, signals are transmitted to the focus adjustment circuit and the aperture driving circuit arranged in the visual photographing lens through the mount contact.
[0043]
An EEPROM 100a serving as a storage unit attached to the CPU 100 stores film counter and other photographing information and stores individual difference correction data obtained by calibration.
[0044]
The line-of-sight detection circuit 101 transmits an eyeball image output from the image sensor 14 (CCD-EYE) to the CPU 100. The CPU 100 A / D converts the eyeball image signal from the image sensor 14 by A / D conversion means inside the CPU, and this image information From Each feature point of the eyeball image necessary for eye-gaze detection is extracted according to a predetermined algorithm, and the rotation angle of the photographer's eyeball is calculated from the position of each feature point.
[0045]
The posture detection circuit 102 includes a plurality of pairs of photosensors, LEDs (HV1, HV2), and a shield that is movable by gravity, and the posture is detected when the shield moves due to gravity and the output of the photosensor changes. The photosensors are arranged at an angle of about 90 ° to each other, and are also arranged at an angle from the horizontal axis.
[0046]
The automatic focus detection circuit 103 sends photoelectric conversion voltages obtained from a plurality of line sensors to the CPU 100, and the CPU 100 sequentially A / D-converts the line sensor signals by a built-in A / D converter and performs a predetermined calculation. Perform focus detection.
[0047]
The signal input circuit 104 inputs signals from various switches, and causes the CPU 100 to generate an interrupt signal when the switch state changes.
[0048]
SW-1 is turned on by the first stroke of the release button, and a switch for starting photometry, AF (autofocus) and line-of-sight detection operation, SW-2 is a release switch turned on by the second stroke of the release button, and SW-MDIAL is This is a mode dial for selecting a calibration mode and a camera lock position in addition to various camera shooting modes. SW-CALDEL is an individual difference correction data deletion switch. SW-DIAL1 and SW-DIAL2 are electronic dial switches, which are input to an up / down counter of a signal input circuit, and count the rotational click amount of the electronic dial.
[0049]
The LCD drive circuit 105 displays various information on the in-finder LCD 24 and the monitor LCD 42, and also drives a buzzer (not shown) to generate a buzzer sound.
[0050]
The LED drive circuit 106 is a circuit for turning on the LED 21 (LED-L2, LED-L1, LED-C, LED-R1, LED-R2) for superimposing illumination of a distance measuring point in the finder. . LED-L2, LED-L1, LED-C, LED-R1, and LED-R2 are focus detection (ranging points) marks FP-L2, FP-L1, FP-C, FP-R1, and FP- in the viewfinder, respectively. The distance measuring point mark in the viewfinder is illuminated by turning on the LED 21.
[0051]
The IRED drive circuit 107 is an infrared LED (IRED-0, IRED-1, IRED-2, IRED-3, IRED-4, IRED-5, IRED-6 for illuminating the photographer's eyeball with infrared light. , IRED-7). This circuit incorporates a circuit for preventing overcurrent and a safety circuit for preventing accidents caused by energization for a long time.
[0052]
Two IREDs are used in pairs and are illuminated from below the eyeball. FIG. 2 shows a specific IRED position.
[0053]
When the camera is in the horizontal position, IRED-0 and IRED-1 are used as a pair for the naked eye, and IRED-4 and IRED-5 are used as a pair for the glasses. IRED-3 and IRED-0 are used as a pair, and IRED-7 and IRED-4 are used as a pair for glasses. In the vertical position under the grip, IRED-1 and IRED-2 are used as a pair for the naked eye, and IRED-5 and IRED-6 are used as a pair for glasses. That is, in any posture, an IRED pair with a narrow interval is used for the naked eye, and an IRED pair with a wide interval is used for a long distance between the glasses and the eyeball.
[0054]
When the switch SW-1 is pressed, after performing a predetermined operation, the “line of sight” flow in FIG. 3 is called. Hereinafter, the entire line-of-sight detection operation will be described in accordance with this.
[0055]
When the subroutine “line-of-sight main” is called, the program is executed from step # M01. The line-of-sight detection routine is not necessarily executed just once, but the line-of-sight position is not output, but is executed a plurality of times (try several times by changing the control method) and finally detects the position of the gazing point. It is like that.
[0056]
First, the set calibration number is checked, and individual difference correction data corresponding to the calibration number is read from the EEPROM 100a (# M02).
[0057]
Check if the calibration number is not correct or if the calibration number is not set (including when the line of sight is OFF) (# M03). If not, branch to step # M09 and return. .
[0058]
When the calibration number is correctly set and the individual difference correction data can be read, the line-of-sight detection parameter is initialized (# M04). Here, initialization of the parameter indicating the line-of-sight detection state is performed only for the first time. The subroutine “Gaze Main” is not executed after the second time when it is repeatedly executed.
[0059]
Next, accumulation of the line-of-sight detection area sensor is performed by a subroutine “YBASIS” (# M05). In this subroutine, the IRED for illuminating the eyeball and its drive current are determined, the sensor is preliminarily accumulated, and the accumulation control when the line of sight is detected is determined. Image data of an eyeball image is obtained by the determined accumulation control method.
[0060]
Next, parameters for extracting feature points of the image data are initialized (# M06). Then, a subroutine “Read_Detect” (# M07) for extracting feature points while reading the accumulated data is executed. The internal operation of this subroutine will be described in detail later.
[0061]
When sensor reading and feature point extraction are completed, the gaze point position is calculated from the detected feature points in a subroutine “P_Process” (# M08). Here, from the pupil edge coordinates obtained by “Read_Detect”, the center coordinates of the pupil circle and the coordinates of the corneal reflection image (Prikinje image: P image) of the illumination light are used to calculate the rotation angle of the eyeball from a predetermined calculation formula. Ask for. When the rotation angle of the eyeball can be detected, a point to be watched by the person is calculated using the individual difference correction data.
[0062]
Finally, this program is returned (# M10).
[0063]
If the gazing point cannot be obtained at one time, the “line of sight main” is called repeatedly, and the line of sight is detected by another control method (which is performed by changing the illumination, accumulation control method, and calculation parameter).
[0064]
Next, the subroutine “Read_Detect” executed in step # M07 will be described with reference to the flowchart of FIG. The subroutine “Read_Detect” is called after the sensor accumulation is completed.
[0065]
When the subroutine “Read_Detect” is called, the processing is executed from step # R01. First, the sensor is read roughly using the block reading function of the sensor (# R02). The sensor block reading function is a function for outputting data in which a plurality of pixels of the sensor are collected. The sensor output is output as 16 blocks of “4 × 4” as one block (an average value of “4 × 4”). Output).
[0066]
By doing so, for example, the sensor output of “100 × 70” pixels can be viewed roughly with the data of “25 × 18” blocks. Even if it is not possible to secure a memory of “100 × 70” pixels by reducing the number of data in this way, it is possible to secure a memory of “25 × 18” pixels even with an embedded microcomputer in a device such as a camera. It is.
[0067]
Further, even in a sensor in which a block output cannot be obtained in hardware, an equivalent calculation can be performed by obtaining an average in software and reducing the number of data. The average value is not necessarily required, and all sensor pixels may be thinned out.
[0068]
Next, the maximum value, the minimum value, and the average value of the block-read image are obtained (# R03). The maximum value is used when obtaining a Purkinje image (hereinafter referred to as P image), and the minimum value is used when searching for a pupil. The average value is used as brightness data for the entire image.
[0069]
Here, the range in which the minimum luminance of the block image is detected and the increase (weighting) will be described with reference to FIG.
[0070]
In this example, the lower left part of the image is partially darker, and the address (xmin ′, ymin ′) portion has the lowest luminance (see FIG. 5A). When the entire block image is searched for the minimum luminance, a portion other than the pupil may be searched in this way. Therefore, first, the portion where the minimum luminance is detected is limited to the central area 1 (horizontal center: xg1, vertical range: yg1). To limit to the center part,
1) The surrounding area becomes dark due to uneven illumination and low light intensity.
[0071]
2) The eyeball is not in the periphery of the image (if it is in the periphery of the image, gaze detection is impossible).
That is why.
[0072]
Next, in the area 1 area, the doughnut-shaped region outside the area 2 (horizontal center: xg2 and vertical: yg2 range) is increased by an amount of weight (weighting amount) wt1. In the area 2, the doughnut-shaped region outside the area 3 (horizontal center: xg3, vertical: yg3 range) is raised by the amount of weight wt2 (see FIG. 5B). The area 3 is lifted by the amount of weight wt3. The amount of baldness is “wt1>wt2> wt3”, and the amount of baldness in the center portion is small and the amount of baldness in the periphery is large. For this reason, the central portion is easily selected as the position of the lowest luminance.
[0073]
If the amount corresponding to the block position is wt (x, y) and the block image value is b1 (x, y), the minimum luminance position is
[B1 (x, y) + wt (x, y)] (however, the calculation range is within area 1)
Can be obtained as the position of (x, y) at which. If this position is (xmin, ymin), the minimum luminance MINblk is obtained as b1 (xmin, ymin). That is, the original image value obtained by subtracting the value from the minimum value obtained by raising is stored as the minimum value (see FIG. 5C).
[0074]
Actually, wt1 = 4, wt2 = 2, wt3 = 0, xg1 = 18, yg1 = 14, xg2 = 12, yg2 = 8, xg3 = 6, yg3 = 4.
[0075]
When calculating the average luminance, the calculation is performed without raising the brightness.
[0076]
Returning to the description of the flowchart of FIG. 4, next, blink detection is performed using the lowest value of the block image and a predetermined threshold (# R04). Since the line of sight cannot be detected during blinking, the process is performed. For example, the current gaze detection is stopped and the next gaze detection is performed with exactly the same control.
[0077]
Next, the read range is initialized. Usually, the entire range is designated first (# R05). When the sensor is attached to the camera and the surrounding area is removed, an area that cannot be removed is designated.
[0078]
Next, the average luminance (BlkAve) obtained from the block data is compared with the threshold (# R06). If the average luminance is low, the process proceeds to step # R08, and the pupil position is not estimated based on the block data. This is because if the overall brightness is low, the peripheral part of the sensor will be low, and the pupil position will not necessarily be the minimum brightness. If the pupil position is estimated using the minimum brightness, the pupil position cannot be estimated correctly. . If the average luminance is higher than the threshold value, the process branches to step # R07, and the subroutine “Estimate pupil position” is called to estimate the pupil position.
[0079]
In Step # R08, the subroutine “Reading & Sequential Processing” is called, and feature points are extracted by sensor reading and sequential processing in the determined area, and the process returns (# R09).
[0080]
As described above, the lowest luminance position is detected by raising (weighting) the block image, so that the correct pupil position can be estimated even when the illumination light does not spread around the sensor and the light amount has dropped. I made it.
[0081]
Further, when the overall luminance is low, it is difficult to detect the pupil position, so the pupil position is not estimated. For this reason, the pupil estimation is not forced and time is not wasted or erroneous estimation is not performed.
[0082]
Next, the sub-routine “pupil position estimation” executed in step # R07 of FIG. 4 will be described with reference to the flowchart of FIG.
[0083]
When the subroutine “Estimate pupil position” is called (# E01), the pupil spread in the horizontal direction is first detected by the subroutine “BlkHorArea”. The blocked image data is represented by b1 (x, y) (x: horizontal direction, y: vertical direction), the minimum luminance obtained in step # R03 in FIG. 4 is “MINblk”, and the addresses are xmin0 and ymin0. .
[0084]
First, a range where the luminance is “MINblk + α” or less at the address ymin0 in the vertical direction is obtained. Next, the range in the vertical direction is obtained at the horizontal address xmin0 position where the luminance is equal to or less than “MINblk + α” (# E03). The horizontal address is not necessarily the position of xmin0. For example, the range may be obtained in the vertical direction from the center position of the range obtained in step # E02.
[0085]
Here, the size of the pupil is not searched in a plane when the surrounding dark part and the low-luminance part of the pupil are connected (see FIG. 7 (a)). This is because the correct pupil position cannot be detected (see FIG. 7B).
[0086]
However, if the detection is performed only in the horizontal direction and the vertical direction from the detected lowest luminance position, the correct pupil size can be detected (see FIG. 7C).
[0087]
Next, eyeglass frame determination is performed.
[0088]
If it is determined in step # E04 that the frame is not an eyeglass frame, the center of the range calculated in step # E02 and step # E03 is obtained (# E05). That is, the center of the horizontal range obtained in step # E02 and the center of the vertical range obtained in step # E03 are obtained.
Next, the overall brightness is determined (# E06). Average value of block image BlkAve And the threshold value ThrHI, and if smaller than the threshold value, the process branches to step # E07. In step # E07, the position of the P image is detected from the block image. If the overall brightness is high, it becomes difficult to detect the Purkinje image portion from the block image. Since the pupil range is not detected small due to the influence of the P image, the P image position is not detected.
[0090]
If the luminance is low, the P image position is detected using the block image (# E07). An isolated point with high brightness is detected as a P image. When the camera is in the horizontal position, an isolated value is detected in the vertical direction. Since there are two P images in the horizontal direction, the two blocks may continuously have high brightness. For this reason, the P image detection by the block image is not performed in the horizontal direction.
[0091]
Next, the detected P image and its surroundings are replaced with the lowest luminance level (# E08). By replacing with the lowest luminance level, it is possible to estimate the size of the pupil again by performing the same processing as in steps # E02 and # E03.
[0092]
In Steps # E09 and # E10, the spread of the pupil in the horizontal direction and the vertical direction is obtained again. Detection is performed from the pupil center position obtained in step # E05.
[0093]
Next, the actual position on the sensor is obtained from the pixel position of the block (# E11). A margin is added to the finally detected pupil size to determine a readout area (# E12).
[0094]
If the x coordinate of the left side of the pupil is xpp1, the right side is xpp2, the upper side is ypp1, the lower side is ypp2, the margin of the region (pupil portion) is mag, and the P image portion is magp, the sensor area to be read out is the horizontal direction. Is from “xpp1 −mag” to “xpp2 + mag”, and the vertical direction is from “ypp1 −mag” to “ypp2 + mag + magp”.
[0095]
When the camera is in the horizontal position, the P image is below the pupil, so that the downward reading area is given a margin more than the others by the mapp. When the camera is in the vertical position, the “magp” has a margin so as to be widened below the x coordinate.
[0096]
The margins “mag” and “magp” vary depending on the luminance of the entire sensor, and the pupil is reduced when the luminance is increased, so that the margin is set to be small.
[0097]
Average brightness obtained in step # R03

When the sensor reading area is set, the subroutine is returned (# E15).
[0098]
In this way, by changing the pupil area estimated by pupil estimation with the overall brightness, even if the pupil becomes small when it is bright, it does not read out a useless area, and the pupil does not protrude even when it is dark You can set the area.
[0099]
Here, step # E04 in FIG. 6 will be described.
[0100]
If the frame of the glasses enters the image, it may appear as a black elongated shape. This is because the frame portion of the spectacles is totally reflected, and the illumination light does not come on the sensor at all.
[0101]
As shown in FIG. 8 (a), since the vertical width of the sensor is not so large in the horizontal position, the dark portion of the spectacle frame has come out of the sensor. As shown in b), in the vertical position, the dark part of the frame of the glasses enters the sensor, which may adversely affect the detection of the pupil position. Therefore, it is necessary to identify the frame of the glasses and the pupil and detect only the correct pupil. Therefore, paying attention to the shape of the spectacle frame, the determination is made based on the shape of the low luminance portion detected in steps # E02 and # E03.
[0102]
Specifically, in the case of the vertical position, it is determined whether the horizontal direction (x-axis direction on the sensor) is short and the vertical direction (y-axis direction on the sensor) is long. When the position of A in FIG. 8C can be recognized as the minimum luminance, the pupil portion is correctly selected, so the horizontal length xd and the vertical length yd of the low luminance portion are approximately the same. It's getting bigger, and either isn't big or small. However, if the B portion is detected as the minimum luminance, the horizontal length xf is very short and the vertical length yf is extremely long. The eyeglass frame is identified from this feature.
[0103]
Assuming that the horizontal threshold of the spectacle frame is xfth, the vertical threshold is yfch, the horizontal length of the low-luminance portion is ldkx, and the vertical length is ldky,
ldkx <xfth and ldky> yfth
If it meets, the eyeglass frame is determined.
[0104]
Specifically, detection is performed with xfth = 5 and yfth = 10 (in the case of the vertical position). Although the description has been made in the vertical position according to the figure, the horizontal direction and the vertical direction are interchanged in the horizontal position. Judgment of frame is not possible only by the length in the vertical direction (vertical position) No This is because when the entire screen is dark, the size of the low-brightness portion becomes large.
[0105]
If the eyeglass frame is determined, the process branches to step # E13. Here, since the pupil portion could not be estimated well, the region restriction parameter was initialized, and the minimum luminance value data obtained from the block data is also an incorrect value for the spectacle frame portion, so it is reset to the default value ( # E14).
[0106]
In this example, if it is determined that the frame is a spectacle frame, the region restriction is stopped, but conversely, the region may be limited so that only a portion obtained by removing the spectacle frame portion is calculated. Alternatively, recalculation may be performed so that the spectacle frame portion is excluded and the pupil portion is estimated.
[0107]
When the process ends, the process proceeds to step # E15, and the subroutine is returned.
[0108]
As described above, the characteristics of the spectacle frame are discriminated, and when the spectacle frame is discriminated, the line-of-sight calculation is performed in a predetermined area, so that an incorrect area is not set.
[0109]
Although not described in the flowchart, when it is determined that the frame is a spectacle frame, an area other than that may be set, or the pupil position may be estimated again in the area excluding the spectacle frame area.
[0110]
In this way, erroneous estimation of the pupil position can be eliminated.
[0111]
Next, the subroutine “Read & Sequential Processing” in FIG. 4 will be described using the flowchart in FIG.
[0112]
When this subroutine “Reading & Sequential Processing” is called (# x01), first, reading start communication is performed to the sensor (# x02). Next, A / D conversion is initialized in order to A / D convert the sensor output (# x03). Next, initialization of parameters for performing sequential calculation and setting of the number of pulses for reading out only a limited region are performed (# x04).
[0113]
Next, it is determined whether or not the horizontal line to be read includes an area to be read (# x05), and if outside the read area, the process branches to step (# x19). In step # x19, a horizontal line skip pulse for skipping one horizontal line is output. When this pulse is sent to the sensor, the sensor does not read out one horizontal line.
[0114]
In step # x05, if the horizontal line to be read includes an area to be read, the process branches to step # x06. Here, one line is read out. First, a readout pulse is sent for each pixel, and the output from the sensor is A / D converted and temporarily stored in a predetermined RAM. A portion outside the designated area in the horizontal line to be read is not subjected to A / D conversion, and only the data in the designated area can be obtained in a short time by outputting the read pulse at a high speed.
[0115]
These operations may be configured so as to perform pulse output, A / D conversion, and data storage in hardware, in addition to operating with software timing. In this case, since it is not necessary for software to intervene in A / D conversion and pulse output, the software can extract feature points during that time.
[0116]
Next, it is determined whether or not the camera position is the vertical position (# x07). If the vertical position, the number of readout lines is entered in the A register (# x08), if the horizontal position (if not the vertical position), the pupil detected so far The total number of edges is calculated and stored in the A register (# x09). Next, it is determined whether or not the A register is 0 (# x11). If it is not 0, the process branches to (# x12), and the subroutine “P image detection” is executed (# x12).
[0117]
When the camera position is the vertical position, “P image detection” is performed after reading of the sensor by at least one line or more. Further, in the case of the horizontal position, it is known from the relationship of the illumination position that the P image is lower than the pupil position. Therefore, in order to eliminate useless calculation, the P position is detected after at least one pupil edge is detected. Image detection is performed.
[0118]
Next, it is determined whether reading of a sufficient number of lines to detect the pupil edge is completed (# x13), and if sufficient, the process branches to step # x14. In step # x14, the subroutine “pupil edge detection” is called to detect the pupil edge.
[0119]
Image data can only be stored for several lines due to memory limitations, and if the corresponding number of lines is stored, it is determined that there is a sufficient number of lines. When detecting the pupil edge only in the horizontal direction (that is, detecting only the left and right edge of the pupil when the camera is in the horizontal position), one line is sufficient. The edge of the pupil is calculated based on the lowest value of the sensor output read up to that point.
[0120]
Next, the process waits until the reading of data for one line is completed, and moves the read image data from the temporarily stored RAM to the RAM for calculation (# x15). This operation is not necessary when the reading is completed in software in step # x06, but if the sensor is read in hardware, the sensor reading is checked here.
[0121]
Next, a subroutine “minimum luminance detection” is executed (# x16). Although the minimum luminance has already been detected by block reading in step # R03 in FIG. 4, an accurate value is obtained using a value for each pixel. The minimum luminance is used when detecting the pupil, but is detected every time the sensor output is read in one line, and is updated every line.
[0122]
The remaining idle feed pulse of the portion outside the area in one line is output (# x17). When performing sensor reading in hardware, it is necessary to send an out-of-region idle feed pulse. Next, the number of read lines is counted up (# x18). Here, counting up counts the number of lines read in the limited area.
[0123]
Next, the number of drive pulses sent to the sensor (corresponding to one line) is counted up (# x20). It is determined whether the count has reached the number of horizontal lines of the sensor (# x21). If not, the process returns to step # x05 to repeat sensor reading.
[0124]
In step # 20, the number of horizontal lines of the sensor is counted up, and in the next step # 21, it is determined whether or not pulses are output by the number of horizontal lines.
[0125]
Even if the sensor reading of the designated area is completed, it is necessary to send a reading pulse for the number of horizontal lines of the sensor, so the process returns to step # x05 and the reading is repeated. In this case, only the read idle feed pulse is output in step # x19.
[0126]
When the sensor reading is completed, the subroutine is returned (# x22).
[0127]
Next, the minimum luminance detection will be described with reference to FIG.
[0128]
FIG. 10A is a diagram of an eyeball image with spectacles (glasses) ghost, and sensor reading is performed for each horizontal line from the upper left of the screen. The initial value of the minimum luminance is a value obtained by adding a predetermined amount to the minimum luminance value obtained from the block data. Since there is a difference between the block data and the actual value, it is necessary to add a predetermined amount without using the minimum value of the block data as it is.
[0129]
FIGS. 10B and 10C are cross sections taken along line EE ′ of FIG. 10A, and FIG. 10B shows the minimum luminance obtained by the conventional minimum luminance detection. Since there is a ghost on the lower side of the screen, FIG. 10B is generally on the upper right. The minimum brightness is determined at the uppermost part of the pupil and is not updated thereafter. Therefore, when an edge is detected with reference to the minimum luminance, the luminance of the pupil exceeds the threshold value based on the minimum luminance at the portion F in FIG. 10B. In this case, the lower edge of the image in FIG. 10A, that is, the right edge in FIG. 10B cannot be detected.
[0130]
Therefore, as shown in FIG. 10 (c), the minimum luminance value is set to a partial minimum luminance (for each horizontal line), and the threshold value is also raised so as to follow the lifting of the pupil [FIG. 10 (c). The edge that could not be detected in the conventional example can also be detected.
[0131]
Next, the subroutine “minimum luminance detection” executed in step # x16 of FIG. 9 will be described with reference to the flowchart of FIG.
[0132]
When the subroutine “minimum luminance detection” (# N01) is called from step # x16 in FIG. 9, first, the flag “MinChange” indicating whether or not the minimum luminance value is newly detected is cleared, and the minimum luminance is detected. Parameter setting is performed (# N02). Next, a range for detecting the minimum luminance in the horizontal line is set (# N03). The lowest luminance detection start pixel position (j) and the detection end position (jend) are set.
[0133]
A threshold for detecting the minimum luminance is set (# N04). If the lowest luminance value detected so far is EYEMIN and the threshold for detecting the pupil position is EYEMINLVL, the detection threshold is
minthr1 = EYEMIN + EyeMinRiseThr
minthr2 = EYEMINLVL
And set. EyeMinRiseThr is an amount for allowing the minimum luminance to be lifted. When the minimum luminance of the horizontal line to be detected is increased within the allowable amount, the minimum luminance value is newly updated. (Note that the minimum brightness value is updated to a larger value and does not become the minimum brightness value in the true sense)
The relationship between EYEMINLVL and EYEMIN is EYEMINLVL> EYEMIN. The relationship between EYEMINLVL and minthr1 is EYEMINLVL> minthr1.
[0134]
Next, the minimum brightness determination is performed by repeating the following loop for the pixels within the calculation range and the neighboring pixels.
[0135]
First, the sensor output value im (j) is compared with the minimum luminance threshold minthr1 (# N05). If it is smaller than minthr1, the process branches to step # N06, and it is compared whether the two previous pixels are smaller than the threshold value minthr2 (# N06). If it is smaller, the process branches to step # N07, and this time, it is compared whether the one troublesome pixel is smaller than the threshold value minthr2 (# N07). If it is smaller, the process branches to Step # N08, the threshold is updated, and the flag “MinChange” is set.
In summary,
im (j) < minthr1 and im (J-2) < minthr2
And im (J-1) < minthr2
In the case of, the threshold value is updated.
[0137]
If the threshold value is updated or not updated by determination (that is, if the condition is not satisfied in any of steps # N05, # N06, and # N07), the process proceeds to step # N09. Then, the pixel index j is counted up here to determine whether it is within the designated calculation range. If it is within the calculation range, the process returns to step # N05 to repeat the determination.
[0138]
When all the calculation ranges are determined, the process proceeds to step # N10, and it is determined by the flag “MinChange” whether the minimum luminance has been updated. If the flag is set, the process branches to step # N11 to update the minimum luminance threshold.
[0139]
EYEMIN ← minthr1
EYEMINLVL ← minthr2
Finally, the subroutine is returned (# N12).
[0140]
In this way, when the minimum luminance for one line is obtained, detection is performed with the amount of EyeMinRiseThr added to the minimum luminance EYEMIN so far as a threshold, so the minimum luminance value changes following the lifting of the pupil, The threshold value for detecting an edge also increases with the rise of the pupil. By doing so, the edge can be detected correctly even if the luminance of the pupil is increased.
[0141]
(Correspondence between Invention and Embodiment) In the present embodiment, the image input means of the present invention includes a CCD-EYE 14 and a line-of-sight detection circuit 101. Become. Also, feature area detection means, area setting means, shape comparison means Consists of a CPU 100 and a program in the CPU 100.
[0142]
The above is the correspondence between each configuration of the embodiment and each configuration of the present invention. However, the present invention is not limited to the configuration of the embodiment, and the functions shown in the claims or the embodiment It goes without saying that any configuration may be used as long as the function of the can be achieved.
[0143]
(Modification)
The present invention is applied to a single-lens reflex camera, but can also be applied to cameras such as a lens shutter camera and a video camera. Furthermore, the present invention can be applied to other optical devices such as a microscope, a head mounted display device, and a game machine.
[0144]
【The invention's effect】
As explained above, according to the present invention, A shape comparison unit that compares the shape of the feature region with a predetermined shape is provided, and when the shape comparison unit determines that the shape of the feature region is a predetermined shape, the region setting unit performs the region setting again. I am doing so.
[0145]
Therefore, Since the observer uses glasses, it is possible to obtain image information of a luminance portion lower than the luminance of the pupil portion, and prevent this portion from being erroneously detected as the pupil portion. be able to.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an electrical configuration of a main part of a single-lens reflex camera according to an embodiment of the present invention.
2 is a view showing a viewfinder eyepiece of the camera shown in FIG. 1;
FIG. 3 is a flowchart showing an operation at the time of line-of-sight detection of the camera of FIG. 1;
4 is a flowchart showing a sensor reading & feature extraction operation in step # M07 of FIG. 3;
FIG. 5 is a diagram for helping to explain the operation of FIG. 4;
6 is a flowchart showing an operation at the time of pupil position estimation in step # R07 in FIG. 4;
7 is a diagram for helping to explain the operation of FIG. 6; FIG.
8 is a diagram for helping to explain the operation when detecting the eyeglass frame of FIG. 6; FIG.
FIG. 9 is a flowchart showing an operation at the time of reading & sequential processing in step # R08 of FIG. 4;
10 is a diagram for explaining the operation of FIG. 9; FIG.
11 is a flowchart showing an operation at the time of detecting the minimum luminance in step # x16 in FIG. 9;
FIG. 12 is a diagram for explaining the principle of line-of-sight detection.
[Explanation of symbols]
14 Image sensor (CCD-EYE)
100 CPU
101 Eye-gaze detection circuit
103 Focus detection circuit
106 LED drive circuit
107 IRED drive circuit

Claims (7)

Image input means for inputting image information including an eyeball image, feature area detection means for detecting a feature area including a feature point of image information obtained by the image input means, and the image information based on the feature area A line-of-sight detection apparatus that performs line-of- sight detection on the basis of image information of the set area.
A shape comparison unit that compares the shape of the feature region with a predetermined shape is provided, and the region setting unit executes region setting when the shape comparison unit determines that the feature region has a predetermined shape. A line-of-sight detection device characterized by re-working . The line-of-sight detection device according to claim 1, wherein the image input unit is a sensor formed of a plurality of photoelectric conversion elements. The image input means outputs a representative signal in image information obtained for each block obtained by dividing the sensor, and the region setting means detects a feature point of the image information from the signal output. The gaze detection apparatus according to claim 2. The line- of- sight detection apparatus according to claim 1 , wherein the area setting unit sets a predetermined area when the shape comparing unit determines that the feature area has a predetermined shape . When the shape comparison unit determines that the feature region has a predetermined shape, the region setting unit sets a region that does not include the feature region determined to have the predetermined shape. The visual axis detection device according to claim 1 . The said predetermined shape is defined by the magnitude | size of the 1st direction of an image, and the magnitude | size of the 2nd direction perpendicular | vertical to a 1st direction, The 1, 2, 3, 4 or 5 characterized by the above-mentioned. The line-of-sight detection device described. An optical apparatus comprising the visual line detection device according to claim 1 .

Images