JP4123076B2 - Driver status detection device - Google Patents

Description

所定の式は、以下に示すものである。
【００４８】
【数２】

なお、ｄ（ｘ，ｙ）は、撮像画像内の任意位置の画素値であり、ｄ’（ｘ，ｙ）は変換後の画素値である。
【００４９】
その後、画像処理部３１は、現在の顔画像の探索領域内から、前回の顔画像内の参照領域に最も類似する位置を求めて、移動量（ｘｄ，ｙｄ）、すなわちオプティカルフローを算出する（ＳＴ３１）。
【００５０】
具体的には、画像処理部３１は、まず、探索領域内から参照領域に最も類似する領域を求め、最も類似する領域の中心点を、参照領域に最も類似する位置とする。そして、画像処理部３１は、求められた最も類似する領域の中心点と、探索領域の中心点とから移動量（ｘｄ，ｙｄ）を算出し、オプティカルフローとする。
【００５１】
ここで、ステップＳＴ３１について詳細に説明する。上述したように、顔画像上には予め参照領域が設定されている。また、探索領域は参照領域を取り囲むように設定される。また、参照領域と探索領域とは時間を異にして設定される。具体的には、図８に示すように、参照領域は時刻ｔにおいて設定され、探索領域は時刻ｔ後の時刻（ｔ＋１）において設定される。
【００５２】
図８は、図７に示すステップＳＴ３１における移動量（ｘｄ，ｙｄ）の算出方法の説明図である。ステップＳＴ３１の処理において、画像処理部３１は、まず、時刻ｔ＋１の探索領域内に候補領域を作成する。この候補領域は、参照領域と同じ大きさを有する領域である。具体的には、探索領域内の任意の位置を（ｘｄ，ｙｄ）とした場合、−（ｓｗ−ｔｗ）／２＜ｘｄ＜（ｓｗ−ｔｗ）／２、且つ、−（ｓｈ−ｔｈ）／２＜ｙｄ＜（ｓｈ−ｔｈ）の範囲から切り出した領域である。
【００５３】
画像処理部３１は、探索領域内の所定箇所に上記のような候補領域を設定し、設定した候補領域と参照領域とを比較等して、類似度を求める。次に、画像処理部３１は、候補領域を他の位置に動かし、動かした位置の候補領域と参照領域とを比較等して類似度を求める。
【００５４】
その後、画像処理部３１は、候補領域を順次移動させていき、探索領域内での各箇所において参照領域との類似度を算出する。類似度は、例えば、濃淡データを基準に判断される。ここで、濃淡データを基準に類似度を算出する場合において、類似度をｃｏｓθとすると、類似度は以下の式にて表される。
【００５５】
【数３】

上式においては、参照領域の濃淡データをＴとし、候補領域の濃淡データをＳとしている。また、ｘｄは、探索領域内のＸ座標値を示し、ｙｄは、探索領域内のＹ座標値を示している。
【００５６】
以上から、画像処理部３１は、類似度が最大となる位置Ｓを定め、点Ｓと点Ｏとの座標値の差を移動量（ｘｄ，ｙｄ）として取得し、これをオプティカルフローとする。
【００５７】
再度、図７を参照して説明する。移動量（ｘｄ，ｙｄ）の算出後、画像処理部３１は、類似度の範囲が閾値以上か否かを判断する（ＳＴ３２）。すなわち、画像処理部３１は、まず、候補領域によって探索領域内を走査していき、探索領域内の各箇所の類似度を算出する。その後、画像処理部３１は、得られた類似度の分散を求め、この分散により類似度の範囲が閾値以上か否かを判断する。
【００５８】
ここで、類似度の範囲が小さい場合とは、探索領域内の各箇所において、同じような類似度が検出される場合である。例えば、参照領域が真っ白な画像である場合など、特徴が少ない場合には探索領域内のどの箇所と比較しても似たような類似度の結果が得られることとなる。そして、このような場合、それぞれ類似度の差が小さいことから、類似度が最大となる点Ｓの検出が不正確になりやすい。このため、図７のステップＳＴ３２の処理では、所定の閾値と比較し、好適なものと不適なものとの選別するようにしている。
【００５９】
類似度の範囲が閾値以上であると判断した場合（ＳＴ３２：ＹＥＳ）、画像処理部３１は、参照領域を有効な領域とし、ｆｄに「１」を代入する（ＳＴ３３）。そして、処理はステップＳＴ３５に移行する。
【００６０】
一方、類似度の範囲が閾値以上でないと判断した場合（ＳＴ３２：ＮＯ）、画像処理部３１は、参照領域を無効な領域とし、ｆｄに「０」を代入する（ＳＴ３４）。そして、処理はステップＳＴ３５に移行する。このように、画像処理部３１は、特徴量としての類似度の変化量と、予め設定される閾値とを比較することにより、有効か否かを判断している。そして、無効の場合には、得られたオプティカルフローを後の処理に用いないようにしている。なお、以下の説明において、参照領域はすべて有効領域であるとする。
【００６１】
ステップＳＴ３５において、画像処理部３１は、領域の数だけ上記のステップＳＴ３１〜ＳＴ３４を行ったか否かを判断する（ＳＴ３５）。すなわち、画像処理部３１は、すべての参照領域について、探索領域内から類似する位置を特定したか否かを判断している。
【００６２】
いずれかの参照領域について、探索領域内から類似する位置を特定していないと判断した場合（ＳＴ３５：ＮＯ）、処理はステップＳＴ３１に戻り、類似する位置を特定していない参照領域について、上記ステップＳＴ３１〜ＳＴ３４の処理を繰り返すこととなる。
【００６３】
一方、すべての参照領域について、探索領域内から類似する位置を特定したと判断した場合（ＳＴ３５：ＹＥＳ）、画像処理部３１は、図１２に示す処理を実行する。
【００６４】
なお、オプティカルフローの計算方法は本実施形態の他に、八木信行監修, "ディジタル映像処理", 映像情報メディア学会編, pp.129-139, 2000, オーム社 などにて動画像から動きを検出する手法が複数紹介されていおり、それらを用いることもできる。
【００６５】
ここで、オプティカルフローの検出例を説明する。図９は、画角内にハンドルのスポークが進入した場合のオプティカルフローの例を示す説明図であり、（ａ）は時刻ｔにおけるオプティカルフローの例を示し、（ｂ）は時刻（ｔ＋１）におけるオプティカルフローの例を示し、（ｃ）は時刻（ｔ＋２）におけるオプティカルフローの例を示し、（ｄ）は時刻（ｔ＋３）におけるオプティカルフローの例を示している。なお、図９においては、図５と同様に、左から順に参照領域Ａ〜Ｇが配置されている。
【００６６】
まず、図９（ａ）に示すように、画角内にハンドルのスポークが進入し始める。このとき、参照領域Ｇのみから画像左上方に向けてオプティカルフローが検出されている。その後、図９（ｂ）に示すように、画角内にハンドルのスポークが更に進入して、運転者の顔の一部を覆う状態となる。このとき、参照領域Ｄ〜Ｇから画像左上方に向けてオプティカルフローが検出されている。
【００６７】
そして、図９（ｃ）に示すように、画角内にハンドルのスポークが更に進入して、運転者の顔の約半分を覆う状態となる。このとき、参照領域Ｂ〜Ｅから画像左上方に向けてオプティカルフローが検出されている。その後、図９（ｄ）に示すように、画角内にハンドルのスポークが更に進入して、運転者の眼についても覆う状態となる。このとき、参照領域Ａ〜Ｄから画像左上方に向けてオプティカルフローが検出されている。
【００６８】
このオプティカルフローを図１０に示す。図１０は、図９に示す例にて得られたオプティカルフローを示すグラフであり、（ａ）は画像縦方向における移動量を示し、（ｂ）は画像横方向における移動量を示している。なお、図１０（ａ）において、縦軸は画像縦方向の移動量を示し、横軸は時刻を示している。また、図１０（ｂ）において、縦軸は画像横方向の移動量を示し、横軸は時刻を示している。
【００６９】
まず、図１０（ａ）に示すように、画像縦方向については、画像に向かって最も右側に位置する参照領域Ｇの移動量が大きく出力される。また、時間的な遅れと共に左側の領域Ａ〜Ｆで移動量が出力されるが、左側に位置する参照領域ほど小さい移動量が出力される。
【００７０】
また、図１０（ｂ）に示すように、画像横方向については、画像中央部に当たる参照領域Ｄの移動量が最大となり、参照領域Ｄから離れた参照領域ほど小さい移動量が検出される。
【００７１】
ここで、オプティカルフロー（移動量）を時系列的に現したものが動きパターンであるため、この図１０に示すデータを予め記憶パターンとして記憶しておくことで、画角内に進入するハンドルを異物として判定することが可能となる。
【００７２】
また、ハンドルを右に切り戻した場合、すなわち、ハンドルを左に切った場合は、図１０で説明した場合と逆に上下左右の反対成分の移動量が出力される。また、画像縦方向の移動量は、領域Ａが最大となり領域Ｇが最小となる。
【００７３】
次に、オプティカルフローの他の検出例及び画像例を説明する。図１１は、画角内にハンドルのスポーク以外の異物が進入した場合のオプティカルフローの例を示す説明図であり、（ａ）は手が進入したときの画像例を示し、（ｂ）は手が進入したときのオプティカルフローの例を示し、（ｃ）は雑誌が進入したときの画像例を示し、（ｄ）は雑誌が進入したときのオプティカルフローの例を示している。なお、図１１においても、図５と同様に、左から順に参照領域Ａ〜Ｇが配置されている。
【００７４】
まず、図１１（ａ）に示すように、運転者が左を上げて左眼を擦る場合、図１１（ｂ）に示すように、画像上方に向けて参照領域Ｄ〜Ｇからオプティカルフローが検出される。また、図１１（ｃ）に示すように、運転者が雑誌等を下から持ち上げる場合、図１１（ｄ）に示すように、画像上方に向けて参照領域Ａ〜Ｇのすべてからオプティカルフローが検出される。
【００７５】
この図１１に示すオプティカルフロー（移動量）についても、時系列的にに示すデータとして、記憶しておくことで、画角内に進入する手や雑誌を異物として判定することが可能となる。
【００７６】
なお、図９〜図１１から明らかなように、異物の進入は、画像端部に設けられた参照領域に基づいて行われる。進入を検出するためには、端部において進入を判断すればよく、端部に限定することで、できるだけ計算負荷を軽減することができるからである。
【００７７】
図１２は、動きパターンに基づいて、異物の進入を判定する動作を示すフローチャートである。同図に示すように、画像処理部３１は、上記に示した動きパターンと、予め記憶される複数の記憶パターンそれぞれとの相関を求める（ＳＴ４０）。
【００７８】
相関を求める方法としては、例えば数３と同様にして求めたり、フーリエ変換やウェブレット変換により周波数解析した情報を用いて求めたりする。ここで具体的に、動きパターンＰ及び記憶パターンＤは、
【数４】

となっている。なお、上記「ｓｔａｔｅ ｃｏｄｅ」は進入した異物を表すコードである。また、「ｄａｔａ」は、移動量を示すものである。
【００７９】
その後、画像処理部３１は、複数の記憶パターンのうち、最も相関の高い記憶パターンを検出する（ＳＴ４１）。検出後、画像処理部３１は、検出された記憶パターンが示す状態を、現在画角内に進入した異物とする（ＳＴ４２）。そして、画像処理部３１は、この検出結果を報知禁止部３２に出力する。
【００８０】
なお、図１２の処理において、画像処理部３１は、最も相関の高い記憶パターンの相関値が所定の閾値を超えない場合には、異物の進入自体がなかったとして結果を得る。
【００８１】
そして、報知禁止部３２は、ここでの結果に基づいて報知の禁止をするか否かを決定し、禁止する場合には報知部２２に対し報知信号の出力を禁止すべく禁止信号等を出力することとなる。
【００８２】
このようにして、本実施形態に係る運転者状態検出装置１では、２枚以上の顔の画像間のオプティカルフローを計算し、計算結果に応じて報知を禁止している。まず、オプティカルフローを検出すると、運転者の顔周辺の物体等の動きを捕らえることができる。このため、運転者の手などの異物が画角内に進入してきた場合であっても、その動きを捕らえることができ、異物が顔部位を覆ってしまうなど、画像認識処理が不安定となる異常状態の検知が可能となる。そして、オプティカルフローの計算結果に応じて報知を禁止するため、画像認識処理が不安定となる場合に、報知動作を禁止することができる。
【００８３】
従って、誤報知の頻度を低減させることができる。
【００８４】
また、画像端部からの異物の進入を検出している。すなわち、異物の進入を画像端部において検出している。このため、できるだけ計算負荷を軽減させ、且つ的確に異物の進入を判断して、不可状態を精度良く検出することができる。
【００８５】
また、オプティカルフローの計算は予め画像上に設定された７つの参照領域にて行われ、これら７つは画像下端部に配置されている。このため、画像内への異物の進入判定について高速化を図ることができる。
【００８６】
なお、上記参照領域は、画像下端部に７つ配置されていたが、これに限るものではない。例えば、図１３及び図１４に示すようにしてもよい。図１３及び図１４は、参照領域の配置例を示す図である。まず、図１３に示すように、参照領域は、画像下端部及び左右端部に配置されている。この場合であっても、同図に示すように、オプティカルフローが検出されている。
【００８７】
また、図１４に示すように、画像周囲部、すなわち画像の下端部、左右端部、上端部のそれぞれに配置するようにしてもよい。この場合であっても、同図に示すように、オプティカルフローが検出されている。ここで、参照領域の数が多い方が、より詳細にオプティカルフローを検出できる。
【００８８】
従って、画像下端部及び左右端部に参照領域を配置した場合、画像内に進入する異物の判定精度を向上させることができる。また、画像周囲部に参照領域を配置した場合には、画像内に進入する異物の判定精度を更に向上させることができる。
【００８９】
さらに、特許文献１に記載の装置では、蛇角センサを用いる構成が提案されている。すなわち、蛇角センサにより運転者の視認方向を推定し、その推定方向を向いていないときに脇見であると判定するようにして、誤報知を抑制している。ところが、本実施形態では、上記したように舵角センサを用いずとも誤報知が抑制でき、構成の簡素化及びコストの低減を図ることができる。
【００９０】
次に、本発明の第２実施形態を説明する。第２実施形態に係る運転者状態検出装置２は、第１実施形態のものと同様であるが、異物進入判定装置３０内に、報知禁止部３２に代えて演算処理中断部（演算処理中断手段）３３を備える点で、第１実施形態のものと異なっている。
【００９１】
以下、第２実施形態に係る運転者状態検出装置２について説明する。図１５は、本発明の第２実施形態に係る運転者状態検出装置２のブロック図である。同図に示す演算処理中断部３３は、画像処理部３１の計算結果に応じて、運転者状態判定部２１での演算処理を中断させるものである。すなわち、報知禁止部３２は報知部２２又は報知器４０に対して報知の禁止を行うものであるのに対し、演算処理中断部３３は、運転者状態判定部２１が実行する報知前段の処理を中断させて報知を禁止させるものである。このため、第２実施形態に係る装置２は、運転者状態の判定結果及びオプティカルフローの計算結果に基づいて、運転者状態の判定処理を継続するか否かを決定するものといえる。
【００９２】
図１６は、本実施形態に係る運転者状態検出装置２の詳細な動作を示すフローチャートである。まず、図１６に示すステップＳＴ５０〜ＳＴ５６については、図６に示したステップＳＴ１０〜ＳＴ１６と同様であるため、説明を省略する。
【００９３】
ステップＳＴ５７において、画像処理部３１は、オプティカルフローによる異物の進入判定処理を実行する（ＳＴ５７）。この処理は、上記ステップＳＴ５０〜ＳＴ５７の間に行われている。
【００９４】
その後、演算処理中断部３３は、画角内への異物の進入が検出されたか否かを判断する（ＳＴ５８）。画角内への異物の進入が検出されたと判断した場合（ＳＴ５８：ＹＥＳ）、処理はステップＳＴ５０に戻る。すなわち、演算処理中断部３３は、運転者状態判定部２１による演算処理を中断させ、処理をステップＳＴ５０に戻す。
【００９５】
一方、画角内への異物の進入が検出されなかったと判断した場合（ＳＴ５８：ＮＯ）、演算処理中断部３３は、運転者状態判定部２１による演算処理を中断させない。このため、以下の処理が実行される。
【００９６】
すなわち、運転者状態判定部２１は、眼の位置に基づき脇見検出処理を実行し（ＳＴ５９）、運転者が脇見状態であるか否かを判断する（ＳＴ６０）。運転者が脇見状態であると判断した場合（ＳＴ６０：ＹＥＳ）、処理はステップＳＴ６３に移行する。
【００９７】
一方、運転者が脇見状態でないと判断した場合（ＳＴ６０：ＮＯ）、運転者状態判定部２１は、閉眼発生頻度に基づいて居眠りの検出処理を実行する（ＳＴ６１）。そして、運転者状態判定部２１は、運転者が居眠り状態であるか否かを判断する（ＳＴ６２）。運転者が居眠り状態でないと判断した場合（ＳＴ６２：ＮＯ）、処理はステップＳＴ１０に戻り、再度運転者状態の検出が行われていくこととなる。
【００９８】
一方、運転者が居眠り状態であると判断した場合（ＳＴ６２：ＹＥＳ）、処理はステップＳＴ６３に移行する。ステップＳＴ６３において、報知部２２は、報知信号を報知器４０に出力し、運転者への注意喚起が行われることとなる（ＳＴ６３）。その後、処理はステップＳＴ１０に戻る。
【００９９】
このようにして、本実施形態に係る運転者状態検出装置２によれば、２枚以上の顔の画像間のオプティカルフローを計算し、計算結果に応じて運転者状態判定部２１により演算処理を中断させている。まず、オプティカルフローを検出すると、運転者の顔周辺の物体等の動きを捕らえることができる。このため、運転者の手などの異物が画角内に進入してきた場合であっても、その動きを捕らえることができ、異物が顔部位を覆ってしまうなど、画像認識処理が不安定となる異常状態の検知が可能となる。そして、オプティカルフローの計算結果に応じて報知前段の演算処理を中断させるため、画像認識処理が不安定となる場合に、報知動作を禁止することができる。
【０１００】
従って、誤報知の頻度を低減させることができる。
【０１０１】
また、第１実施形態と同様に、できるだけ計算負荷を軽減させ、且つ的確に異物の進入を判断して、不可状態を精度良く検出することができる。また、画像内への異物の進入判定について高速化を図ることができる。また、構成の簡素化及びコストの低減を図ることができる。さらに、図１３及び図１４に示すように、進入判定の高速化を図ってもよい。
【０１０２】
また、従来技術では、異常処理が或る程度の時間継続して発生した場合、ステップＳＴ５５のような判段処理を行って、正常な運転者状態に復帰させるバックアップ処理が行われている。しかし、その状態が異常か否かを判定するに当たっては、ある程度の時間を要するという問題点があった。つまり、その間は、脇見検出も居眠り検出もできない時間が発生することになる。ところが、本実施形態では、ハンドルや手などの画角内への進入による処理の発散である場合、運転者状態検出ができないのは、実際にハンドルや手が画角内に入っている時間のみに限定でき、実質、運転者状態検出精度を向上させることができる。従って、判定を数秒内に行わなければならない脇見判定などに特に有効であり、効果的に誤報知の防止ができる。
【０１０３】
次に、本発明の第３実施形態を説明する。第３実施形態に係る運転者状態検出装置３は、第２実施形態のものと同様であるが、異物進入判定装置３０内に、演算切換部（演算切替手段）３４を備える点で、第２実施形態のものと異なっている。
【０１０４】
以下、第３実施形態に係る運転者状態検出装置３について説明する。図１７は、本発明の第３実施形態に係る運転者状態検出装置３のブロック図である。同図に示す演算処理切換部３４は、画角内にハンドルが進入してきた場合に、顔部位の移動状態を検出している処理から、顔部位の位置を検出する処理に切り換えるものである。
【０１０５】
すなわち、顔部位が眼である場合、眼の移動状態を検出して脇見を検出する処理から、眼の位置を検出する処理に切り換えるものである。ここで、眼の位置を検出する処理に切り換えるとは、追跡領域をクリアして再度画像全体から眼の位置を検出することをいう。
【０１０６】
図１８は、本実施形態に係る運転者状態検出装置３の詳細な動作を示すフローチャートである。まず、図１８に示すステップＳＴ７０〜ＳＴ７５については、図１６に示したステップＳＴ５０〜ＳＴ５５と同様であるため、説明を省略する。
【０１０７】
ステップＳＴ７６において、画像処理部３１は、オプティカルフローによる異物の進入判定処理を実行する（ＳＴ７６）。この処理は、上記ステップＳＴ７０〜ＳＴ７５の間に行われている。
【０１０８】
その後、画像処理部３１は、画角内へのハンドルの進入が検出されたか否かを判断する（ＳＴ７７）。画角内へのハンドルの進入が検出されたと判断した場合（ＳＴ７７：ＹＥＳ）、眼の追跡領域をクリアし（ＳＴ７８）、処理はステップＳＴ７０に戻る。すなわち、演算処理切換部３４は、眼の移動状態による脇見検出を行うべきところを、眼の位置の検出処理に戻す指令を送出し、処理の切換を行う。そして、この指令を受けた運転者状態判定部２１は、ステップＳＴ７８に示すように目の追跡領域をクリアする。
【０１０９】
画角内へのハンドルの進入が検出されなかったと判断した場合（ＳＴ７７：ＮＯ）、画像処理部３１は、画角内へのハンドル以外の進入が検出されたか否かを判断する（ＳＴ７９）。画角内へのハンドル以外の進入が検出されたと判断した場合（ＳＴ７９：ＹＥＳ）、処理はステップＳＴ７０に戻る。
【０１１０】
一方、画角内へのハンドル以外の進入が検出されなかったと判断した場合（ＳＴ７９：ＮＯ）、処理はステップＳＴ８０に移行する。ステップＳＴ８０〜ＳＴ８４は、図１６に示したステップＳＴ５９〜ＳＴ６３と同様であるため、省略する。
【０１１１】
このようにして、本実施形態に係る運転者状態検出装置３によれば、第２実施形態と同様に、誤報知の頻度を低減させることができる。
【０１１２】
また、異物がハンドルである場合には、例えば、顔部位を眼とした場合、ハンドルのスポーク部により眼が覆い隠されることがある。また、ハンドルのスポーク部により眼が隠れてしまう状態は運転者がハンドル操作をしている状態であるため、カーブ区間を走行中である可能性が高い。さらに、カーブ区間を走行する場合には、運転者はカーブの先を視認しており、正面方向を視認していない場合が多い。
【０１１３】
このような場合に、カーブ区間の走行が終わったときには、運転者は素早く正面方向を視認するが、依然としてスポーク部により眼が隠れていることがある。その後、運転者がハンドルを元に戻しスポーク部により眼が隠されなくなったとしても、スポーク部により眼が隠される前にカーブ方向の視認が検出されていた場合には、隠される前と後とで眼の位置が大きく異なってしまう。このため、検出が不正確になりやすく、誤報知の可能性が増加する。
【０１１４】
ところが、本実施形態によれば、異物がハンドルである場合に、顔部位の移動状態を検出する処理から顔部位の位置を検出処理に切り換えているので、上記問題が発生せず、誤報知の頻度を一層低減させることができる。
【０１１５】
また、第２実施形態と同様に、できるだけ計算負荷を軽減させ、且つ的確に異物の進入を判断して、不可状態を精度良く検出することができる。また、画像内への異物の進入判定について高速化を図ることができる。また、構成の簡素化及びコストの低減を図ることができ、効果的に誤報知の防止ができる。さらに、図１３及び図１４に示すように、進入判定の高速化を図ってもよい。
【０１１６】
なお、本実施形態では、異物の進入に対しての誤報知低減を図っているが、本発明はこれに限定されず、進入に限らず画角内からの異物の退出についても同様に行うことができる。例えば、手が眼を覆っているときは、正しい脇見判定や開閉眼判定を行うことができないが、画像認識処理が不安定になる要因は、その時間だけでなく、眼から手が離れる際にも発生する。つまり、眼の追跡処理の判定状態によっては、離れる手を眼のデータだと誤認識してしまう可能性がある。このため、誤報知を発する可能性もあることとなる。本発明は、この異物の退出の際の誤報知も防止できる。
【図面の簡単な説明】
【図１】本発明の第１実施形態に係る運転者状態検出装置の構成図である。
【図２】本発明の第１実施形態に係る運転者状態検出装置１の詳細構成図である。
【図３】本実施形態に係る運転者状態判定装置２０の動作の概略を示すデータフローダイヤグラムである。
【図４】参照領域及び探索領域の説明図である。
【図５】画像に配置される参照領域の説明図である。
【図６】本実施形態に係る運転者状態検出装置１の詳細な動作を示すフローチャートである。
【図７】図６に示した異物進入判定処理（ステップＳＴ２１）の詳細なフローチャートである。
【図８】図７に示すステップＳＴ３１における移動量（ｘｄ，ｙｄ）の算出方法の説明図である。
【図９】画角内にハンドルのスポークが進入した場合のオプティカルフローの例を示す説明図であり、（ａ）は時刻ｔにおけるオプティカルフローの例を示し、（ｂ）は時刻（ｔ＋１）におけるオプティカルフローの例を示し、（ｃ）は時刻（ｔ＋２）におけるオプティカルフローの例を示し、（ｄ）は時刻（ｔ＋３）におけるオプティカルフローの例を示している。
【図１０】図９に示す例にて得られたオプティカルフローを示すグラフであり、（ａ）は画像縦方向における移動量を示し、（ｂ）は画像横方向における移動量を示している。
【図１１】画角内にハンドルのスポーク以外の異物が進入した場合のオプティカルフローの例を示す説明図であり、（ａ）は手が進入したときの画像例を示し、（ｂ）は手が進入したときのオプティカルフローの例を示し、（ｃ）は雑誌が進入したときの画像例を示し、（ｄ）は雑誌が進入したときのオプティカルフローの例を示している。
【図１２】動きパターンに基づいて、異物の進入を判定する動作を示すフローチャートである。
【図１３】参照領域の第１配置例を示す図である。
【図１４】参照領域の第２配置例を示す図である。
【図１５】本発明の第２実施形態に係る運転者状態検出装置２のブロック図である。
【図１６】本実施形態に係る運転者状態検出装置２の詳細な動作を示すフローチャートである。
【図１７】本発明の第３実施形態に係る運転者状態検出装置３のブロック図である。
【図１８】本実施形態に係る運転者状態検出装置３の詳細な動作を示すフローチャートである。
【符号の説明】
１〜３…運転者状態検出装置
１０…撮影部（撮影手段）
２１…運転者状態判定部（運転者状態判定手段）
２２…報知部（報知手段）
３１…画像処理部（画像処理手段）
３２…報知禁止部（報知禁止手段）
３３…演算処理中断部（演算処理中断手段）
３４…演算処理切換部（演算処理切換手段）[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a driver state detection device.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, there has been known an apparatus that takes a face image of a vehicle driver, monitors the face direction or line of sight, and makes a side look determination. In this device, when a driver's side aside is detected, a warning device or the like is used to alert the driver (see Patent Documents 1 and 2).
[0003]
[Patent Document 1]
JP-A-6-243367
[0004]
[Patent Document 2]
JP 2000-172966 A
[0005]
[Problems to be solved by the invention]
However, in the armpit detection devices described in Patent Documents 1 and 2, the driver's hand may enter the angle of view where the driver is touching the nose, such as when shooting a face image. In such a case, in a conventional apparatus, if a hand covers a face part such as a nose or an eye that is a detection target, the image recognition process may become unstable and erroneous notification may occur. For this reason, the conventional device has a problem that the frequency of false notifications increases.
[0006]
Note that this problem is not limited to the case where a hand enters the angle of view, but also occurs when a foreign object other than the face enters. In addition, the problem occurs not only in the aside detection device but also in a device that acquires a driver's face image and performs a predetermined process, such as a dozing detection device.
[0007]
[Means for Solving the Problems]
  The driver state detection apparatus according to the present invention includes a photographing unit that photographs a driver's face, and a driver that determines a driver state by detecting a movement state of a facial part from a face image photographed by the photographing unit. An image based on two or more facial images photographed by the photographing means and a notifying means for outputting a notice signal for notifying the driver based on the judgment result by the state judging means and the driver state judging means. The image processing means for detecting whether or not the foreign matter entering and exiting from the image end based on the optical flow calculation result is a handle, and the foreign matter entering and exiting from the image end by the image processing means When it is detected that the vehicle is a steering wheel, the driver state determination unit detects the position of the face part from the face image captured by the imaging unit, and determines the driver state.
[0008]
【The invention's effect】
  According to the driver state detection device of the present invention, when the foreign matter entering / leaving from the image end is a handle, the process of detecting the movement state of the face part is switched to the process of detecting the position of the face part. Can be reduced.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Preferred embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a configuration diagram of a driver state detection device according to a first embodiment of the present invention. As shown in the figure, the driver state detection device 1 according to the present embodiment detects whether or not detection is an undesirable state, that is, an impossible state, when detecting a look-aside and doze as the driver state. To do. The driver state detection device 1 prohibits notification to the driver when it is detected that the driver state is disabled.
[0011]
The driver state detection device 1 is based on an imaging unit (imaging unit) 10 that captures the face of a driver of a vehicle and a face image (hereinafter referred to as a face image) captured by the imaging unit 10. A driver state determination device 20 that determines the state is provided. Furthermore, the driver state detection device 1 is based on the face image captured by the photographing unit 10 and the determination result by the foreign object entry determination device 30 and the driver state determination device 20 that detect the entry of a foreign object into the angle of view. And a notification device 40 that performs a notification operation to notify the driver.
[0012]
Specifically, the imaging unit 10 is, for example, a CCD camera for imaging visible light, and is installed below the front of the driver. Further, the foreign object approach determination device 30 detects, for example, an object other than the face (for example, a driver's hand or magazine) in the image captured by the imaging unit 10 as a foreign object.
[0013]
Next, the detailed configuration of the device 1, in particular, the driver state determination device 20 and the foreign object entry determination device 30 will be described with reference to FIG. 2. FIG. 2 is a detailed configuration diagram of the driver state detection device 1 according to the first embodiment of the present invention.
[0014]
As shown in the figure, the driver state determination device 20 includes a driver state determination unit (driver state determination unit) 21 and a notification unit (notification unit) 22. The driver state determination unit 21 detects at least one of the position of the driver's face part, the movement state of the face part, and the open / closed state of the eyes from the face image photographed by the photographing unit 10, and The state is determined. Here, examples of the facial part include a driver's eyes (only one of them may be used), a nose, and a mouth.
[0015]
Moreover, the alerting | reporting part 22 outputs the alerting | reporting signal for alerting | reporting to a driver | operator based on the determination result by the driver | operator state determination part 21. FIG. The notification unit 22 is connected to the notification device 40 and is configured to output a notification signal to the notification device 40. Therefore, when the notification device 40 receives the notification signal, the notification device 40 performs a notification operation.
[0016]
The foreign matter entry determination device 30 includes an image processing unit (image processing unit) 31 and a notification prohibiting unit (notification prohibiting unit) 32. The image processing unit 31 calculates an optical flow between images based on two or more face images photographed by the photographing unit 10. Specifically, the image processing unit 31 sequentially inputs image data from the photographing unit 10, and based on at least two pieces of the image data, an optical that represents a moving direction and a moving speed of an object in the image based on a density difference or the like. The flow is calculated. Here, the calculation of the optical flow uses a reference area (calculation area) set in advance on the image and a search area. These will be described later.
[0017]
The notification prohibition unit 32 prohibits the notification to the driver by prohibiting the output of the notification signal from the notification unit 22 according to the calculation result of the image processing unit 31. When the optical flow calculated by the image processing unit 31 is in an unacceptable state indicating entry of a foreign object or the like, the notification prohibiting unit 32 may cause the determination result of the driver state determination unit 21 to be inaccurate. The output of the notification signal is prohibited. That is, there is a possibility that the position of the driver's face part, the movement state of the face part, and the open / closed state of the eyes may not be accurately determined when a hand or the like enters the angle of view and covers the driver's face part. is there. Therefore, if a notification operation is performed in such a case, there is a possibility that a false alarm is generated, so that the output of a notification signal is prohibited.
[0018]
The notification prohibition unit 32 is not limited to the configuration for prohibiting the signal output of the notification unit 22, and may be configured to send a prohibition command directly to the notification device 40. In other words, the notification prohibition unit 32 may be configured to prohibit notification when the calculated optical flow indicates a disabled state.
[0019]
Next, an outline of the operation of the apparatus 1 will be described. FIG. 3 is a data flow diagram showing an outline of the operation of the driver state determination device 20 according to the present embodiment.
[0020]
First, an image including the driver's face is photographed by the photographing unit 10 (image shown in FIG. 3A), and the image is input to the driver state determination unit 21 and the image processing unit 31. After the input, the driver state determination unit 21 detects the driver state by detecting at least one of the position of the driver's face part, the movement state of the face part, and the open / closed state of the eyes. The determination here is performed, for example, as described in JP-A-8-207617, JP-A-10-40361, JP-A-10-44824, and JP-A-11-339200.
[0021]
Then, after the determination of the driver state, the driver state determination unit 21 sends the determination result to the notification unit 22. The notification unit 22 determines the necessity of notification to the driver according to the determination result, and outputs a notification signal to the notification device 40 when necessary.
[0022]
On the other hand, during the above operation, the image processing unit 31 obtains an optical flow between images based on image data from the photographing unit 10 (FIG. 3b). This optical flow represents the amount of movement in each region in the image. Specifically, the optical flow indicates how fast and in what direction the face portion or the like existing at a predetermined location of the image has moved in the subsequent image. That is, the optical flow includes size and direction information.
[0023]
Here, the calculation of the optical flow will be described in more detail. When obtaining an optical flow, the image processing unit 31 inputs data of a reference area and a search area.
[0024]
FIG. 4 is an explanatory diagram of the reference area and the search area. If the reference area is set on an image at a certain time, the search area is set on an image after that time. In FIG. 4, for the sake of convenience, the width is w pixels and the height is set. This will be described on one image of h pixels.
[0025]
As shown in the figure, the reference area is an area having a width tw pixels and a height th pixels set at the position (tx, ty). The search area is an area having a width sw pixel and a height sh pixel set at the position (sx, sy). This search area is set so as to surround each reference area, and is set in the same number as the reference area. Further, although the reference area and the search area are different in size, they coincide at a specific point O with respect to the center position.
[0026]
In this way, these two regions are set to have the same center and have a relationship of sw> tw and sh> sw. The reference area and the search area here are set at a predetermined position and size without depending on the position of the driver's face.
[0027]
In addition, the reference area is desirably arranged as follows. FIG. 5 is an explanatory diagram of reference areas arranged in an image. For example, as shown in FIG. 5, a plurality (for example, seven) of reference areas are arranged in the horizontal direction at the lower end of the image.
[0028]
Again, a description will be given with reference to FIG. After calculating the optical flow, the image processing unit 31 obtains a motion pattern from the obtained optical flow for each region (FIG. 3c). This motion pattern represents, for example, motion obtained from an optical flow as time-series data. The movement pattern is limited because, for example, when the spoke part of the handle enters the angle of view, the handle shows only a certain movement. As described above, the motion pattern is determined to some extent by each motion of the object or the like.
[0029]
After calculating the motion pattern, the image processing unit 31 sends the motion pattern data to the notification prohibiting unit 32. After the transmission, the notification prohibition unit 32 collates the obtained motion pattern with a previously stored memory pattern (FIG. 3d). Then, by this collation, it is determined whether or not a foreign object has entered the angle of view, and a result is obtained (FIG. 3e).
[0030]
When the entry of a foreign object is determined, the notification prohibition unit 32 prohibits the notification operation from being performed by prohibiting the output of the notification signal that should have been output by the notification unit 22. As described above, the device 1 determines whether or not to notify the driver based on the determination result of the driver state and the calculation result of the optical flow. The above is the outline of the operation.
[0031]
Next, detailed operation of the apparatus 1 will be described. FIG. 6 is a flowchart showing a detailed operation of the driver state detection device 1 according to the present embodiment. In FIG. 6, the position of the driver's eyes and the moving state of the eyes are detected as the position of the driver's face and the moving state of the face. Further, in FIG. 6, the driver's eyes are detected based on the driver's eye position and eye movement state, and the driver's doze is detected based on the driver's eye open / closed state. And
[0032]
First, the photographing unit 10 captures an image of the driver's face and sends the obtained face image data to the driver state determination unit 21 and the image processing unit 31. Then, the driver state determination unit 21 and the image processing unit 31 input face image data (ST10).
[0033]
After the input, the driver state determination unit 21 determines whether or not an eye tracking region exists (ST11). Here, the tracking area is an area smaller than the entire image set on the face image, and is provided to easily find the position of the driver's eyes from the face image. Also, since the tracking area is set as a small area including the eyes, the image resolution can be increased and the detection accuracy of the eye position can be improved as compared with the case where the eye position is detected from the entire image. It can be done. For this reason, when the tracking area is set, the driver state determination unit 21 searches for the driver's eyes from the tracking area. On the other hand, when the tracking area is not set, the position of the driver's eyes is searched from the entire image.
[0034]
Further, the size of the tracking area is set so as to allow the amount of eye movement, so that the eye can always be caught in the same area. In this case, processing such as replacing the moving speed of the eye with a moving amount that moves between frames is required so as to allow the moving amount of the eye. A specific example of the allowable range is a range when the driver moves his / her face in a predetermined direction by about −40 ° to 40 °.
[0035]
If it is determined that the eye tracking area is set (ST11: YES), the process proceeds to step ST14. On the other hand, when it is determined that the eye tracking region is not set (ST11: NO), the driver state determination unit 21 detects the eye position from the entire image (ST12). Then, the driver state determination unit 21 acquires eye position data.
[0036]
Thereafter, the driver state determination unit 21 sets a tracking region based on the eye position data (ST13). Then, the driver state determination unit 21 detects eyes from within the tracking region for a newly input face image (ST14). Thereafter, the driver state determination unit 21 determines whether or not eye tracking is correctly performed (ST15).
[0037]
When it is determined that the eye tracking is not performed correctly (ST15: NO), the driver state determination unit 21 clears the eye tracking region (ST16), and the process returns to step ST10. In this case, the tracking area is set again.
[0038]
On the other hand, when it is determined that the eye tracking is correctly performed (ST15: YES), the driver state determination unit 21 performs a look-aside detection process based on the eye position (ST17), and the driver is in the look-ahead state. It is determined whether or not there is (ST18). Here, the look-ahead detection process is not limited to the case of being performed based on the eye position itself, and may be performed based on the eye movement state obtained from the eye position. Further, it may be based on both the eye position itself and the eye movement state.
[0039]
When it is determined that the driver is not looking aside (ST18: NO), the driver state determination unit 21 performs a dozing detection process based on the frequency of occurrence of closed eyes (ST19). Then, the driver state determination unit 21 determines whether or not the driver is in a dozing state (ST20). When it is determined that the driver is not in a doze state (ST20: NO), the process returns to step ST10, and the driver state is detected again.
[0040]
On the other hand, when it is determined that the driver is looking aside (ST18: YES) or when the driver is determined to be asleep (ST20: YES), the process proceeds to step ST22.
[0041]
In addition, during the above process, the image processing unit 31 performs a foreign substance entry determination process using an optical flow (ST21). Then, after determining the entry of a foreign object, the process proceeds to step ST22.
[0042]
In step ST22, the notification prohibition unit 32 determines whether or not entry of a foreign object into the angle of view has been detected (ST22). When it is determined that entry of a foreign object into the angle of view is detected (ST22: YES), the notification prohibition unit 32 prohibits the notification unit 22 from outputting a notification signal. For this reason, the alerting | reporting part 22 does not output an alerting | reporting signal to the alerting | reporting device 40, and a process returns to step ST10.
[0043]
On the other hand, when it is determined that no foreign object has entered the angle of view (ST22: NO), the notification prohibition unit 32 does not prohibit the notification unit 22 from outputting a notification signal. For this reason, the alerting | reporting part 22 will output an alerting | reporting signal to the alerting device 40, and a driver | operator will be alerted (ST23). Thereafter, the process returns to step ST10.
[0044]
It should be noted that a case where a foreign object enters the angle of view and a false notification is made is as follows. For example, when a vehicle approaches a corner and a steering wheel spoke enters the angle of view by a driver's steering operation, or when a magazine is lifted to view a road map or the like. This is because in these cases, the driver's eyes may be covered with handle spokes, magazines, or the like, and accurate detection may not be possible.
[0045]
Next, the foreign object entry determination process (step ST21) will be described in detail. FIG. 7 is a detailed flowchart of the foreign object entry determination process (step ST21) shown in FIG.
[0046]
After inputting the face image data, the image processing unit 31 applies a smoothing filter to the face image and converts the pixel value using a predetermined formula (ST30). Here, the smoothing filter is a filter having 5 rows and 5 columns as shown below.
[0047]
[Expression 1]

The predetermined formula is shown below.
[0048]
[Expression 2]

Note that d (x, y) is a pixel value at an arbitrary position in the captured image, and d ′ (x, y) is a pixel value after conversion.
[0049]
Thereafter, the image processing unit 31 obtains the position most similar to the reference area in the previous face image from the current face image search area, and calculates the movement amount (xd, yd), that is, the optical flow ( ST31).
[0050]
Specifically, the image processing unit 31 first obtains an area most similar to the reference area from the search area, and sets the center point of the most similar area as the position most similar to the reference area. Then, the image processing unit 31 calculates the movement amount (xd, yd) from the obtained center point of the most similar region and the center point of the search region, and sets it as an optical flow.
[0051]
Here, step ST31 will be described in detail. As described above, a reference area is set in advance on the face image. The search area is set so as to surround the reference area. The reference area and the search area are set at different times. Specifically, as shown in FIG. 8, the reference area is set at time t, and the search area is set at time (t + 1) after time t.
[0052]
FIG. 8 is an explanatory diagram of a method of calculating the movement amount (xd, yd) in step ST31 shown in FIG. In the process of step ST31, the image processing unit 31 first creates a candidate area within the search area at time t + 1. This candidate area is an area having the same size as the reference area. Specifically, when an arbitrary position in the search area is (xd, yd), − (sw−tw) / 2 <xd <(sw−tw) / 2 and − (sh−th) / This is an area cut out from the range of 2 <yd <(sh−th).
[0053]
The image processing unit 31 sets a candidate area as described above at a predetermined location in the search area and compares the set candidate area with a reference area to obtain a similarity. Next, the image processing unit 31 moves the candidate area to another position, compares the candidate area at the moved position with the reference area, and obtains the similarity.
[0054]
Thereafter, the image processing unit 31 sequentially moves the candidate areas, and calculates the similarity to the reference area at each location in the search area. The similarity is determined based on, for example, grayscale data. Here, in the case where the similarity is calculated based on the density data, assuming that the similarity is cos θ, the similarity is expressed by the following equation.
[0055]
[Equation 3]

In the above formula, the density data of the reference area is T, and the density data of the candidate area is S. Xd represents an X coordinate value in the search area, and yd represents a Y coordinate value in the search area.
[0056]
From the above, the image processing unit 31 determines the position S where the degree of similarity is maximum, acquires the difference between the coordinate values of the point S and the point O as the movement amount (xd, yd), and uses this as the optical flow.
[0057]
Again, a description will be given with reference to FIG. After calculating the movement amount (xd, yd), the image processing unit 31 determines whether the similarity range is equal to or greater than a threshold value (ST32). That is, the image processing unit 31 first scans the search area with the candidate area, and calculates the similarity of each location in the search area. Thereafter, the image processing unit 31 obtains the variance of the obtained similarity, and determines whether the range of the similarity is equal to or greater than a threshold value based on this variance.
[0058]
Here, the case where the range of similarity is small is a case where similar similarity is detected at each location in the search area. For example, when there are few features, such as when the reference region is a pure white image, a similar similarity result is obtained even when compared with any part in the search region. In such a case, since the difference in the degree of similarity is small, the detection of the point S at which the degree of similarity is maximum tends to be inaccurate. For this reason, in the process of step ST32 of FIG. 7, it compares with a predetermined threshold value and sorts a suitable thing and an unsuitable thing.
[0059]
If it is determined that the similarity range is equal to or greater than the threshold (ST32: YES), the image processing unit 31 sets the reference area as an effective area and substitutes “1” for fd (ST33). Then, the process proceeds to step ST35.
[0060]
On the other hand, if it is determined that the similarity range is not equal to or greater than the threshold (ST32: NO), the image processing unit 31 sets the reference area as an invalid area and substitutes “0” for fd (ST34). Then, the process proceeds to step ST35. As described above, the image processing unit 31 determines whether or not the image processing unit 31 is valid by comparing the amount of change in similarity as a feature amount with a preset threshold value. In the case of invalidity, the obtained optical flow is not used for subsequent processing. In the following description, it is assumed that all the reference areas are valid areas.
[0061]
In step ST35, the image processing unit 31 determines whether or not the above steps ST31 to ST34 have been performed for the number of regions (ST35). That is, the image processing unit 31 determines whether or not a similar position has been specified from within the search area for all reference areas.
[0062]
If it is determined that a similar position is not specified from within the search area for any of the reference areas (ST35: NO), the process returns to step ST31, and the above steps are performed for the reference area that does not specify a similar position. The processing of ST31 to ST34 will be repeated.
[0063]
On the other hand, when it is determined that similar positions are specified from the search area for all the reference areas (ST35: YES), the image processing unit 31 executes the process shown in FIG.
[0064]
In addition to this embodiment, the optical flow calculation method can be used to detect motion from moving images by Supervision of Nobuyuki Yagi, "Digital Video Processing", IPSJ, pp.129-139, 2000, Ohmsha, etc. Several methods have been introduced and can be used.
[0065]
Here, an example of optical flow detection will be described. FIG. 9 is an explanatory diagram showing an example of an optical flow when a spoke of a handle enters the angle of view, (a) shows an example of an optical flow at time t, and (b) shows an example of optical flow at time (t + 1). An example of an optical flow is shown, (c) shows an example of an optical flow at time (t + 2), and (d) shows an example of an optical flow at time (t + 3). In FIG. 9, as in FIG. 5, reference areas A to G are arranged in order from the left.
[0066]
First, as shown in FIG. 9A, the spokes of the handle begin to enter the angle of view. At this time, an optical flow is detected from only the reference region G toward the upper left of the image. Thereafter, as shown in FIG. 9 (b), the spokes of the steering wheel further enter the angle of view and cover a part of the driver's face. At this time, an optical flow is detected from the reference areas D to G toward the upper left of the image.
[0067]
And as shown in FIG.9 (c), the spoke | spoke of a handle | steering_wheel will further approach into an angle of view, and will be in the state which covers about half of a driver | operator's face. At this time, an optical flow is detected from the reference areas B to E toward the upper left of the image. Thereafter, as shown in FIG. 9 (d), the spokes of the steering wheel further enter the angle of view, and the driver's eyes are also covered. At this time, an optical flow is detected from the reference areas A to D toward the upper left of the image.
[0068]
This optical flow is shown in FIG. FIG. 10 is a graph showing the optical flow obtained in the example shown in FIG. 9, where (a) shows the amount of movement in the image vertical direction, and (b) shows the amount of movement in the image horizontal direction. In FIG. 10A, the vertical axis indicates the amount of movement in the vertical direction of the image, and the horizontal axis indicates time. In FIG. 10B, the vertical axis indicates the amount of movement in the horizontal direction of the image, and the horizontal axis indicates time.
[0069]
First, as shown in FIG. 10A, in the vertical direction of the image, the amount of movement of the reference region G located on the rightmost side toward the image is output large. In addition, the movement amount is output in the left areas A to F with time delay, but a smaller movement amount is output in the reference area located on the left side.
[0070]
Further, as shown in FIG. 10B, in the horizontal direction of the image, the movement amount of the reference region D corresponding to the center of the image is maximized, and the smaller the movement amount is detected in the reference region far from the reference region D.
[0071]
Here, since the optical flow (movement amount) in time series is a motion pattern, the handle shown in FIG. It can be determined as a foreign object.
[0072]
In addition, when the handle is turned back to the right, that is, when the handle is turned to the left, the movement amounts of the opposite components in the vertical and horizontal directions are output contrary to the case described in FIG. Further, the amount of movement in the vertical direction of the image is maximum in region A and minimum in region G.
[0073]
Next, another detection example and image example of the optical flow will be described. FIG. 11 is an explanatory diagram showing an example of an optical flow when a foreign object other than the spokes of the handle enters the angle of view. FIG. 11A shows an example of an image when a hand enters, and FIG. Shows an example of an optical flow when a magazine enters, (c) shows an example of an image when a magazine enters, and (d) shows an example of an optical flow when a magazine enters. In FIG. 11, as in FIG. 5, reference areas A to G are arranged in order from the left.
[0074]
First, as shown in FIG. 11A, when the driver raises the left and rubs the left eye, as shown in FIG. 11B, the optical flow is detected from the reference regions D to G toward the upper part of the image. Is done. Further, as shown in FIG. 11C, when the driver lifts a magazine or the like from below, as shown in FIG. 11D, the optical flow is detected from all of the reference areas A to G toward the top of the image. Is done.
[0075]
The optical flow (movement amount) shown in FIG. 11 is also stored as data shown in time series, so that it is possible to determine a hand or magazine entering the angle of view as a foreign object.
[0076]
As is apparent from FIGS. 9 to 11, the entry of the foreign matter is performed based on a reference area provided at the end of the image. This is because in order to detect the approach, it is only necessary to determine the approach at the end, and by limiting to the end, the calculation load can be reduced as much as possible.
[0077]
FIG. 12 is a flowchart illustrating an operation for determining the entry of a foreign object based on a motion pattern. As shown in the figure, the image processing unit 31 obtains a correlation between the motion pattern shown above and each of a plurality of storage patterns stored in advance (ST40).
[0078]
As a method for obtaining the correlation, for example, it is obtained in the same manner as in Equation 3, or is obtained by using information obtained by frequency analysis by Fourier transform or Weblet transform. Specifically, the motion pattern P and the memory pattern D are as follows:
[Expression 4]

It has become. The “state code” is a code representing the foreign matter that has entered. “Data” indicates the amount of movement.
[0079]
Thereafter, the image processing unit 31 detects the storage pattern having the highest correlation among the plurality of storage patterns (ST41). After the detection, the image processing unit 31 sets the state indicated by the detected storage pattern as a foreign object that has entered the current angle of view (ST42). Then, the image processing unit 31 outputs this detection result to the notification prohibiting unit 32.
[0080]
In the process of FIG. 12, the image processing unit 31 obtains a result that there is no entry of foreign matter when the correlation value of the storage pattern having the highest correlation does not exceed a predetermined threshold value.
[0081]
Then, the notification prohibition unit 32 determines whether or not to prohibit the notification based on the result here, and outputs a prohibition signal or the like to prohibit the notification unit 22 from outputting the notification signal when prohibiting. Will be.
[0082]
In this manner, the driver state detection device 1 according to the present embodiment calculates the optical flow between two or more face images, and prohibits notification according to the calculation result. First, when the optical flow is detected, it is possible to capture the movement of an object or the like around the driver's face. For this reason, even when a foreign object such as a driver's hand enters the angle of view, the movement can be captured, and the image recognition process becomes unstable, for example, the foreign object covers the face part. An abnormal state can be detected. Since the notification is prohibited according to the calculation result of the optical flow, the notification operation can be prohibited when the image recognition process becomes unstable.
[0083]
Therefore, the frequency of false notification can be reduced.
[0084]
In addition, entry of foreign matter from the edge of the image is detected. That is, the entrance of a foreign object is detected at the end of the image. For this reason, it is possible to reduce the calculation load as much as possible and accurately determine the entry of the foreign matter and detect the impossible state with high accuracy.
[0085]
The optical flow is calculated in seven reference areas set in advance on the image, and these seven are arranged at the lower end of the image. For this reason, it is possible to speed up the determination of the entry of foreign matter into the image.
[0086]
The seven reference areas are arranged at the lower end of the image, but the present invention is not limited to this. For example, you may make it show in FIG.13 and FIG.14. 13 and 14 are diagrams illustrating an example of arrangement of reference areas. First, as shown in FIG. 13, the reference areas are arranged at the lower end and the left and right ends of the image. Even in this case, an optical flow is detected as shown in FIG.
[0087]
Moreover, as shown in FIG. 14, you may make it arrange | position to each image periphery part, ie, the lower end part of an image, a left-right end part, and an upper end part. Even in this case, an optical flow is detected as shown in FIG. Here, the optical flow can be detected in more detail when the number of reference regions is larger.
[0088]
Therefore, when the reference areas are arranged at the lower end portion and the left and right end portions of the image, it is possible to improve the determination accuracy of the foreign matter entering the image. In addition, when the reference area is arranged in the periphery of the image, it is possible to further improve the determination accuracy of the foreign matter that enters the image.
[0089]
Furthermore, in the apparatus described in Patent Document 1, a configuration using a snake angle sensor has been proposed. In other words, the misleading is suppressed by estimating the driver's viewing direction with the snake angle sensor and determining that the driver is looking aside when the driver is not facing the estimated direction. However, in the present embodiment, erroneous notification can be suppressed without using a steering angle sensor as described above, and the configuration can be simplified and the cost can be reduced.
[0090]
Next, a second embodiment of the present invention will be described. The driver state detection device 2 according to the second embodiment is the same as that of the first embodiment, but in the foreign object entry determination device 30, an arithmetic processing interruption unit (arithmetic processing interruption means) is used instead of the notification prohibition unit 32. ) 33 is different from that of the first embodiment.
[0091]
Hereinafter, the driver state detection device 2 according to the second embodiment will be described. FIG. 15 is a block diagram of the driver state detection device 2 according to the second embodiment of the present invention. The arithmetic processing interrupting unit 33 shown in the figure interrupts the arithmetic processing in the driver state determining unit 21 according to the calculation result of the image processing unit 31. That is, while the notification prohibition unit 32 prohibits notification to the notification unit 22 or the notification device 40, the arithmetic processing interruption unit 33 performs the pre-notification process performed by the driver state determination unit 21. It is interrupted and the notification is prohibited. Therefore, it can be said that the device 2 according to the second embodiment determines whether or not to continue the determination process of the driver state based on the determination result of the driver state and the calculation result of the optical flow.
[0092]
FIG. 16 is a flowchart showing a detailed operation of the driver state detection device 2 according to the present embodiment. First, steps ST50 to ST56 shown in FIG. 16 are the same as steps ST10 to ST16 shown in FIG.
[0093]
In step ST57, the image processing unit 31 executes a foreign substance entry determination process using an optical flow (ST57). This process is performed between steps ST50 to ST57.
[0094]
Thereafter, the calculation processing interruption unit 33 determines whether or not the entry of a foreign object within the angle of view has been detected (ST58). If it is determined that the entry of a foreign object within the angle of view has been detected (ST58: YES), the process returns to step ST50. That is, the arithmetic processing interruption unit 33 interrupts the arithmetic processing by the driver state determination unit 21 and returns the processing to step ST50.
[0095]
On the other hand, when it is determined that no foreign object has entered the angle of view (ST58: NO), the arithmetic processing interruption unit 33 does not interrupt the arithmetic processing by the driver state determination unit 21. For this reason, the following processing is executed.
[0096]
That is, the driver state determination unit 21 performs a side-by-side detection process based on the eye position (ST59), and determines whether or not the driver is in the side-by-side state (ST60). When it is determined that the driver is looking aside (ST60: YES), the process proceeds to step ST63.
[0097]
On the other hand, when it is determined that the driver is not looking aside (ST60: NO), the driver state determination unit 21 performs a dozing detection process based on the closed eye occurrence frequency (ST61). Then, the driver state determination unit 21 determines whether or not the driver is in a dozing state (ST62). If it is determined that the driver is not in a doze state (ST62: NO), the process returns to step ST10, and the driver state is detected again.
[0098]
On the other hand, when it is determined that the driver is dozing (ST62: YES), the process proceeds to step ST63. In step ST63, the notification unit 22 outputs a notification signal to the notification device 40 to alert the driver (ST63). Thereafter, the process returns to step ST10.
[0099]
Thus, according to the driver state detection device 2 according to the present embodiment, the optical flow between two or more face images is calculated, and the driver state determination unit 21 performs arithmetic processing according to the calculation result. Suspended. First, when the optical flow is detected, it is possible to capture the movement of an object or the like around the driver's face. For this reason, even when a foreign object such as a driver's hand enters the angle of view, the movement can be captured, and the image recognition process becomes unstable, for example, the foreign object covers the face part. An abnormal state can be detected. Then, since the calculation process before the notification is interrupted according to the calculation result of the optical flow, the notification operation can be prohibited when the image recognition process becomes unstable.
[0100]
Therefore, the frequency of false notification can be reduced.
[0101]
Further, similarly to the first embodiment, the calculation load can be reduced as much as possible, and the entry of a foreign object can be accurately determined to detect a disabled state with high accuracy. In addition, it is possible to speed up the determination of the entry of foreign matter into the image. Further, the configuration can be simplified and the cost can be reduced. Furthermore, as shown in FIGS. 13 and 14, the speed of the entry determination may be increased.
[0102]
Further, in the prior art, when the abnormal process continues for a certain period of time, a backup process is performed in which a decision process such as step ST55 is performed to return to a normal driver state. However, there is a problem that it takes a certain amount of time to determine whether or not the state is abnormal. That is, during that time, there will be a time during which neither aside detection nor dozing detection can be performed. However, in this embodiment, in the case of divergence of processing due to the entry of the handle or hand into the angle of view, the driver state cannot be detected only when the handle or hand is actually within the angle of view. The driver state detection accuracy can be substantially improved. Therefore, it is particularly effective for a look-aside determination in which the determination must be made within a few seconds, and erroneous notification can be effectively prevented.
[0103]
Next, a third embodiment of the present invention will be described. The driver state detection device 3 according to the third embodiment is the same as that of the second embodiment, but the second is that the foreign matter entry determination device 30 includes a calculation switching unit (calculation switching means) 34. It differs from that of the embodiment.
[0104]
Hereinafter, the driver state detection device 3 according to the third embodiment will be described. FIG. 17 is a block diagram of the driver state detection device 3 according to the third embodiment of the present invention. The arithmetic processing switching unit 34 shown in FIG. 6 switches from processing for detecting the moving state of the facial part to processing for detecting the position of the facial part when the handle enters the angle of view.
[0105]
That is, when the face part is an eye, the process is switched from the process of detecting the movement state of the eye to detect a side look to the process of detecting the position of the eye. Here, switching to the process of detecting the eye position means that the tracking area is cleared and the eye position is detected again from the entire image.
[0106]
FIG. 18 is a flowchart showing a detailed operation of the driver state detection device 3 according to the present embodiment. First, steps ST70 to ST75 shown in FIG. 18 are the same as steps ST50 to ST55 shown in FIG.
[0107]
In step ST76, the image processing unit 31 executes a foreign substance entry determination process using an optical flow (ST76). This process is performed between steps ST70 to ST75.
[0108]
Thereafter, the image processing unit 31 determines whether or not the entry of the handle into the angle of view has been detected (ST77). If it is determined that the entry of the handle into the angle of view has been detected (ST77: YES), the eye tracking area is cleared (ST78), and the process returns to step ST70. That is, the arithmetic processing switching unit 34 sends a command for returning to the eye position detection process the place where the eye movement detection based on the eye movement state should be performed, and switches the process. Upon receiving this command, the driver state determination unit 21 clears the eye tracking region as shown in step ST78.
[0109]
If it is determined that the entry of the handle into the angle of view has not been detected (ST77: NO), the image processing unit 31 determines whether or not an entry other than the handle into the angle of view has been detected (ST79). If it is determined that an entry other than the handle into the angle of view has been detected (ST79: YES), the process returns to step ST70.
[0110]
On the other hand, if it is determined that no entry other than the steering wheel into the angle of view has been detected (ST79: NO), the process proceeds to step ST80. Steps ST80 to ST84 are the same as steps ST59 to ST63 shown in FIG.
[0111]
Thus, according to the driver state detection device 3 according to the present embodiment, the frequency of false notifications can be reduced as in the second embodiment.
[0112]
Further, when the foreign object is a handle, for example, when the face part is an eye, the eye may be covered by the spoke part of the handle. Moreover, since the state where the eyes are hidden by the spoke portion of the steering wheel is a state where the driver is operating the steering wheel, there is a high possibility that the vehicle is traveling in a curve section. Furthermore, when traveling in a curve section, the driver often visually recognizes the tip of the curve and does not visually recognize the front direction.
[0113]
In such a case, when the traveling of the curve section is finished, the driver quickly visually recognizes the front direction, but the eyes may still be hidden by the spoke portion. After that, even if the driver puts the steering wheel back and the eyes are no longer hidden by the spoke part, if visual recognition in the curve direction is detected before the eyes are hidden by the spoke part, The position of the eye will be very different. For this reason, detection tends to be inaccurate, and the possibility of false notification increases.
[0114]
However, according to the present embodiment, when the foreign object is a handle, the position of the face part is switched from the process for detecting the movement state of the face part to the detection process. The frequency can be further reduced.
[0115]
Further, similarly to the second embodiment, it is possible to reduce the calculation load as much as possible and accurately determine the entry of a foreign object and detect an impossible state with high accuracy. In addition, it is possible to speed up the determination of the entry of foreign matter into the image. In addition, the configuration can be simplified and the cost can be reduced, and erroneous notification can be effectively prevented. Furthermore, as shown in FIGS. 13 and 14, the speed of the entry determination may be increased.
[0116]
In the present embodiment, the false alarm is reduced for the entry of a foreign object. However, the present invention is not limited to this, and the same applies to the exit of a foreign object from within the angle of view. Can do. For example, when the hand is covering the eyes, correct look-ahead determination and open / closed eye determination cannot be performed, but the reason why the image recognition processing becomes unstable is not only the time but also when the hand leaves the eye. Also occurs. That is, depending on the determination state of the eye tracking process, there is a possibility of misrecognizing that the hand that is leaving is eye data. For this reason, there is a possibility of issuing a false notification. The present invention can also prevent erroneous notification when the foreign object is withdrawn.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a driver state detection device according to a first embodiment of the present invention.
FIG. 2 is a detailed configuration diagram of a driver state detection device 1 according to the first embodiment of the present invention.
FIG. 3 is a data flow diagram showing an outline of the operation of the driver state determination device 20 according to the present embodiment.
FIG. 4 is an explanatory diagram of a reference area and a search area.
FIG. 5 is an explanatory diagram of reference regions arranged in an image.
FIG. 6 is a flowchart showing a detailed operation of the driver state detection apparatus 1 according to the present embodiment.
7 is a detailed flowchart of a foreign object entry determination process (step ST21) shown in FIG.
FIG. 8 is an explanatory diagram of a method for calculating a movement amount (xd, yd) in step ST31 shown in FIG.
FIGS. 9A and 9B are explanatory diagrams showing an example of an optical flow when a spoke of a handle enters the angle of view, FIG. 9A shows an example of an optical flow at time t, and FIG. 9B shows an example of optical flow at time t + 1; An example of an optical flow is shown, (c) shows an example of an optical flow at time (t + 2), and (d) shows an example of an optical flow at time (t + 3).
10 is a graph showing an optical flow obtained in the example shown in FIG. 9, where (a) shows the amount of movement in the image vertical direction, and (b) shows the amount of movement in the image horizontal direction.
FIGS. 11A and 11B are explanatory diagrams showing an example of an optical flow when a foreign object other than the spoke of the handle enters the angle of view, FIG. 11A shows an example of an image when a hand enters, and FIG. Shows an example of an optical flow when a magazine enters, (c) shows an example of an image when a magazine enters, and (d) shows an example of an optical flow when a magazine enters.
FIG. 12 is a flowchart showing an operation for determining the entry of a foreign object based on a motion pattern.
FIG. 13 is a diagram illustrating a first arrangement example of reference areas.
FIG. 14 is a diagram illustrating a second arrangement example of reference regions.
FIG. 15 is a block diagram of a driver state detection device 2 according to a second embodiment of the present invention.
FIG. 16 is a flowchart showing a detailed operation of the driver state detection apparatus 2 according to the present embodiment.
FIG. 17 is a block diagram of a driver state detection device 3 according to a third embodiment of the present invention.
FIG. 18 is a flowchart showing a detailed operation of the driver state detection device 3 according to the present embodiment.
[Explanation of symbols]
1-3 ... Driver state detection device
10 ... Shooting unit (shooting means)
21 ... Driver state determination unit (driver state determination means)
22 ... Notification section (notification means)
31 ... Image processing section (image processing means)
32 ... Notification prohibition section (notification prohibition means)
33. Arithmetic processing interruption unit (arithmetic processing interruption means)
34. Arithmetic processing switching section (arithmetic processing switching means)

Claims (4)

Photographing means for photographing the driver's face;
A driver state determination unit that detects a movement state of a facial part from a face image captured by the imaging unit and determines a driver state;
Informing means for outputting an informing signal for informing the driver based on the determination result by the driver state judging means,
The optical flow between the images is calculated based on two or more face images photographed by the photographing means, and it is detected whether or not the foreign matter entering / leaving from the image end is a handle based on the optical flow calculation result. image processing means for,
When the image processing means detects that the foreign matter entering and exiting from the image edge is a handle, the driver state determination means detects the position of the face part from the face image taken by the photographing means, A driver state detection device characterized by determining a driver state. 2. The optical flow calculation by the image processing means is performed in at least two calculation areas set in advance on the image, and these calculation areas are arranged at the lower end of the image. The driver state detection device described. Optical flow calculation by the image processing means is performed in at least two calculation areas set in advance on the image, and these calculation areas are arranged at the lower end and the left and right ends of the image. The driver state detection device according to claim 1. 2. The optical flow calculation by the image processing means is performed in at least two calculation areas set in advance on the image, and the calculation areas are arranged in an image peripheral portion. The driver state detection device described.

Images