JP4565200B2 - Manual motion segmentation method and apparatus - Google Patents

Description

【００９３】
次に、特徴運動追跡部３０３は、ｉ番目の顔運動方向ベクトルＶ＿ｆａｃｅ［ｉ］から、特徴運動コードを決定する
図９は、特徴運動追跡部３０３が特徴運動コードを決定するための、顔による特徴運動決定条件を示す図である。
図９において、顔による特徴運動決定条件は、運動コード９０１と、条件９０２とを含む。運動コード９０１には、番号「１」〜「８」がセットされ、条件９０２には、運動コード９０１にセットされた各番号と対応して、各々、顔による特徴運動決定条件がセットされる。
つまり、特徴運動追跡部３０３は、ｉ番目の顔運動方向ベクトルＶ＿ｆａｃｅ［ｉ］を、図９の条件９０２と比較して、ｉ番目の顔運動方向ベクトルＶ＿ｆａｃｅ［ｉ］と一致するような顔による特徴運動決定条件を選択する。そして、図９の運動コード９０１のうち、選択した顔による特徴運動決定条件と対応する番号を取り出すことによって、特徴運動コードを決定する。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【００９４】
［ステップＳ４０６］
セグメント位置判定部３０４は、特徴運動コードを、セグメント要素記憶部３０５に記憶されているセグメント要素データ（図６参照）と比較して、特徴運動コードが、動き特徴６０２と一致するかどうかをチェックする。動き特徴６０２には、セグメント確定のための動き特徴（動き特徴）を示すパラメータ（動き特徴パラメータ）がセットされている。
【００９５】
図１０は、動き特徴６０２にセットされる動き特徴パラメータの一例を示す図である。
図１０において、動き特徴パラメータには、動き特徴１００１、判定コード１００２、時間１００３およびセグメント位置１００４が含まれる。動き特徴１００１は、動き特徴の種別を示す。判定コード１００２は、動き特徴を判定する条件となるコード列である。時間１００３は、動き特徴を判定する条件となる時間である。セグメント位置１００４は、動き特徴におけるセグメント位置を示す。
【００９６】
判定コード１００２に含まれるコード列では、各コードは、図９の運動コード９０１（特徴運動コード）と同様の番号「１」〜「８」と、動作静止を示す番号「０」とを用いて表現され、コードとコードの間が”−”で結ばれている。
そして、ステップＳ４０５で決定された特徴運動コードと、例えば「１−０−２」が一致するか否かは、「１」の次に「０」、「２」という順番でコードが続くか否かで判定され、その順番でコードが続いた場合に、一致と判定される。
【００９７】
なお、（ ）内のコードについては、それがなくても一致するとみなされる。例えば、「７−（０）−３」は、コード「７」の次に「０」、「３」と続いた場合も、コード「７」の次に「３」と続いた場合も、条件一致とみなされる。
また、”／”を間に挟んだコードは、どちらのコードでもよいことを示す”／”は図示していない。例えば、「０／３」となっている場合、「０」でも「３」でも条件一致とみなされる。
一方、”＊”は、どのコードでもよいことを表す。
【００９８】
頷き運動を検出する場合、図６の身体特徴６０１は「顔領域」であり、動き特徴６０２として「頷き頂点」がセットされている。このとき、セグメント位置判定部３０４は、ステップＳ４０５で決定された顔の特徴運動コードが、図１０の「頷き頂点」と対応するコード列「７−（０）−３」と一致するかどうかを判定する。
手話動作セグメンテーション装置は、ｊが１であるか否かを判定し、ｊ＝１の場合、ステップＳ４０７に進む。
ｊ＞１の場合、ステップＳ４０９に進む。
【００９９】
［ステップＳ４０７］
手話動作セグメンテーション装置は、特徴運動コードが判定コード１００２の最初のコードと一致しているか否かを判定し、一致している場合、ステップＳ４０８に進む。一致していない場合、Ｓ４０１に進む。
【０１００】
［ステップＳ４０８］
セグメント位置判定部３０４は、判定コードデータを作成する。すなわち、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード番号に、特徴運動コードをセットし、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード開始フレーム番号に、ｉをセットする。
図１１は、セグメント位置判定部３０４が作成する判定コードデータの一例を示す図である。
図１１において、判定コードデータには、コード番号１１０１、コード開始フレーム番号１１０２、およびコード終了フレーム番号１１０３が含まれる。
【０１０１】
例えば、図１０の例の場合、特徴運動コードが「７」であれば、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード番号に「７」をセットし、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード開始フレーム番号にｉをセットする。
その後、ｊに２がセットされ、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
【０１０２】
［ステップＳ４０９］
特徴運動コードがｊ−１番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ−１］のコード番号と一致しているか否かが判断され、一致している場合、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
特徴運動コードがｊ−１番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ−１］のコード番号と一致していない場合、ステップＳ４１０に進む。
【０１０３】
［ステップＳ４１０］
セグメント位置判定部３０４は、ｊ−１番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ−１］のコード終了フレーム番号に、ｉ−１をセットする。その後、手話動作セグメンテーション装置は、ステップＳ４１１に進む。
【０１０４】
［ステップＳ４１１］
判定コード１００２のコード数がｊ個以上であるか否かが判定され、ｊ個以上の場合、手話動作セグメンテーション装置は、ステップＳ４１２に進む。
判定コード１００２のコード数がｊ−１個の場合、ステップＳ４１７に進む。
【０１０５】
［ステップＳ４１２］
判定コード１００２のｊ番目のコードと特徴運動コードとが一致しているか否かが判定され、一致していない場合、手話動作セグメンテーション装置は、ステップＳ４１３に進む。
一致している場合、ステップＳ４１６に進む。
【０１０６】
［ステップＳ４１３］
判定コード１００２のｊ番目のコードが（ ）付であるか否かが判定され、（）付である場合、手話動作セグメンテーション装置は、ステップＳ４１４に進む。
（ ）付でない場合、ステップＳ４１５に進む。
【０１０７】
［ステップＳ４１４］
判定コード１００２のｊ＋１番目のコードと特徴運動コードとが一致しているか否かが判定され、一致していない場合、手話動作セグメンテーション装置は、ステップＳ４１５に進む。
一致している場合、ｊに１が加えられ、その後、ステップＳ４１６に進む。
【０１０８】
［ステップＳ４１５］
ｊに１がセットされ、その後、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
【０１０９】
［ステップＳ４１６］
ｊ番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ］のコード番号に、特徴運動コードがセットされる。また、ｊ番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ］のコード開始フレーム番号に、ｉがセットされる。そして、ｊに１が加えられる。その後、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
【０１１０】
［ステップＳ４１７］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「頷き頂点」の場合、「頷き頂点」と対応するセグメント位置は、Ｙ座標最下点であり、セグメント位置判定部３０４は、Ｙ座標最下点と対応するフレーム番号を求める。
すなわち、１番目の判定コードデータＣｏｄｅ#ｄａｔａ［１］のコード開始フレーム番号から、ｊ−１番目の判定コードデータＣｏｄｅ#ｄａｔａ［ｊ−１］のコード終了フレーム番号までに該当する各フレームについて、顔領域の重心のＹ座標を相互に比較する。そして、Ｙ座標が最小となる（つまり顔領域の重心が最下点にくる）ようなフレームのフレーム番号を、動き特徴におけるセグメント位置とする。
なお、Ｙ座標最下点に該当するフレーム番号が複数出てきた場合は、最初の（番号が最も若い）フレーム番号を、セグメント位置とする。
その後、手話動作セグメンテーション装置は、ステップＳ４１８に進む。
【０１１１】
［ステップＳ４１８］
手話動作セグメンテーション装置は、セグメント位置を出力する。その後、ステップＳ４０１に戻り、上記と同様の処理を繰り返す。
以上の手順により、頷き運動の検出による手話動作セグメンテーション方法が実現できる。
【０１１２】
次に、瞬き運動の検出による手話動作セグメンテーション方法について説明する。
瞬き運動の検出による手話動作セグメンテーション方法では、前述の頷き運動の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３の処理を、次のように変更する。
［ステップＳ４０３ａ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
瞬き運動を検出する場合、身体特徴６０１に「目」が設定され、身体特徴抽出部３０２は、抽出身体特徴として目を抽出する。
【０１１３】
ここで、目の抽出方法について説明する。
最初、上記ステップＳ４０３と同様の方法で、顔領域を抽出する。次に、抽出された顔領域から、以下のようにして、目を抽出する。
図１２は、身体特徴抽出部３０２によって抽出された顔領域の一例を示す図である。
図１２において、抽出された顔領域１２０１には、眉毛による穴領域１２０２、目による穴領域１２０３、および口による穴領域１２０４が存在する（網掛けした部分が肌色領域）。
【０１１４】
図中、参照番号１２０５で示される直線は、顔上下分割ラインである。顔上下分割ライン１２０５は、抽出された顔領域１２０１を上下に分割するラインである。
最初、この顔上下分割ライン１２０５が、顔の上端から顔の下端の間の、顔上下分割比率で指定される位置にひかれる。ここで、顔上下分割比率は、パラメータであり、目の穴領域１２０３が顔上下分割ライン１２０５の上側の領域に入るように設定される。本実施形態では、例えば、顔上下分割比率を「１／２」に設定する。
【０１１５】
次に、顔上下分割ライン１２０５より上側の顔領域中の穴領域を検出する。
検出された穴領域が２つの場合、それらの穴領域を眉毛と判定し、目は閉じられていると判定する。
検出された穴領域が３つの場合、片目が閉じられていると判定し、３つの穴領域のうち下にある１つを目と判定する。
検出された穴領域が４つの場合、両目が開かれていると判定し、４つの穴領域のうち下にある２つを目と判定する。
図１２の例の場合、穴領域が４つあるため、下の２つの穴領域が、目による穴領域１２０３とわかる。
【０１１６】
次に、身体特徴抽出部３０２は、目領域情報を生成する。すなわち、抽出した目の数と、目の面積とを、ｉ番目の目領域情報ｅｙｅ［ｉ］にセットする。
図１３は、身体特徴抽出部３０２によって生成される目領域情報の一例を示す図である。
図１３において、目領域情報は、目の数１３０１、１つめの（目の）面積１３０２、および２つめの面積１３０３を含む。
身体特徴抽出部３０２は、最初、抽出した目の数を、目の数１３０１にセットする。そして、抽出した目の数に応じ、次のようにして目の面積をセットする。
【０１１７】
抽出した目の数が０の場合、１つめの面積１３０２、および２つめの面積１３０３にそれぞれ０をセットする。
抽出した目の数が１の場合、目（目による穴領域１２０３）の面積を計算して、１つめの面積１３０２にセットする。２つめの面積１３０３には、０をセットする。
抽出した目の数が２の場合、２つの目の面積を各々計算して、１つめの面積１３０２には、左目（目による穴領域１２０３の左側の方）の面積をセットし、２つめの面積１３０３には、右目の面積をセットする。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１１８】
瞬き運動の検出による手話動作セグメンテーション方法では、上記のステップＳ４０５の処理を、次のように変更する。
［ステップＳ４０５ａ］
特徴運動追跡部３０３は、＜式２＞を用いて、ｉ番目の目領域情報ｅｙｅ［ｉ］と、ｉ−１番目の目領域情報ｅｙｅ［ｉ−１］とから、目の特徴運動コードを求める。また、ｉ番目の目領域情報ｅｙｅ［ｉ］の１つめの面積ｓ１＿ｅｙｅ［ｉ］と、ｉ−１番目の目領域情報ｅｙｅ［ｉ−１］の１つめの面積ｓ１＿ｅｙｅ［ｉ−１］とから、ｉ番目の目の１つめの目の目領域面積変化ｄ１＿ｅｙｅ［ｉ］を求める。また、ｉ番目の目領域情報ｅｙｅ［ｉ］の２つめの面積ｓ２＿ｅｙｅ［ｉ］と、ｉ−１番目の目領域情報ｅｙｅ［ｉ−１］の２つめの面積ｓ２＿ｅｙｅ［ｉ−１］とから、ｉ番目の目の２つめ目領域面積変化ｄ２＿ｅｙｅ［ｉ］を求める。
【数２】

【０１１９】
図１４は、特徴運動追跡部３０３が特徴運動コードを決定するための、目による特徴運動決定条件を示す図である。
図１４において、目による特徴運動決定条件は、運動コード１４０１と、条件１４０２とを含む。運動コード１４０１には、番号「０」〜「６」がセットされ、条件１４０２には、運動コード１４０１にセットされた各番号と対応して、各々、目による特徴運動決定条件がセットされる。
条件１４０２中のαは、目が閉じているかどうかを判定する目面積閾値であり、例えば「１」がセットされる。βは、目の大きさが変化しているかどうかを判定する目大きさ変化閾値であり、例えば「５」がセットされる。
【０１２０】
つまり、特徴運動追跡部３０３は、ｉ番目の目領域情報ｅｙｅ［ｉ］、ｉ番目の１つめの目領域面積変化ｄ１＿ｅｙｅ［ｉ］、および２つめの目領域面積変化ｄ２＿ｅｙｅ［ｉ］を、図１４の条件１４０２と比較して、ｉ番目の目領域情報ｅｙｅ［ｉ］、ｉ番目の１つめの目領域面積変化ｄ１＿ｅｙｅ［ｉ］、および２つめの目領域面積変化ｄ２＿ｅｙｅ［ｉ］と一致するような目による特徴運動決定条件を選択する。そして、図１４の運動コード１４０１のうち、選択した目による特徴運動決定条件と対応する番号を取り出すことによって、特徴運動コードを決定する。
例えば、目が両方閉じている場合、ｓ１＿ｅｙｅ［ｉ］≦α、ｓ２＿ｅｙｅ［ｉ］≦αとなり、このとき、特徴運動コードは、０となる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１２１】
瞬き運動の検出による手話動作セグメンテーション方法では、上記ステップＳ４１７の処理を、次のように変更する。
［ステップＳ４１７ａ］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「瞬き」の場合、「瞬き」と対応するセグメント位置は、目領域消失点であり、セグメント位置判定部３０４は、目領域消失点と対応するフレーム番号を求める。
すなわち、２番目の判定コードデータＣｏｄｅ#ｄａｔａ［２］のコード開始フレーム番号を、セグメント位置とする。
その後、手話動作セグメンテーション装置は、ステップＳ４１８に進む。
以上の手順により、瞬き運動の検出による手話動作セグメンテーション方法が実現できる。
【０１２２】
次に、口形変化（閉口）の検出による手話動作セグメンテーション方法について説明する。
この場合、前述の頷き運動の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３の処理を、次のように変更する。
［ステップＳ４０３ｂ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
口形変化（閉口）を検出する場合、身体特徴６０１に「口」が設定され、身体特徴抽出部３０２は、抽出身体特徴として口を抽出する。
【０１２３】
ここで、口の抽出方法について説明する。
最初、上記ステップＳ４０３と同様の方法で、顔領域を抽出する。次に、抽出された顔領域から、以下のようにして、口を抽出する。
図１２において、最初、ステップＳ４０３と同様にして、顔上下分割ライン１２０５がひかれる。次に、顔上下分割ライン１２０５より下側の顔領域中の穴領域を検出する。
【０１２４】
検出された穴領域が２つ以上の場合、それらの穴領域のうち、顔領域の下端との間の距離が口高さ条件に最も近いものを口とする。ここで、口高さ条件は、パラメータであり、標準的な人の口の位置を、顔領域の下端との間の距離によって表したものである。本実施形態では、口高さ条件を「１０」に設定する。
検出された穴領域が１つの場合、その穴領域を口とする。
穴領域が検出されない場合、口を閉じているとみなす。
図１２の例の場合、顔上下分割ライン１２０５より下側には、穴領域が１つだけなので、その穴領域が、口による穴領域１２０４とわかる。
【０１２５】
次に、身体特徴抽出部３０２は、口領域情報を生成する。すなわち、抽出した口の面積と、口のＹ軸方向最大長とを、ｉ番目の口領域情報ｍｏｕｔｈ［ｉ］にセットする。
図１５は、身体特徴抽出部３０２が生成する口領域情報の一例を示す図である。
図１５において、口領域情報は、口の面積１５０１、および口のＹ軸方向最大長１５０２を含む。
身体特徴抽出部３０２は、抽出した口の面積を計算して、口の面積１５０１にセットし、さらに、口のＹ軸方向の長さの最大長を計算して、口のＹ軸方向最大長１５０２にセットする。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１２６】
口形変化の検出による手話動作セグメンテーション方法では、上記のステップＳ４０５の処理を、次のように変更する。
［ステップＳ４０５ｂ］
特徴運動追跡部３０３は、＜式３＞を用いて、ｉ番目の口領域情報ｍｏｕｔｈ［ｉ］と、ｉ−１番目の口領域情報ｍｏｕｔｈ［ｉ−１］とから、口の特徴運動コードを求める。また、ｉ番目の口領域情報ｍｏｕｔｈ［ｉ］の面積ｓ＿ｍｏｕｔｈ［ｉ］と、ｉ−１番目の口領域情報ｍｏｕｔｈ［ｉ−１］の面積ｓ＿ｍｏｕｔｈ［ｉ−１］とから、ｉ番目の口の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］を求める。
【数３】

【０１２７】
また、＜式４＞を用いて、ｉ番目の口領域情報ｍｏｕｔｈ［ｉ］のＹ軸方向最大長ｈ＿ｍｏｕｔｈ［ｉ］と、ｉ−１番目の口領域情報ｍｏｕｔｈ［ｉ−１］のＹ軸方向最大長ｈ＿ｍｏｕｔｈ［ｉ−１］とから、ｉ番目の口のＹ軸方向変化ｙ＿ｍｏｕｔｈ［ｉ］を求める。
【数４】

【０１２８】
図１６は、特徴運動追跡部３０３が特徴運動コードを決定するための、口による特徴運動決定条件を示す図である。
図１６において、口による特徴運動決定条件は、運動コード１６０１と、条件１６０２とを含む。運動コード１６０１には、番号「０」および「１」がセットされ、条件１６０２には、運動コード１６０１にセットされた各番号と対応して、各々、口による特徴運動決定条件がセットされる。
条件１６０２中のγは、口形の変化の有無を判定するための口面積変化閾値であり、本実施形態では、例えば「５」がセットされる。λは、口Ｙ軸方向変化閾値であり、例えば「３」がセットされる。
【０１２９】
つまり、特徴運動追跡部３０３は、ｉ番目の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］およびＹ軸方向最大長ｈ＿ｍｏｕｔｈ［ｉ］を、図１６の条件１６０２と比較して、ｉ番目の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］およびＹ軸方向最大長ｈ＿ｍｏｕｔｈ［ｉ］と一致するような口による特徴運動決定条件を選択する。そして、図１６の運動コード１６０１のうち、選択した口による特徴運動決定条件と対応する番号を取り出すことによって、特徴運動コードを決定する。
例えば、口が閉じている場合は、ｓ＿ｍｏｕｔｈ［ｉ］≦γとなり、このとき、特徴運動コードは、「０」となる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１３０】
口形変化の検出による手話動作セグメンテーション方法では、上記のステップＳ４１７の処理を、次のように変更する。
［ステップＳ４１７ｂ］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「口形変化」の場合、「口形変化」と対応するセグメント位置は、変化始点終点であり、セグメント位置判定部３０４は、変化始点終点と対応するフレーム番号を求める。
すなわち、２番目の判定コードデータＣｏｄｅ#ｄａｔａ［２］のコード開始フレーム番号と、２番目の判定コードデータＣｏｄｅ#ｄａｔａ［２］のコード終了フレーム番号とを、セグメント位置として出力する。
その後、手話動作セグメンテーション装置は、ステップＳ４１８に進む。
以上の手順により、口形変化の検出による手話動作セグメンテーション方法が実現できる。
【０１３１】
次に、手および胴体の静止動作の検出による手話動作セグメンテーション方法について説明する。
この場合、前述の頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３の処理を、次のように変更する。
［ステップＳ４０３ｃ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
手および胴体の静止動作を検出する場合、身体特徴６０１に「手領域、胴体」が設定され、身体特徴抽出部３０２は、抽出身体特徴として手領域および胴体を抽出する。
【０１３２】
ここで、手領域および胴体の抽出方法について説明する。
最初、身体特徴抽出部３０２は、上記ステップＳ４０３と同様の方法で、手領域を抽出する。すなわち、入力画像から肌色領域を抽出して、抽出された肌色領域のうち頭部領域と重複しない部分を取り出し、それを手領域とする。
図７の場合、肌色領域のうち頭部領域と重複しない領域、すなわち手の肌色領域７０３が抽出される。
胴体については、ステップＳ４０２で検出された人物領域を、そのまま胴体とする。
【０１３３】
次に、身体特徴抽出部３０２は、手領域情報を生成する。すなわち、抽出した手領域に関し、その重心、面積、Ｘ軸方向最大長およびＹ軸方向最大長を、ｉ番目の手領域情報ｈａｎｄ［ｉ］にセットする。次いで、抽出した胴体に関し、その重心、面積、Ｘ軸方向最大長およびＹ軸方向最大長を、ｉ番目の胴体情報ｂｏｄｙ［ｉ］にセットする。
図１７は、身体特徴抽出部３０２が生成する手領域情報の一例を示す図である。
図１７において、手領域情報には、手の数１７０１、１つめの手の重心座標１７０２、１つめの手の面積１７０３、２つめの手の重心座標１７０４、および２つめの手の面積１７０５が含まれる。
身体特徴抽出部３０２は、最初、抽出した手の数を、手の数１７０１にセットする。そして、抽出した手の数に応じ、次のようにして手の重心座標および手の面積をセットする。
【０１３４】
抽出した手の数１７０１が０の場合、１つめの手の重心座標１７０２、および２つめの手の重心座標１７０４に、それぞれ（０，０）を設定し、また、１つめの手の面積１７０３、および２つめの手の面積１７０５に、それぞれ０を設定する。
抽出した手の数１７０１が「１」の場合、手領域の重心座標および面積を計算して、１つめ手の重心座標１７０２、および１つめの手の面積１７０３にセットする。また、２つめの手の重心座標１７０４に（０、０）をセットし、２つめの手の面積１７０５に０をセットする。
抽出した手の数１７０１が「２」の場合、２つの手領域のうち左側の領域の重心座標および面積を計算して、１つめ手の重心座標１７０２、および１つめの手の面積１７０３にセットする。また、２つの手領域のうち右側の領域の重心座標および面積を計算して、２つめ手の重心座標１７０４、および２つめの手の面積１７０５にセットする。
胴体情報ｂｏｄｙ［ｉ］は、顔領域情報ｆａｃｅ［ｉ］と同様、図８の構成で実現できる。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１３５】
手および胴体の静止動作の検出による手話動作セグメンテーション方法では、上記のステップＳ４０５の処理を、次のように変更する。
［ステップＳ４０５ｃ］
特徴運動追跡部３０３は、＜式５＞を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］と、ｉ番目の胴体情報ｂｏｄｙ［ｉ］と、ｉ−１番目の胴体情報ｂｏｄｙ［ｉ−１］とから、手領域および胴体の特徴運動コードを求める。また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］を求める。また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］を求める。
【数５】

【０１３６】
また、＜式６＞を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の面積ｓ１＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の１つめの手の面積ｓ１＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の１つめの手の面積変化量ｄ１＿ｈａｎｄ［ｉ］を求める。また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の面積ｓ２＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の２つめの手の面積ｓ２＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の２つめの手の面積変化量ｄ２＿ｈａｎｄ［ｉ］を求める。
【数６】

【０１３７】
また、＜式７＞を用いて、ｉ番目の胴体情報ｂｏｄｙ［ｉ］の重心ｇ＿ｂｏｄｙ［ｉ］と、ｉ−１番目の胴体情報ｂｏｄｙ［ｉ−１］の重心ｇ＿ｂｏｄｙ［ｉ−１］とから、ｉ番目の胴体の移動量ｍ＿ｂｏｄｙ［ｉ］を求める。
【数７】

【０１３８】
図１８は、胴体および手領域による特徴運動決定条件を示す図である。
図１８において、胴体および手領域による特徴運動決定条件は、運動コード１８０１と、条件１８０２とを含む。運動コード１８０１には、番号「０」および「１」がセットされ、条件１８０２には、運動コード１８０１にセットされた各番号と対応して、各々、胴体および手領域による特徴運動決定条件がセットされる。
条件１８０２中のχは、手領域の停止判定閾値であり、本実施形態では、例えば「５」がセットされる。δは、手領域の形状変化判定閾値であり、例えば「１０」がセットされる。εは、胴体の停止判定閾値であり、例えば「５」がセットされる。
【０１３９】
つまり、特徴運動追跡部３０３は、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］、ｉ番目の手領域の１つめの手の面積変化量ｄ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の面積変化量ｄ２＿ｈａｎｄ［ｉ］、およびｉ番目の胴体の移動量ｍ＿ｂｏｄｙ［ｉ］を、図１８の条件１８０２と比較して、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］、ｉ番目の手領域の１つめの手の面積変化量ｄ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の面積変化量ｄ２＿ｈａｎｄ［ｉ］、およびｉ番目の胴体の移動量ｍ＿ｂｏｄｙ［ｉ］と一致するような手領域および胴体による特徴運動決定条件を選択する。そして、図１８の運動コード１８０１のうち、選択した手領域および胴体による特徴運動決定条件と対応する番号を取り出すことによって、特徴運動コードを決定する。
例えば、手領域が左右に移動中の場合、ｉ番目の手領域の移動量ｍ＿ｈａｎｄ［ｉ］＞χとなり、このとき、特徴運動コードは、「１」となる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１４０】
手および胴体の静止動作の検出による手話動作セグメンテーション方法では、上記のステップＳ４１７の処理を、次のように変更する。
［ステップＳ４１７ｃ］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「動作静止」の場合、「動作静止」と対応するセグメント位置は、動作静止始点であり、セグメント位置判定部３０４は、動作静止始点と対応するフレーム番号を求める。
または、静止区間の中間点と対応するフレーム番号を求めてもよい。この場合、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード開始フレーム番号と、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード終了フレーム番号とを求めて、それらの中間値を計算すればよい。
その後、手話動作セグメンテーション装置は、ステップＳ４１８に進む。
以上の手順により、手および胴体の静止動作の検出による手話動作セグメンテーション方法が実現できる。
【０１４１】
次に、手と顔との接触動作の検出による手話動作セグメンテーション方法について説明する。
この場合、前述の頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３の処理を、次のように変更する。
［ステップＳ４０３ｄ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
手と顔との接触動作を検出する場合、身体特徴６０１に「顔領域、手領域」が設定され、抽出身体特徴として顔領域および手領域を抽出する。
【０１４２】
ここで、顔領域および手領域の抽出方法について説明する。
最初、上記ステップＳ４０３と同様の方法で、顔領域を抽出し、また、上記ステップＳ４０３ｃと同様の方法で、手領域を抽出する。
次に、抽出した顔領域に関し、その重心、面積、Ｘ軸方向最大長およびＹ軸方向最大長を、ｉ番目の顔領域情報ｆａｃｅ［ｉ］にセットする。また、抽出した手領域に関し、その重心、面積、Ｘ軸方向最大長およびＹ軸方向最大長を、ｉ番目の手領域情報ｈａｎｄ［ｉ］にセットする。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１４３】
手と顔との接触動作の検出による手話動作セグメンテーション方法では、上記のステップＳ４０５の処理を、次のように変更する。
［ステップＳ４０５ｄ］
特徴運動追跡部３０３は、＜式８＞を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］と、ｉ番目の顔領域情報ｆａｃｅ［ｉ］とから、手領域および顔領域の特徴運動コードを求める。また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］と、ｉ番目の顔領域情報ｆａｃｅ［ｉ］の重心ｇ＿ｆａｃｅ［ｉ］とから、ｉ番目の手領域の１つめの手と顔との距離ｌ１＿ｆｈ［ｉ］を求める。また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］と、ｉ番目の顔領域情報ｆａｃｅ［ｉ］の重心ｇ＿ｆａｃｅ［ｉ−１］とから、ｉ番目の手領域の２つめの手と顔との距離ｌ２＿ｆｈ［ｉ］を求める。
【数８】

【０１４４】
ただし、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の面積ｓ１＿ｈａｎｄ［ｉ］が０の場合は、ｌ１＿ｆｈ［ｉ−１］≦Φであれば、ｌ１＿ｆｈ［ｉ］＝０とする。ｌ１＿ｆｈ［ｉ−１］＞Φであれば、ｌ１＿ｆｈ［ｉ］＝１０００とする。
同様に、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の面積ｓ２＿ｈａｎｄ［ｉ］が０の場合、ｌ２＿ｆｈ［ｉ−１］≦Φであれば、ｌ２＿ｆｈ［ｉ］＝０とする。ｌ２＿ｆｈ［ｉ−１］＞Φであれば、ｌ２＿ｆｈ［ｉ］＝１０００とする。ここで、Φは、顔と手との距離閾値であり、本実施形態では、例えば「２０」がセットされる
【０１４５】
図１９は、特徴運動追跡部３０３が特徴運動コードを決定するための、手と顔との接触動作による特徴運動決定条件を示す図である。
図１９において、手と顔との接触動作による特徴運動決定条件は、運動コード１９０１と、条件１９０２とを含む。運動コード１９０１には、番号「０」および「１」がセットされ、条件１９０２には、運動コード１９０１にセットされた各番号と対応して、各々、手と顔との接触動作による特徴運動決定条件がセットされる。
条件１９０２中のωは、手領域と顔領域の接触閾値であり、本実施形態では、例えば「５」がセットされる。
【０１４６】
つまり、特徴運動追跡部３０３は、ｉ番目の手領域の１つめの手と顔との距離ｌ１＿ｆｈ［ｉ］、およびｉ番目の手領域の２つめの手と顔との距離ｌ２＿ｆｈ［ｉ］を、図１９の条件１９０２と比較して、ｉ番目の手領域の１つめの手と顔との距離ｌ１＿ｆｈ［ｉ］、およびｉ番目の手領域の２つめの手と顔との距離ｌ２＿ｆｈ［ｉ］と一致するような特徴運動決定条件を選択する。そして、図１９の運動コード１９０１のうち、選択した特徴運動決定条件と対応する番号を取り出すことによって、特徴運動コードを決定する。
例えば、右手が顔と重なっている場合、ｉ番目の手領域の一つめの手と顔の距離ｌ１＿ｆｈ［ｉ］＝０となり、このとき、特徴運動コードは、「０」となる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１４７】
手と顔との接触動作の検出による手話動作セグメンテーション方法では、上記のステップＳ４１７の処理を、次のように変更する。
［ステップＳ４１７ｄ］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「手と顔との接触動作」の場合、「手顔接触」と対応するセグメント位置は、「接触始点終点」であり、セグメント位置判定部３０４は、手顔接触区間の始点および終点と対応するフレーム番号を求める。
すなわち、１番目の判定コードデータＣｏｄｅ#ｄａｔａ［１］のコード開始フレーム番号と、１番目の判定コードデータＣｏｄｅ#ｄａｔａ［１］のコード終了フレーム番号とを、セグメント位置とする。
その後、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
以上の手順により、手と顔との接触動作の検出による手話動作セグメンテーション方法が実現できる。
【０１４８】
次に、手の有効性変化の検出による手話動作セグメンテーション方法について説明する。
この場合、前述の頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３の処理を、次のように変更する。
［ステップＳ４０３ｅ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
手の有効性変化を検出する場合、身体特徴６０１に「手領域」が設定され、身体特徴抽出部３０２は、抽出身体特徴として手領域を抽出する。
【０１４９】
なお、手領域は、上記ステップＳ４０３ｃと同様の方法で抽出する。
次に、身体特徴抽出部３０２は、抽出した手領域に関し、その重心、面積、Ｘ軸方向最大長およびＹ軸方向最大長を、ｉ番目の手領域情報ｈａｎｄ［ｉ］にセットする。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１５０】
手の有効性変化の検出による手話動作セグメンテーション方法では、上記のステップＳ４０５の処理を、次のように変更する。
［ステップＳ４０５ｅ］
特徴運動追跡部３０３は、前述の＜式５＞を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］から、手の有効性および動きによる特徴運動コードを求める。
また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］から、１つめの手が、上記ステップＳ４０２で空間分割して得られた複数の領域（図５参照）のうちどの領域に属するかを判定して、その領域コードを求め、一つめの手の手領域空間コードｓｐ１＿ｈａｎｄ［ｉ］にセットする。なお、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の面積ｓ１＿ｈａｎｄ［ｉ］が０の場合、１つめの手の手領域空間コードｓｐ１＿ｈａｎｄ［ｉ］に「６」をセットする。
【０１５１】
また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］から、領域コードを求めて、２つめの手の手領域空間コードｓｐ２＿ｈａｎｄ［ｉ］にセットする。なお、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の面積ｓ２＿ｈａｎｄ［ｉ］が０の場合、２つめの手の手領域空間コードｓｐ２＿ｈａｎｄ［ｉ］に「６」をセットする。
【０１５２】
また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］を求める。
また、ｉ番目の手領域情報ｈａｎｄ［ｉ］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ−１］とから、ｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］を求める。
【０１５３】
図２０は、特徴運動追跡部３０３が特徴運動コードを決定するための、手の有効性変化による特徴運動決定条件を示す図である。
図２０において、手の有効性変化による特徴運動決定条件は、運動コード２００１と、条件２００２とを含む。運動コード２００１には、番号「０」〜「５」がセットされ、条件２００２には、運動コード２００１にセットされた各番号と対応して、各々、手と顔との接触動作による特徴運動決定条件がセットされる。
条件２００２中のχは、手領域の停止判定閾値であり、本実施形態では、例えば「５」がセットされる。
【０１５４】
つまり、特徴運動追跡部３０３は、ｉ番目の手領域の１つめの手の手領域空間コードｓｐ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の手領域空間コードｓｐ２＿ｈａｎｄ［ｉ］、およびｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］を、図２０の条件２００２と比較して、ｉ番目の手領域の１つめの手の手領域空間コードｓｐ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］、ｉ番目の手領域の２つめの手の手領域空間コードｓｐ２＿ｈａｎｄ［ｉ］、およびｉ番目の手領域の２つめの手の移動量ｍ２＿ｈａｎｄ［ｉ］と一致するような手と顔との接触動作による特徴運動決定条件を選択する。
例えば、右手が移動中で、かつ左手が入力画像５０１（図５参照）の一番下の位置まで降りた状態の場合、ｉ番目の手領域の１つめの手の移動量ｍ１＿ｈａｎｄ［ｉ］＞χとなり、ｉ番目の手領域の２つめの手の手領域空間コードｓｐ２＿ｈａｎｄ［ｉ］＝７となり、このとき、特徴運動コードは、「２」となる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１５５】
手の有効性変化の検出による手話動作セグメンテーション方法では、上記のステップＳ４１７の処理を、次のように変更する。
［ステップＳ４１７ｅ］
セグメント位置判定部３０４は、動き特徴１００１およびセグメント位置１００４（図１０参照）に基づき、動き特徴におけるセグメント位置を求める。
動き特徴が「手の有効性変化点」の場合、「手有効性変化点」と対応するセグメント位置は、「コード変化点」であり、セグメント位置判定部３０４は、コード変化点と対応するフレーム番号を求める。
すなわち、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード開始フレーム番号と、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード終了フレーム番号とを、セグメント位置とする。
その後、ステップＳ４１８に進む。
以上の手順により、手の有効性変化の検出による手話動作セグメンテーション方法が実現できる。
【０１５６】
次に、上記の各動作の検出の組み合わせによる手話動作セグメンテーション方法について説明する。
この場合、前述の頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、ステップＳ４０３を次のように変更する。
［ステップＳ４０３ｆ］
身体特徴抽出部３０２は、入力画像から、セグメント要素記憶部３０５に記憶されている身体特徴６０１（図６参照）と対応する画像を抽出する。
上記の各動作を検出する場合、身体特徴６０１に「顔領域」、「目」、「口」、「手領域、胴体」、「手領域、顔領域」および「手領域」が設定され、身体特徴抽出部３０２は、抽出身体特徴として、顔領域、目、口、手領域および胴体を抽出する。
なお、顔領域は、上記ステップＳ４０３と同様の方法で抽出する。目は、上記ステップＳ４０３ａと同様の方法で抽出する。口は、上記ステップＳ４０３ｂと同様の方法で抽出する。手領域および胴体は、上記ステップＳ４０３ｃと同様の方法で抽出する。
【０１５７】
次に、身体特徴抽出部３０２は、抽出した顔領域、目、口、手領域および胴体に関する情報を、各々、顔領域情報ｆａｃｅ［ｉ］、目領域情報ｅｙｅ［ｉ］、口領域情報ｍｏｕｔｈ［ｉ］、手領域情報ｈａｎｄ［ｉ］、および胴体情報ｂｏｄｙ［ｉ］にセットする。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１５８】
そして、手話動作セグメンテーション装置は、上記ステップＳ４０５からステップＳ４１７までを処理し、続いて、上記ステップＳ４０５ｂからステップＳ４１７ｂまでを処理する。以降、上記ステップＳ４０５ｃからステップＳ４１７ｃまでの処理、上記ステップＳ４０５ｄからステップＳ４１７ｄまでの処理、および上記ステップＳ４０５ｅからステップＳ４１７ｄまでの処理を、順番に処理する。
これにより、上記の各動作の検出の組み合わせによる手話動作セグメンテーション方法が実現できる。
【０１５９】
次に、上記各方法において、検出した動作の継続時間を考慮してセグメンテーションを行うような手話動作セグメンテーション方法について説明する。
図２１は、頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、検出した動作の継続時間を考慮してセグメンテーションを行う場合の処理の流れを示すフローチャートである。
図２１の方法は、図４の方法において、ステップＳ４１１を次のように変更し、さらにステップＳ２１０１を追加したものである。
［ステップＳ４１１ａ］
判定コード１００２のコード数がｊ個以上か否かが判定され、ｊ個以上の場合、ステップＳ４１２に進む。
ｊ−１個の場合、ステップＳ２１０１に進む。
【０１６０】
［ステップＳ２１０１］
最初、１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［１］のコード開始フレーム番号と対応するフレームから、ｊ−１番目の判定コードデータＣｏｄｅ＿ｄａｔａ［ｊ−１］のコード終了フレーム番号と対応するフレームまでのフレーム数が、特徴継続時間にセットされる。
次に、動き特徴パラメータ（図１０参照）の時間１００３に、値が設定されているか否かが判定され、さらに、特徴継続時間が、時間１００３に設定された値より小さいか否かが判定される。
そして、時間１００３に値が設定されており、かつ特徴継続時間が時間１００３に設定された値より小さい場合、ステップＳ４１５に進む。
時間１００３に何も設定されていないか、特徴継続時間が時間１００３に設定された値以上の場合、ステップＳ４１７に進む。
以上の手順により、検出した動作の継続時間を考慮してセグメンテーションを行うような手話動作セグメンテーション方法が実現できる。
【０１６１】
次に、セグメント要素に加えて、非セグメント要素を検出してセグメンテーションを行うような手話動作セグメンテーション方法について説明する。
（第３の実施形態）
図２２は、本発明の第３の実施形態に係る手話動作セグメンテーション装置の構成を示すブロック図である。
図２２の装置は、図３の装置において、非セグメント要素記憶部２２０１をさらに備えている。非セグメント要素記憶部２２０１には、セグメントを行わない条件となる非セグメント要素が、予め記憶されている。他の構成要素は、図３のものと同様のものである。
すなわち、図２２の装置は、セグメント要素に加えて、非セグメント要素をも検出し、それらに基づいて手話動作のセグメンテーションを行うような手話動作セグメンテーション方法を実行する。
【０１６２】
以下、上記のように構成された手話動作セグメンテーション装置の処理手順について説明する。
最初、非セグメント要素として、両手の接近を検出する場合を説明する。
図２３および２４は、図２２の手話動作セグメンテーション装置の処理の流れの一例を示すフローチャートである。
図２３および２４の方法は、図２１の方法において、ステップＳ４０３の後にステップＳ２４０１を追加し、また、ステップＳ４０５の後にステップＳ２４０２〜Ｓ２４０５を追加し、さらに、ステップＳ４１８をステップＳ４１８ａのように変更したものである。
これら各ステップ（Ｓ２４０１〜Ｓ２４０５，Ｓ４１８ａ）では、以下のような処理が行われる。
【０１６３】
［ステップＳ２４０１］
身体特徴抽出部３０２は、入力画像から、非セグメント要素記憶部２２０１に記憶されている身体特徴と対応する画像を抽出する。
図２５は、非セグメント要素記憶部２２０１に記憶されている非セグメント要素データの一例を示す図である。
図２５において、非セグメント要素データは、身体特徴２５０１と、非セグメント動き特徴２５０２とを含む。
例えば、両手の接近を検出する場合、「手領域」が身体特徴２５０１にセットされている。
身体特徴抽出部３０２は、手領域を非セグメント身体特徴として抽出する。手領域は、上記ステップＳ４０３ｃの手順で抽出できる。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１６４】
［ステップＳ２４０２］
次の手順で、非セグメント特徴運動コードが決定される。
ｉ番目の手領域情報ｈａｎｄ［ｉ］の手の数が２の場合、特徴運動追跡部３０３は、＜式９＞を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］と、２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］とから、ｉ番目の手領域の手間距離ｌ＿ｈａｎｄ［ｉ］を求める。
【数９】

【０１６５】
次に、＜式１０＞を用いて、ｉ番目の手領域の手間距離ｌ＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域の手間距離ｌ＿ｈａｎｄ［ｉ−１］とから、手間距離変化量ｄｌ＿ｈａｎｄ［ｉ］を求める。
【数１０】

【０１６６】
ｉ番目の手領域情報ｈａｎｄ［ｉ］の手の数が２でないか、または、ｉ番目の手領域情報ｈａｎｄ［ｉ］の手の数と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の手の数とが異なる場合、特徴運動追跡部３０３は、手間距離変化量ｄｌ＿ｈａｎｄ［ｉ］に、負でない値、例えば１０００をセットする。
手間距離変化量ｄｌ＿ｈａｎｄ［ｉ］≦−θの場合、非セグメント特徴運動コードは、「１」となる。手間距離変化量ｄｌ＿ｈａｎｄ［ｉ］＞−θの場合、非セグメント特徴運動コードは、「０」となる。ここで、θは、手間距離変化閾値であり、本実施形態では、例えば「５」がセットされる。
非セグメントコード番号ｋに何もセットされていない場合、非セグメントコード番号ｋに「１」がセットされ、非セグメント特徴フレーム数に「０」がセットされる。
ここで非セグメントコード番号ｋは、非セグメント特徴運動コードを構成するコードの個数を示し、非セグメント特徴フレーム数は、非セグメント動き特徴の検出期間に相当するフレーム数、つまり検出開始フレームから検出終了フレームまでのフレーム数を示す。
その後、手話動作セグメンテーション装置は、ステップ３００３に進む。
【０１６７】
［ステップＳ２４０３］
セグメント位置判定部３０４は、非セグメント特徴運動コードを、非セグメント要素記憶部２２０１に記憶されている非セグメント要素データ（図２５参照）と比較して、非セグメント特徴運動コードが、非セグメント動き特徴２５０２と一致するかどうかをチェックする。非セグメント動き特徴２５０２には、非セグメント確定のための動き特徴（非セグメント動き特徴）を示すパラメータ（非セグメント動き特徴パラメータ）がセットされている。
【０１６８】
図２６は、非セグメント動き特徴２５０２にセットされる非セグメント動き特徴パラメータの一例を示す図である。
図２６において、非セグメント動き特徴パラメータは、非セグメント動き特徴２６０１、判定コード２６０２、および時間２６０３を含む。非セグメント動き特徴２６０１は、非セグメント動き特徴の種別を示す。判定コード２６０２は、非セグメント動き特徴を判定する条件となるコード列である。時間２６０３は、非セグメント動き特徴を判定する条件となる時間である。
判定コード２６０２は、図１０の動き特徴パラメータ中の判定コード１００２と同様の方法で記述される。時間２６０３には、非セグメント動き特徴２６０１の最低継続時間が設定される。
【０１６９】
判定コード２６０２と、ステップＳ２４０２で決定された非セグメント特徴運動コードのｋ番目のコード、つまり非セグメント特徴運動コードを構成する最後のコードとが異なる場合、手話動作セグメンテーション装置は、ステップＳ２４０４に進む。
同じ場合、ステップＳ２４０５に進む。
【０１７０】
［ステップＳ２４０４］
非セグメント特徴フレーム数に「０」がセットされ、非セグメントコード番号ｋに「１」がセットされる。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１７１】
［ステップＳ２４０５］
非セグメント特徴フレーム数に「１」が加算される。
ｋ＞２ので非セグメント確定コード列条件のｋ−１番目のコードと、非セグメント特徴運動コードとが異なる場合、ｋに「１」が加算される。
その後、手話動作セグメンテーション装置は、ステップＳ４０６に進む。
【０１７２】
［ステップＳ４１８ａ］
非セグメント動き特徴パラメータ（図２６参照）中の時間２６０３に値が設定されていない場合、非セグメント時間最小値に０がセットされる。
時間２６０３に値が設定されている場合、非セグメント時間最小値に、時間２６０３の値をセットする。
非セグメント特徴フレーム数が、非セグメント時間最小値に相当するフレーム数より小さい場合は、ステップＳ４１７でセットしたセグメント位置を出力する。
その後、手話動作セグメンテーション装置は、ステップＳ４０１に進む。
以上の手順により、セグメント要素に加えて、非セグメント要素（両手接近）をも検出し、それらに基づいてセグメンテーションを行うような手話動作セグメンテーション方法が実現できる。
【０１７３】
次に、非セグメント要素として口形変化を検出する場合を説明する。
この場合、上記のステップＳ２４０１を次のように変更する。
［ステップＳ２４０１ａ］
身体特徴抽出部３０２は、入力画像から、非セグメント要素記憶部２２０１に記憶されている身体特徴と対応する画像を抽出する。
図２５において、口形変化を検出する場合、「口」が身体特徴２５０１にセットされている。
身体特徴抽出部３０２は、口を非セグメント身体特徴として抽出する。口は、上記ステップＳ４０３ｂの手順で抽出できる。
その後、手話動作セグメンテーション装置は、ステップＳ４０４に進む。
【０１７４】
また、上記のステップＳ２４０２を次のように変更する。
［ステップＳ２４０２ａ］
次の手順で、非セグメント特徴運動コードが決定される。
特徴運動追跡部３０３は、最初、上記のステップＳ４０５ｂと同様の手順で、ｉ番目の口領域情報の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］、およびｉ番目の口領域情報のＹ軸方向変化ｙ＿ｍｏｕｔｈ［ｉ］を求める。
次に、ｉ番目の口領域情報の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］、およびｉ番目の口領域情報のＹ軸方向変化ｙ＿ｍｏｕｔｈ［ｉ］を、図１６の条件１６０２と比較して、ｉ番目の口領域情報の口領域面積変化ｄ＿ｍｏｕｔｈ［ｉ］、およびｉ番目の口領域情報のＹ軸方向変化ｙ＿ｍｏｕｔｈ［ｉ］と一致するような口による特徴運動決定条件を選択する。そして、図１６の運動コード１６０１のうち、選択した口による特徴運動決定条件と対応する番号を取り出すことによって、非セグメント特徴運動コードを決定する。
例えば、口を動かしていない場合、口の面積およびＹ軸方向最大長が変化せず、このとき、非セグメント特徴運動コードは、「０」となる。
非セグメントコード番号ｋに何もセットされていない場合、非セグメントコード番号ｋに「１」がセットされ、非セグメント特徴フレーム数に「０」がセットされる。
その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
以上の手順により、セグメント要素に加えて、非セグメント要素（口形変化）をも検出して、それらに基づいてセグメンテーションを行うような手話動作セグメンテーション方法が実現できる。
【０１７５】
次に、非セグメント要素として、右手と左手との動作の対称性を検出する場合を説明する。
この場合、上記のステップＳ２４０２を次のように変更する。
［ステップＳ２４０２ｂ］
次の手順で、非セグメント特徴運動コードが決定される。
特徴運動追跡部３０３は、最初、ｉ番目の手領域情報ｈａｎｄ［ｉ］の手の数が１以下か否かを判定し、１以下の場合、非セグメント特徴運動コードに０をセットする。その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
ｉ番目の手領域情報ｈａｎｄ［ｉ］の手の数が２の場合、特徴運動追跡部３０３は、〈式１１〉を用いて、ｉ番目の手領域情報ｈａｎｄ［ｉ］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ］と、２つめの手の重心ｇ２＿ｈａｎｄ［ｉ］と、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ−１］と、２つめの手の重心ｇ２＿ｈａｎｄ［ｉ−１］とから、ｉ番目の１つめの手領域運動方向ベクトルｖｈ［１］［ｉ］、およびｉ番目の２つめの手領域運動方向ベクトルｖｈ［２］［ｉ］を求める。
【数１１】

【０１７６】
次に、特徴運動追跡部３０３は、＜式１２＞により、ｉ番目の１つめの手領域の移動量ｄｖｈ［１］［ｉ］、およびｉ番目の２つめの手領域の移動量ｄｖｈ［２］［ｉ］を求める。
【数１２】

【０１７７】
図２７は、特徴運動追跡部３０３が非セグメント特徴運動コードを決定するための、手話動作の対称性による非セグメント特徴運動決定条件である。
図２７において、手話動作の対称性による非セグメント特徴運動決定条件は、運動コード２７０１と、条件２７０２とを含む。運動コード２７０１には、番号「０」〜「８」がセットされ、条件２７０２には、運動コード２７０１にセットされた各番号と対応して、各々、手話動作の対称性による非セグメント特徴運動決定条件がセットされる。
次に、特徴運動追跡部３０３は、図２７の手話動作の対称性による非セグメント特徴運動決定条件により、ｉ番目の１つめの手領域運動コードＣｈ［１］［ｉ］、およびｉ番目の２つめの手領域運動コードＣｈ［２］［ｉ］を求める。
非セグメント特徴フレーム数が０の場合、１つめの非セグメント条件始点Ｐｓｈ［１］に、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の１つめの手の重心ｇ１＿ｈａｎｄ［ｉ−１］をセットし、２つめの非セグメント条件始点Ｐｓｈ［２］に、ｉ−１番目の手領域情報ｈａｎｄ［ｉ−１］の２つめの手の重心ｇ２＿ｈａｎｄ［ｉ−１］をセットする。
【０１７８】
ここで、非セグメント要素記憶部２２０１には、予め、手話動作の対称性による非セグメントコード決定条件が記憶されている。
図２８は、非セグメント要素記憶部２２０１に記憶される、手話動作の対称性による非セグメントコード決定条件の一例を示す図である。
図２８の非セグメントコード決定条件には、図示しない手話認識装置が認識対象とする動作（手話動作）において現れるような対照性が、番号１〜１０の条件として設定される。
例えば、手話動作の場合、左右の手は、胴体と垂直または平行な面に対して、互いに対称的な軌跡をとることが多い。なお、認識対象となる、手動作以外の動作についても、同様の条件を設定することができる。
【０１７９】
次に、セグメント位置判定部３０４は、１つめの非セグメント条件始点Ｐｓｈ［１］＝（Ｘｐｓ１，Ｙｐｓ１）と、２つめのセグメント条件始点Ｐｓｈ［２］＝（Ｘｐｓ２、Ｙｐｓ２）と、ｉ番目の１つめの手領域運動コードＣｈ［１］［ｉ］と、ｉ番目の２つめの手領域運動コードＣｈ［２］［ｉ］とをもとに、手話動作の対称性の特徴運動コード（すなわち、ｉ番目の１つめの手領域運動コードＣｈ［１］［ｉ］、およびｉ番目の２つめの手領域運動コードＣｈ［２］［ｉ］）が、図２８の条件（番号１〜１０のいずれかの条件）に一致するか否かを判定し、一致する場合、非セグメント特徴コードに１をセットする。一致しない場合、非セグメント特徴コードに０をセットする。
その後、手話動作セグメンテーション装置は、ステップ２４０３に進む。
以上の手順により、セグメント要素に加えて、非セグメント要素（右手と左手の動作の対称性）をも検出して、それらに基づいてセグメンテーションを行うような手話動作セグメンテーション方法が実現できる。
【０１８０】
ところで、上記の、セグメント要素に加えて、非セグメント要素（右手と左手の動作の対称性）をも検出して、それらに基づいてセグメンテーションを行うような手話動作セグメンテーション方法では、右手と左手の動作の対称性を検出する際に、手話者の動作が平面的に捉えられおり、従って、検出可能な右手と左手の動作の対称性は、２次元的なものに限られる。
そこで、次に、手話者の動作を立体的に捉えて、右手と左手の動作の３次元的な対称性を検出する方法を説明する。
【０１８１】
図２２において、画像入力部３０１は、２台のテレビカメラを含み、３次元画像を入力する。それによって、手話者の動作が立体的に捉えられる。
この場合も、図２２の装置の処理動作は、図２３および２４と基本的には同様であるが、以下の点が変更される。
すなわち、図２３のステップＳ４０３において、身体特徴抽出部３０２は、２台のテレビカメラによる３次元入力画像から、身体特徴、ここでは手領域の画像を抽出する。
３次元画像から手領域を抽出するには、２次元画像から手領域を抽出する場合と同様、ＲＧＢ色情報をもとに肌色領域を抽出すればよい。ただしこの場合、ＲＧＢ色情報には、例えば、３次元画像を構成する各画素のＲＧＢ色情報が、３次元座標の関数として記述される。
その他、「ファジイパターン照合を用いた色彩画像からの顔検出システム」（呉，陳，谷内田；電子情報通信学会論文誌 Ｄ−ＩＩ Ｖｏｌ．Ｊ８０−Ｄ−ＩＩ Ｎｏ．７ ｐｐ．１７７４〜１７８５，１９９７．７）に記載されている方法を用いてもよい。
【０１８２】
手領域を抽出すると、身体特徴抽出部３０２は、ｉ番目の１つめの手領域の３次元座標ｈ［１］［ｉ］、およびｉ番目の２つめの手領域の３次元座標ｈ［２］［ｉ］を求める。
２台のテレビカメラによる３次元画像から抽出された手領域の３次元位置座標を求めるには、一方のテレビカメラによる２次元画像と、他方のテレビカメラによる２次元画像との間に生じる視差を利用すればよい。
【０１８３】
また、上記のステップＳ２４０２ｂを、次のように変更する。
［ステップＳ２４０２ｃ］
２台のテレビカメラのうちの一方、たとえば左側のテレビカメラの画像から算出した手領域の情報を使って、ステップＳ２４０２ｂと同様の処理が行われる。
ただし、特徴運動追跡部３０３は、ｉ番目の１つめの手領域の３次元移動ベクトルＶｔｈ［１］［ｉ］、ｉ番目の２つめの手領域の３次元移動ベクトルＶｔｈ［２］［ｉ］を求める際、〈式１３〉を使って求める。
【数１３】

非セグメント特徴フレーム数が３より小さい場合、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
以上の手順で、右手と左手との動作の３次元的な対称性を検出することができる。
【０１８４】
次に、前述の、セグメント要素に加えて、非セグメント要素（右手と左手の動作の対称性）をも検出して、それらに基づいてセグメンテーションを行うような手話動作セグメンテーション方法において、右手と左手との動作の対称性の変化をどのようにして検出するかを説明する。
【０１８５】
動作の対称性の変化は、例えば、動作面の変化を捉えることにより検出することができる。ここで、動作面とは、動作の軌跡を含むような面をいう。
例えば、手の動作面は、手動作の軌跡を含むような面である。そして、左手および右手の少なくとも一方の動作面が変化した場合、動作の対称性の変化が生じたとみなす。
【０１８６】
動作面の変化を検出するには、例えば、動作面の法線ベクトルの変化を検出すればよい。
そこで、次に、動作面が変化したことを、動作面の法線ベクトルの変化によって検出する方法を説明する。
動作面が変化したことを、動作面の法線ベクトルの変化によって検出する場合、上記のステップＳ２４０２を、次のように変更する。
【０１８７】
［ステップＳ２４０２ｄ］
特徴運動追跡部３０３は、〈式１４〉を使って、ｉ番目の１つめの手領域の３次元移動ベクトルＶｔｈ［１］［ｉ］と、ｉ−１番目の１つめの手領域の３次元移動ベクトルＶｔｈ［１］［ｉ−１］とから、ｉ番目の１つめの手領域運動面の法線ベクトルＶｃｈ［１］［ｉ］を求め、ｉ番目の２つめの手領域の３次元移動ベクトルＶｔｈ［２］［ｉ］と、ｉ−１番目の２つめの手領域の３次元移動ベクトルＶｔｈ［２］［ｉ−１］とから、ｉ番目の２つめの手領域運動面の法線ベクトルＶｃｈ［２］［ｉ］を求める。
【数１４】

【０１８８】
また、〈式１５〉を使って、ｉ番目の１つめの手領域運動面の法線ベクトルＶｃｈ［１］［ｉ］と、ｉ−１番目の１つめの手領域運動面の法線ベクトルＶｃｈ［１］［ｉ−１］とから、ｉ番目の１つめの手領域運動余弦ＣｏｓΘｈ［１］［ｉ］を求め、ｉ番目の２つめの手領域運動面の法線ベクトルＶｃｈ［２］［ｉ］と、ｉ−１番目の２つめの手領域運動面の法線ベクトルＶｃｈ［２］［ｉ−１］とから、ｉ番目の２つめの手領域運動余弦ＣｏｓΘｈ［２］［ｉ］を求める。
【数１５】

【０１８９】
ｉ番目の１つめの手領域運動余弦ＣｏｓΘｈ［１］［ｉ］、およびｉ番目の２つめの手領域運動余弦ＣｏｓΘｈ［２］［ｉ］が、〈式１６〉の条件の少なくとも一方を満たさない場合、非セグメント特徴コードに０をセットする。ここで、α＿ｖｃは、法線方向変化閾値であり、例えば、０．１がセットされる。
【数１６】

その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
以上の手順により、動作面が変化したことを、動作面の法線ベクトルの変化によって検出することができる。
【０１９０】
上記の方法の他に、動作面の変化を検出するのに、動作コードベクトルを用いる方法がある。
そこで、次に、動作面が変化したことを、動作コードベクトルを用いて検出する方法を説明する。
動作面が変化したことを、動作コードベクトルを用いて検出する場合、上記のステップＳ２４０２を、次のように変更する。
【０１９１】
［ステップＳ２４０２ｅ］
特徴運動追跡部３０３は、ｉ番目の１つめの３次元座標ｈ１［ｉ］と、ｉ−１番目の１つめの３次元座標位置ｈ１［ｉ−１］とから、ｉ番目の１つめの手領域３次元運動コードＣｏｄｅ＿ｈ１［ｉ］を求める。ｉ番目の２つめの３次元座標ｈ２［ｉ］と、ｉ−１番目の２つめの３次元座標位置ｈ２［ｉ−１］とから、ｉ番目の２つめの手領域３次元運動コードＣｏｄｅ＿ｈ２［ｉ］を求める。
【０１９２】
ここで、３次元運動コードの算出方法は、例えば、「動作認識装置」（特開平７−２８２２３５号公報）に記載されている。この方法では、手領域の運動が、０から２６の２７個のコードによって表現される。これら２７個のコードは、それぞれ互いに異なる方向を持つ３次元ベクトルと対応している。
一方、非セグメント要素記憶部２２０１には、予め、同一動作面テーブルが記憶されている。
【０１９３】
図２９は、非セグメント要素記憶部２２０１に記憶される同一動作面テーブルの一例を示す図である。
図２９において、同一動作面テーブルには、９個の同一動作面（動作面番号「１」〜「９」）が記載されている。各同一動作面は、上記２７個のコードと同様のコードを用いて表現されている。
特徴運動追跡部３０３は、ｉ番目の１つめの３次元座標ｈ１［ｉ］に基づいて、図２９のテーブルから、ｉ番目の１つめの手領域が含まれる動作面番号と、ｉ番目の２つめの手領域が含まれる動作面番号とを抽出する。
【０１９４】
１つめの手領域動作面候補Ｍｏｖｅ＿ｐｌａｎｅ１に何も設定されていない場合、抽出した１つめの手領域の動作面番号を全て、１つめの手領域動作面候補Ｍｏｖｅ＿ｐｌａｎｅ１にセットし、また、抽出した２つめの手領域の動作面番号を全て、２つめの手領域動作面候補Ｍｏｖｅ＿ｐｌａｎｅ２にセットする。その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
【０１９５】
次に、特徴運動追跡部３０３は、抽出した１つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ１に設定されている動作面番号と一致するものがあるか否か、および、抽出した２つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ２に設定されている動作面番号と一致するものがあるか否かを判定する。
【０１９６】
判定の結果、抽出した１つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ１に設定されている動作面番号と一致するものがない場合、もしくは、抽出した２つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ２に設定されている動作面番号と一致するものがない場合、特徴運動追跡部３０３は、Ｍｏｖｅ＿ｐｌａｎｅ１に設定されている動作面番号を全て削除し、また、Ｍｏｖｅ＿ｐｌａｎｅ２に設定されている動作面番号を全て削除し、さらに、非セグメント特徴コードに０をセットする。その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
【０１９７】
一方、抽出した１つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ１に設定されている動作面番号と一致するものがある場合、１つめの手領域の動作面番号のうち一致する番号のみをＭｏｖｅ＿ｐｌａｎｅ１に設定し、一致しない番号をＭｏｖｅ＿ｐｌａｎｅ１から削除する。
また、抽出した２つめの手領域の動作面番号の中に、Ｍｏｖｅ＿ｐｌａｎｅ１に設定されている動作面番号と一致するものがある場合、２つめの手領域動作面候補Ｍｏｖｅ＿ｐｌａｎｅ２に、１つ以上の動作面番号が設定されていれば、２つめの手領域の動作面番号のうち一致する番号のみを、Ｍｏｖｅ＿ｐｌａｎｅ２に設定し、一致しない番号を、Ｍｏｖｅ＿ｐｌａｎｅ２から削除する。その後、手話動作セグメンテーション装置は、ステップＳ２４０３に進む。
以上の手順により、動作面が変化したことを、動作コードベクトルを用いて検出することができる。
【０１９８】
次に、図示しない手話認識装置と、図３または図２２の手話動作セグメンテーション装置とに付随して設けられ、アニメーション画面を通じ、手話動作セグメンテーション装置がセグメンテーションを行いやすいような遷移動作を行わせるよう利用者を誘導するセグメント要素誘導装置について説明する。
（第４の実施形態）
図３０は、本発明の第４の実施形態に係るセグメント要素誘導装置の構成を示すブロック図である。
図３０のセグメント要素誘導装置は、図示しない手話認識装置と、図３または図２２の手話動作セグメンテーション装置とに付随して設けられる。
図３０において、セグメント要素誘導装置は、認識結果入力部３００１、セグメント結果入力部３００２、誘導制御情報生成部３００３、出力部３００４および誘導規則記憶部３００５を備えている。
【０１９９】
認識結果入力部３００１に接続された手話認識装置から、現在の認識状況情報が認識結果入力部３００１に入力される。セグメント結果入力部３００２に接続された手話動作セグメンテーション装置から、現在のセグメント状況情報がセグメント結果入力部３００２に入力される。
【０２００】
認識結果入力部３００１は、入力された認識状況情報を、誘導制御情報生成部３００３に送る。セグメント結果入力部３００２は、入力されたセグメント状況情報を、誘導制御情報生成部３００３に送る。誘導制御情報生成部３００３は、認識状況情報とセグメント状況情報とをもとに、誘導規則記憶部３００５に記憶された誘導規則を使って誘導制御情報を生成し、出力部３００４に送る。出力部３００４は、出力部３００４に接続された手話アニメーション装置等（図示せず）に、誘導制御情報を出力する。
【０２０１】
以下、上記のように構成されたセグメント要素誘導装置の処理について説明する。
図３１は、図３０のセグメント要素誘導装置の処理の流れを示すフローチャートである。
図３１の各ステップでは、それぞれ以下のような処理が行われる。
［ステップＳ３１０１］
認識結果入力部３００１は、認識結果入力部３００１に接続された手話認識装置から入力される認識状況情報をチェックする。
図３２は、認識結果入力部３００１に入力される認識状況情報の一例を示す図である。
図３２において、認識状況情報は、フレーム番号３２０１、および状況フラグ３２０２を含む。フレーム番号３２０１には、カレントフレーム、つまり手話認識装置が認識状況情報作成時に処理中のフレームのフレーム番号がセットされる。状況フラグ３２０２には、認識に成功していれば「０」、失敗していれば「１」がセットされる。
認識状況情報が入力されると、認識結果入力部３００１は、その認識状況情報を誘導制御情報生成部３００３に送る。
その後、セグメント要素誘導装置は、ステップＳ３１０２に進む。
【０２０２】
［ステップＳ３１０２］
セグメント結果入力部３００２は、手話動作セグメンテーション装置から入力されたセグメント状況情報をチェックする。
図３３は、セグメント結果入力部３００２に入力されるセグメント状況情報の一例を示す図である。
図３３において、セグメント状況情報は、フレーム番号３３０１、および未セグメントフレーム数３３０２を含む。フレーム番号３３０１には、カレントフレーム、つまり手話動作セグメンテーション装置がセグメント状況情報作成時に処理中のフレームのフレーム番号がセットされる。未セグメントフレーム数３３０２には、最後にセグメント対象となったフレームから、カレントフレームまでのフレーム数がセットされる。
セグメント状況情報が入力されると、セグメント結果入力部３００２は、そのセグメント情報を誘導制御情報生成部３００３に送る。
その後、セグメント要素誘導装置は、ステップＳ３１０３に進む。
【０２０３】
［ステップＳ３１０３］
誘導制御情報生成部３００３は、誘導規則記憶部３００５に記憶されている誘導規則を使って、誘導制御情報を作成する。
図３４は、誘導制御情報生成部３００３が作成する誘導制御情報の一例を示す図である。
図３４において、誘導制御情報は、制御部位数３４０１、制御部位３４０２、および制御動作３４０３を含む。制御部位数３４０１には、ＣＧキャラクタ（アニメーション）において、制御対象となる部位の数がセットされる。制御部位３４０２には、ＣＧキャラクタにおいて、制御対象となる部位がセットされる。制御動作３４０３には、制御対象となる部位の動作がセットされる。なお、制御部位３４０２、および制御動作３４０３については、制御部位数３４０１にセットされた部位数に等しい回数、続けてセットが行われる。
【０２０４】
次に、誘導制御情報生成部３００３は、現在入力されている認識状況情報およびセグメント状況情報に応じて、誘導規則記憶部３００５から誘導規則を抽出する。
図３５は、誘導規則記憶部３００５に記憶される誘導規則の一例を示す図である。
図３５において、誘導規則は、認識状況３５０１、非セグメントフレーム数３５０２、制御部位３５０３、および制御動作３５０４を含む。
例えば、図３２の認識状況情報と、図３３のセグメント状況情報とが入力されているとき、認識状況およびグメント状況は、図３５の認識状況３５０１および非セグメントフレーム数３５０２の第２行目に記載の条件と一致する。よって、図３４の誘導制御情報では、制御部位数３４０１に「１」が、制御部位３４０２に「頭」が、制御動作３４０３に「頷き」が、それぞれセットされる。
こうして生成された誘導制御情報は、出力部３００４に送られる。
その後、セグメント要素誘導装置は、ステップＳ３１０４に進む。
【０２０５】
［ステップＳ３１０４］
出力部３００４は、誘導制御情報生成部３００３から送られてきた誘導制御情報を、アニメーション生成装置等に出力する。その際、出力部３００４は、必要に応じ、誘導制御情報を、アニメーション生成装置等から要求される形式に変形する。
その後、セグメント要素誘導装置は、ステップＳ３１０１に進む。
以上の手順により、セグメント要素誘導方法が実現できる。
【０２０６】
次に、上記のセグメント要素誘導方法において、手話動作の認識率に応じて、アニメーションの速度を変化させる場合を説明する。
すなわち、手話認識装置の手話動作認識率を、セグメント要素誘導装置側に与える。セグメント要素誘導装置には、手話動作認識率が悪い場合、提示するアニメーションの速度を下げ、それによって、遷移動作をゆっくり行うよう、手話者を誘導するためのアニメーション速度調整装置を設ける。
【０２０７】
図３６は、図３０のセグメント要素誘導装置に設けられるアニメーション速度調整装置の構成を示すブロック図である。
図３６において、アニメーション速度調整装置は、認識結果入力部３６０１と、セグメント結果入力部３６０２と、速度調整情報生成部３６０３と、速度調整規則記憶部３６０４と、出力部３６０５とを備えている。
認識結果入力部３６０１へは、図示しない手話認識装置からの認識結果情報が入力される。セグメント結果入力部３６０２へは、図３または図２２の手話動作セグメンテーション装置からのセグメンテーション結果情報が入力される。速度調整規則記憶部３６０４には、予め、速度調整規則が記憶されている。速度調整情報生成部３６０３は、速度調整規則を参照しつつ、少なくとも認識結果情報、好ましくは、識結果情報およびセグメンテーション結果情報に基づいて、アニメーションの速度を制御するための制御情報（アニメーション速度調整情報）を生成する。
なお、ここでは、速度調整情報生成部３６０３は、認識結果情報に基づいてアニメーション速度調整情報を生成する場合を説明する。
【０２０８】
上記のように構成されたアニメーション速度調整装置が設けられたセグメント要素誘導装置では、図３１と同様の処理が行われる。ただし、以下の点が異なる。
図３１のステップＳ３１０３を、次のように変更する。
［ステップＳ３１０３ａ］
速度調整情報生成部３６０３は、誤認識フラグｆｌａｇ＿ｒｅｃに何も設定されていない場合、０を設定する。認識結果情報に含まれる状況フラグが１の場合、誤認識フラグｆｌａｇ＿ｒｅｃに１を加える。状況フラグが０で、誤認識フラグｆｌａｇ＿ｒｅｃ＞０の場合、誤認識フラグｆｌａｇ＿ｒｅｃから１を引く。
【０２０９】
図３７は、速度調整規則記憶部３６０４に記憶される速度調整規則の一例を示す図である。
図３７において、速度調整規則は、速度調整量３７０１と、条件３７０２とを含む。条件３７０２は、速度調整量を決めるための条件である。条件３７０２中のｄ＿ｓｐｄは、速度調整パラメータであり、例えば、５０がセットされる。
速度調整情報生成部３６０３は、速度調整規則記憶部３６０４に記憶されている速度調整規則を参照しつつ、誤認識フラグｆｌａｇ＿ｒｅｃに応じた速度調整量ｄ＿ｓｐｄを求める。
こうして求められた速度調整量は、出力部３６０５に送られる。
なお、上記以外の処理は、ステップＳ３１０３と同様であり、省略する。
【０２１０】
また、ステップＳ３１０４を、次のように変更する。
［ステップＳ３１０４ａ］
出力部３６０５は、速度調整量ｄ＿ｓｐｄを、図示しないアニメーション生成装置に送る。アニメーション生成装置は、デフォルトのアニメーション速度Ｓｐｄ＿ｄｅｆから、速度調整量ｄ＿ｓｐｄの分だけ、アニメーションの速度が遅くなるよう調整する。
以上の手順により、手話動作認識率が悪い場合に、提示するアニメーションの速度を下げ、それによって、遷移動作をゆっくり行うよう、手話者を誘導することができる。
【０２１１】
次に、上記のセグメント要素誘導装置（図２２参照；なお、図３６のアニメーション速度調整装置は、設けられていてもいなくてもよい）において、手話者からテレビカメラが見えないように、テレビカメラ隠蔽部を設けた場合を説明する。
なお、テレビカメラが露出していると、手話者がテレビカメラを意識して緊張し、手話動作がぎこちなくなることがある。その結果、セグメンテーションが正しく行われず、手話認識装置の認識率が悪くなることがある。
【０２１２】
図３８は、図２２のセグメント要素誘導装置に設けられるテレビカメラ隠蔽部の構成の一例を示す模式図である。
図３８において、テレビカメラ３８０２は、手話者３８０１と対向して設置され、モニタ３８０３は、テレビカメラ３８０２と手話者３８０１とを結ぶ直線の鉛直下方に、鉛直上方を向いて設置される。
テレビカメラ隠蔽部は、順方向からの光を透過させ、かつ逆方向からの光を反射するようなハーフミラー３８０４を備え、このハーフミラー３８０４を、手話者３８０１とテレビカメラ３８０２とを結ぶ直線上であって、かつモニタ３８０３の鉛直上方の位置に、その直線に対して４５度となるような角度で設置することによって実現される。
【０２１３】
すなわち、モニタ３８０３からの光は、ハーフミラー３８０４で反射されて手話者３８０１に到達するので、手話者３８０１は、モニタ３８０３（に表示されるアニメーション）を見ることができる。
一方、手話者３８０１からテレビカメラ３８０２へ向かう光は、ハーフミラー３８０４を透過するが、テレビカメラ３８０２から手話者３８０１へ向かう光は、ハーフミラーで反射される。よって、テレビカメラ３８０２で手話者３８０１を撮影できるにもかかわらず、撮影の際、手話者３８０１からは、テレビカメラ３８０２が見えない。
このようなテレビカメラ隠蔽部を設けることにより、手話者からテレビカメラが見えないようにすることができる。
【図面の簡単な説明】
【図１】本発明の第１の実施形態に係る手動作分節方法を用いた手動作認識方法を示すフローチャートである。
【図２】図１の方法を実現するコンピュータ装置の構成の一例を示すブロック図である。
【図３】本発明の第２の実施形態に係る手話動作セグメンテーション装置の構成を示すブロック図である。
【図４】図３の手話動作セグメンテーション装置の処理の流れの一例を示すフローチャートである。
【図５】身体特徴抽出部３０２が設定する領域コードの一例を示す図である。
【図６】セグメント要素記憶部３０５に記憶されるセグメント要素データの一例を示す図である。
【図７】身体特徴抽出部３０２によって抽出された肌色領域の一例を示す図である。
【図８】身体特徴抽出部３０２が生成する顔領域情報の一例を示す図である。
【図９】特徴運動追跡部３０３が特徴運動コードを決定するための、顔による特徴運動決定条件を示す図である。
【図１０】動き特徴６０２にセットされる動き特徴パラメータの一例を示す図である。
【図１１】セグメント位置判定部３０４が作成する判定コードデータの一例を示す図である。
【図１２】身体特徴抽出部３０２によって抽出された顔の肌色領域の一例を示す図である。
【図１３】身体特徴抽出部３０２によって作成される目領域情報の一例を示す図である。
【図１４】特徴運動追跡部３０３が特徴運動コードを決定するための、目による特徴運動決定条件を示す図である。
【図１５】身体特徴抽出部３０２が生成する口領域情報の一例を示す図である。
【図１６】特徴運動追跡部３０３が特徴運動コードを決定するための、口による特徴運動決定条件を示す図である。
【図１７】身体特徴抽出部３０２が生成する手領域情報の一例を示す図である。
【図１８】特徴運動追跡部３０３が特徴運動コードを決定するための、胴体と手領域とによる特徴運動決定条件を示す図である。
【図１９】特徴運動追跡部３０３が特徴運動コードを決定するための、手と顔との接触動作による特徴運動決定条件を示す図である。
【図２０】特徴運動追跡部３０３が特徴運動コードを決定するための、手の有効性変化による特徴運動決定条件を示す図である。
【図２１】頷き動作の検出による手話動作セグメンテーション方法（図４参照）において、検出した動作の継続時間を考慮してセグメンテーションを行う場合の処理の流れを示すフローチャートである。
【図２２】本発明の第３の実施形態に係る手話動作セグメンテーション装置の構成を示すブロック図である。
【図２３】図２２の手話動作セグメンテーション装置の処理の流れの一例を示すフローチャートである。
【図２４】図２２の手話動作セグメンテーション装置の処理の流れの一例を示すフローチャートである。
【図２５】非セグメント要素記憶部２２０１に記憶されている非セグメント要素データの一例を示す図である。
【図２６】非セグメント動き特徴２５０２にセットされる非セグメント動き特徴パラメータの一例を示す図である。
【図２７】特徴運動追跡部３０３が特徴運動コードを決定するための、手話動作の対称性による非セグメント特徴運動決定条件である。
【図２８】非セグメント要素記憶部２２０１に記憶される、手話動作の対称性による非セグメントコード決定条件の一例を示す図である。
【図２９】非セグメント要素記憶部２２０１に記憶される同一動作面テーブルの一例を示す図である。
【図３０】本発明の第４の実施形態に係るセグメント要素誘導装置の構成を示すブロック図である（図示しない手話認識装置と、図３または図２２の手話動作セグメンテーション装置とに付随して設けられる）。
【図３１】図３０のセグメント要素誘導装置の処理の流れを示すフローチャートである。
【図３２】認識結果入力部３００１に入力される認識状況情報の一例を示す図である。
【図３３】セグメント結果入力部３００２に入力されるセグメント状況情報の一例を示す図である。
【図３４】誘導制御情報生成部３００３が作成する誘導制御情報の一例を示す図である。
【図３５】誘導規則記憶部３００５に記憶される誘導規則の一例を示す図である。
【図３６】図３０のセグメント要素誘導装置に設けられるアニメーション速度調整装置の構成を示すブロック図である。
【図３７】速度調整規則記憶部３６０４に記憶される速度調整規則の一例を示す図である。
【図３８】図２２のセグメント要素誘導装置に設けられるテレビカメラ隠蔽部の構成の一例を示す模式図である。
【符号の説明】
２０１ ＣＰＵ
２０２ ＲＡＭ
２０３ プログラム記憶部
２０４ 入力部
２０５，３００４，３６０５ 出力部
２０６ 撮像部
２０７ 画像記憶部
２０８ 手話用手動作記憶部
２０９ 遷移動作記憶部
３０１ 画像入力部
３０２ 身体特徴抽出部
３０３ 特徴運動追跡部
３０４ セグメント位置判定部
３０５ セグメント要素記憶部
２２０１ 非セグメント要素記憶部
３００１，３６０１ 認識結果入力部
３００２，３６０２ セグメント結果入力部
３００３ 誘導制御情報生成部
３００５ 誘導規則記憶部
３６０３ 速度調整情報生成部
３６０４ 速度調整規則記憶部
３８０１ 手話者
３８０２ テレビカメラ
３８０３ モニタ
３８０４ ハーフミラー[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a manual action segmentation method and apparatus, and more particularly, to a method and apparatus for automatically segmenting a hand action for each word when automatically recognizing a hand action for sign language or the like. .
[0002]
[Prior art]
In recent years, personal computers can be easily input using a pointing device or the like without performing troublesome keyboard operations, and have been widely used by users other than specialists.
Recently, along with the development of technology for automatically recognizing user's voice, household appliances using personal computers that allow voice input and micro computers that can be used to give voice instructions are also emerging. (Hereinafter, home appliances using personal computers and microcomputers will be referred to as computer devices). If this technology further advances, users will be able to perform input operations on computer devices in a manner close to interpersonal communication. In addition, a user who has difficulty in manual operation can easily use the computer device by voice input.
[0003]
In communication between people, in addition to voice, hand and head movements, facial expressions, and the like are also used. If the computer apparatus can automatically recognize the movement of a specific part of the user's body, the user can perform an input operation in a manner closer to interpersonal communication. In addition, a user who has difficulty in voice operation can easily use a computer device by inputting a sign language, and for example, it is possible to translate a sign language using the computer device.
In order to meet these demands, the present applicant has developed a computer device that recognizes the movement of a specific part of the user's body, including hand movements for sign language and the like. The processing performed in this conventional computer device is as follows when, for example, a hand movement for sign language is to be recognized.
That is, first, the user is photographed and the image is stored. Next, it is specified which part of the image is a hand. Then, a hand movement is detected, and a sign language word is determined based on the detected hand movement. The determination is made with reference to a dictionary describing the behavioral characteristics of sign language. Thus, the computer device “recognizes” the sign language performed by the user.
[0004]
In the following, a process for determining a sign language word based on the hand movement will be specifically described.
In general, sign language words are expressed by several unit actions and combinations thereof. Here, the unit operation means a minimum operation that can be divided, such as raising, lowering, and bending. If each unit motion is A, B, C,..., The sign language words are (A), (B), (C),..., (A, B), (A, C), (B, C), ..., (A, B, C), ... And sign language is performed by combining these sign language words.
For example, if the sign language word (A) means “power” and the sign language word (B, C) means “disconnect”, the sign language words (A) and (B, C) are presented, that is, the unit actions A, B And C are continuously performed, the meaning of “turning off the power” is expressed.
[0005]
In the case of sign language between people, if the speaker performs unit actions A, B, and C continuously with the intention of the sign language words (A) and (B, C), A series of unit actions can usually be intuitively recognized as sign language words (A) and (B, C). However, when sign language is input to the computer device, even if the user performs the unit operations A, B, and C continuously with the intention of the sign language words (A) and (B, C), The series of unit actions A, B and C cannot be recognized as sign language words (A) and (B, C).
[0006]
Therefore, conventionally, the user inserts a predetermined action (hereinafter referred to as segmentation action a) such as stillness between the sign language word (A) and the sign language word (B, C). That is, when it is desired to input “turn off the power”, the user presents the sign language word (A), the segment action a and the sign language word (B, C), that is, the unit action A, the segment action a, and the unit action B. And C are executed sequentially. The computer device detects a series of motions to be presented, and segments (segments) the segment motion a before and after the segment motion a to obtain a sign language word (A) and a sign language word (B, C).
[0007]
[Problems to be solved by the invention]
As described above, in the conventional motion recognition method performed in the computer device, for example, when a user intends to input a sentence composed of a plurality of words into the computer device by a manual operation for sign language, Input had to be made while inserting segmental motions one by one between the corresponding manual motion and the manual motion corresponding to the next word. This is because the conventional motion recognition method cannot automatically segment the detected motion in units of words.
[0008]
In addition, as a method of segmenting a series of detected unit operations (operation code sequences) in units of words, for example, processing similar to that performed by a word processor that segments character code sequences into units of words and converts them into characters. It is possible to do it.
However, in this case, the segment position in the operation code string is obtained by referring to the dictionary in which the word is registered, so the segment position may not be obtained uniquely. In that case, the computer apparatus presents options based on several segment positions to the user, and the user must select his / her intended one from the presented options. Therefore, the user takes time to input, and the input speed does not increase.
[0009]
For example, when the user performs unit actions A, B, and C successively with the intention of sign language words (A) and (B, C), sign language words (A), (B) are stored in the dictionary of the computer device. , (C),..., (A, B), (A, C), (B, C),..., (A, B, C),. When the segment position is obtained, the segment position cannot be determined as one. Therefore, the computer device performs segmentation at several possible positions, and sign language words (A) and (B, C), sign language words (A, B) and (C), and sign language words (A, B, C). ) And other options. In response, the user selects the one he or she intends from the presented options and notifies the computer device of it.
That is, in the method of obtaining the segment position based on such an operation code string, a series of detected operations cannot be automatically segmented in units of words.
[0010]
Therefore, an object of the present invention is to recognize a user's hand movement and automatically segment a detected hand movement into words even if the user side does not present a segment position. It is an object of the present invention to provide a manual motion segmentation method and apparatus.
[0011]
[Means for Solving the Problems and Effects of the Invention]
  The first invention isBlink, which is a feature of transitional motion that appears unconsciously when transitioning from motion indicating one word to motion indicating another word, which does not appear in the user's body during the motion indicating one word, Transition feature data that describes at least one of the features related to at least one of whispering, closing, torso stillness, and hand-to-face contact is stored in advance.In a manual motion segmentation device comprising transition motion storage means, photographing means, image storage means, extraction means, detection means, and manual motion segmentation means, when recognizing a manual motion performed by a user, the manual motion is determined in units of words Manual segmentation method to automatically segment into meaningful units consisting of words, ShootingAn image capturing step in which the shadow unit captures a user and the image data is stored in the image storage unit in units of frames, and an extraction unit extracts the image data from the image storage unit, and a transition operation appears from the image data. The transition motion extraction step for extracting image data corresponding to the color of the body part for each frame, and the detection means compare the image data extracted in the transition motion extraction step with each other to determine the motion of the body part where the transition motion appears. The transition motion detection step for detecting and coding the motion, and the manual motion segmentation means extract the motion code stored in the transition motion storage step from the transition motion storage means, and the motion code and the motion code in the transition motion detection step Compare the motion code to find the time position where the transition motion appeared, and determine the segment position to segment the manual motion based on the time position And a hand operation segmentation step that.
[0012]
According to the first aspect of the present invention, the transition motion that appears when the transition from the motion indicating one word to the motion indicating another word does not appear on the user's body during the motion indicating one word. Since the manual motion is segmented based on the user's side, the detected manual motion can be automatically segmented into word units or meaningful units made up of a plurality of words without presenting the segment position. .
[0022]
  According to a second invention, in the first invention,Pre-stores non-transition feature data describing the characteristics of non-transition actions that appear in the middle of an action that shows one word, instead of appearing on the user's body when moving from one action that shows a word to another. DoNon-transitional motion storage means is further provided,, ExtractionThe output means extracts the image data from the image storage means, and extracts from the image data the image data corresponding to the color of the body part where the non-transition action appears for each frame, and the detection means The non-transition motion detection step for comparing the image data extracted in the transition motion extraction step with each other to detect the motion of the body part in which the non-transition motion appears and coding the motion, and the manual motion segmentation means are stored in the non-transition motion memory. The non-transition that retrieves the operation code stored in the step from the non-transition operation storage means, compares the operation code with the operation code converted into the operation code in the non-transition operation detection step, and obtains the time position where the non-transition operation appears An operation segmentation step. In the manual operation segmentation step, even at a time position where a transition motion appears, at a time position where a non-transition motion appears. It is characterized in that it does not perform the segment.
[0023]
  Above2According to the invention, the segmentation is not performed at the time position where the non-transition motion that appears in the middle of the motion indicating one word does not appear in the user's body at the transition from the motion indicating one word to the motion indicating another word. Therefore, segmentation of words is not mistakenly performed, and segmentation accuracy can be improved.
[0027]
  The third invention is the first invention,, ShootingIn the shadow step, the user is photographed three-dimensionally, the three-dimensional image data is stored in the image storage means, the extraction means takes out the three-dimensional image data from the image storage means, and from the three-dimensional image data, A non-transition action extraction step for extracting three-dimensional image data corresponding to the right hand and the left hand, and a non-transition action detection step for detecting a three-dimensional vector indicating the movement of the right hand and the left hand based on the three-dimensional image data. Then, the hand movement segmentation means detects a change in the movement plane of the right hand and the left hand based on the three-dimensional vector, and if neither the movement plane of the right hand or the movement plane of the left hand changes, A non-transition action segmentation step for determining that a non-transition action indicating that it is in the middle of the indicated action has occurred, and obtaining a time position thereof. Even operation appears time position in the non-transition operation appears time position is characterized by not performing segment.
[0028]
  First4The invention of the3In the invention, non-transition operationSegmentThe step is characterized in that changes in the motion surface of the right hand and the motion surface of the left hand are detected based on changes in the normal vectors of the motion surfaces.
[0029]
  First5The invention of the3In the invention ofThe manual motion segmentation device further includes the same motion surface table creation means and a three-dimensional code string conversion means, the same motion surface table creation means,Describes combinations of 3D motion codes that are included in a single plane for a plurality of 3D motion codes corresponding to 3D vectors having different directions.SameCreating an operation surface table in advance;The three-dimensional code string conversion means is a right hand and a left hand.Converting the motion of the above into a three-dimensional motion code sequence expressed by a plurality of three-dimensional motion codes;Including,Non-transitional motion segment stepNow, let ’s take a look at the changes in the movement of the right handA three-dimensional motion code sequence;The detection is based on the same operation surface table.
[0034]
  The sixth invention is:Blink, which is a feature of transitional motion that appears unconsciously when transitioning from motion indicating one word to motion indicating another word, which does not appear in the user's body during the motion indicating one word, A computer comprising transition motion storage means for preliminarily storing transition feature data describing features relating to at least one of whispering, closing, torso stillness, and hand-face contact as motion codes;
  Photographing means for photographing a user and storing the image data in the image storage means in units of frames;
  Extracting the image data from the image storage means, extracting from the image data image data corresponding to the color of the body part in which the transition action appears, for each frame;
  Detection means for comparing the image data extracted by the extraction means with each other, detecting the motion of the body part in which the transition motion appears, and encoding the motion;
  The operation code is extracted from the transition operation storage means, and the operation code is compared with the operation code converted into the operation code by the detection means to obtain a time position where the transition operation appears, and based on the time position For functioning as a manual motion segmenting means for segmenting the manual motionRecorded the programComputer readableIt is a recording medium.
[0037]
  First7The present invention is a manual motion segmentation device for automatically segmenting a manual motion into a meaningful unit consisting of a word unit or a plurality of words when recognizing a manual motion performed by the user. When a transition from an action indicating one word to an action indicating another word does not appear in the middle of the action indicating one word, UnconsciouslyCharacteristics of transition behavior that appearsFeatures relating to at least one of blinking, whispering, closing, torso rest, and hand-face contactTheAs operation codeThe described transition feature dataIn advanceMemoryTransition action memoryThe user and the userImage storage means in frame unitsRememberphotographMeans,From image storage meansimage dataThe image data, Body parts where the transition motion appearsColorAnd the corresponding imagedataTheEvery frameExtractExtractionMeans,Compare the image data extracted by the extraction means with each other,The body part where the transition motion appearsPlaceDetect motionAnd codingDodetectionMeans,The operation code is extracted from the transition operation storage means, and the operation code and the operation code converted into the operation code by the detection means are obtained.Compare and find the time position where the transition action appeared, Based on the time positionSegmenting hand movementManual segmentationMeans.
[0038]
  First8The invention of the7In the invention ofPre-stores non-transition feature data describing the characteristics of non-transition motion that appears in the middle of the motion indicating one word, but does not appear in the user's body during the transition from motion indicating one word to motion indicating another word A non-transition action storage means that further extracts the image data from the image storage means, and extracts from the image data image data corresponding to the color of the body part where the non-transition action appears for each frame. The detection means further compares the image data corresponding to the body part in which the non-transition motion appears, extracted by the extraction means, detects the motion of the body part in which the non-transition action appears, performs motion coding, and performs manual motion The segmenting means further extracts the action code of the non-transition feature data from the non-transition action storage means, and the body in which the action code and the non-transition action coded by the detection means appear. Compare the position and the action code indicating the motion of the corresponding image to obtain the time position where the non-transition action appears, and even if the time position where the transition action appears, at the time position where the non-transition action appears Characterized by no segmentationIt is said.
[0039]
  First9The invention ofIn the seventh invention, the photographing means photographs the user three-dimensionally and stores the three-dimensional image data in the image storage means, and the extracting means further extracts the three-dimensional image data from the image storage means. The three-dimensional image data corresponding to the right hand and the left hand in which the non-transition action appears is extracted from the three-dimensional image data, and the detecting means further includes the right hand and the left hand based on the three-dimensional image data extracted by the extracting means. The three-dimensional vector indicating the movement is detected, and the hand movement segmenting means further detects a change in the movement plane of the right hand and the left hand based on the three-dimensional vector, and the movement plane of the right hand and the left hand movement plane are detected. If neither changes, it is determined that a non-transition action indicating that it is in the middle of an action indicating one word has occurred, and its time position is obtained. In operation it appeared time position is characterized by not performing segmentaling.
[0040]
  First10The invention of the9In the invention ofThe manual motion segmenting means is characterized by detecting changes in the motion surface of the right hand and the motion surface of the left hand based on a change in a normal vector of the motion surfaces.Yes.
[0041]
  First11The invention ofIn the ninth invention, the manual motion segmentation device further describes a combination of three-dimensional motion codes that are included in one plane with respect to a plurality of three-dimensional motion codes corresponding to three-dimensional vectors each having a different direction. Means for preparing the single motion surface table in advance, and means for converting the movements of the right hand and the left hand into a three-dimensional motion code sequence expressed by a plurality of three-dimensional motion codes, Detecting changes in the right and left hand movement planes based on the same movement plane tableing.
[0045]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a flowchart showing a manual motion recognition method using the manual motion segmentation method according to the first embodiment of the present invention, and FIG. 2 is a block diagram showing an example of the configuration of a computer device that realizes the method of FIG. It is.
2, the computer apparatus includes a CPU 201, a RAM 202, a program storage unit 203, an input unit 204, an output unit 205, an imaging unit 206, an image storage unit 207, a sign language hand movement storage unit 208, A transition operation storage unit 209.
[0046]
The computer apparatus of FIG. 2 is an apparatus that recognizes a hand movement for sign language performed by a user and executes a predetermined process. Specifically, for example, a general-purpose personal computer system is assumed that implements sign language input and automatic sign language translation by installing predetermined program data and connecting a television camera. Alternatively, it may be a home appliance or the like equipped with a microcomputer that performs power on / off or operation mode switching according to a user's manual operation.
The manual motion recognition method shown in FIG. 1 includes a manual motion segmentation process for segmenting a detected manual motion into word units or meaningful units composed of a plurality of words when recognizing a user's manual motion. included.
[0047]
Here, for the purpose of facilitating understanding of the present invention, an outline thereof will be described in advance.
As described in the section of the prior art, in sign language, a sentence is generally expressed using a plurality of sign language words. Each sign language word is composed of a combination of one or more unit actions. On the other hand, since the computer device detects the user's manual motion as a series of unit motions, in order to cause the computer device to recognize the manual motion, the sequence of unit motions is segmented into word units intended by the user in some way. It is necessary to do.
In the conventional segmentation method, the user inserts a stationary motion or the like between the motion corresponding to a certain word and the motion corresponding to the next word, and the computer device detects a stationary motion or the like, thereby performing a series of operations. The unit movement of was segmented. That is, the user side has to intentionally present the segment position.
[0048]
On the other hand, in natural sign language between people, each word is presented continuously. The inventor of the present application unconsciously recognizes the user's body in such a natural sign language between the end of the action corresponding to a word and the start of the action corresponding to the next word. We focused on the appearance of a specific movement. For example, blinking, closing, whispering, and the like (hereinafter, an operation that the user performs unconsciously at a word and a break between words is referred to as a transition operation). Natural still motions inserted between words and word breaks are also included in the transition motion. These transition operations are rarely performed during a manual operation corresponding to one word. Therefore, the inventor of the present application has proposed that the transition motion is used for the segmentation of the manual motion.
[0049]
That is, in the method of FIG. 1, when the computer device detects a hand motion for sign language performed by the user, the computer device also detects a transition motion in parallel therewith. Then, by obtaining the time position where the transition motion appears, the manual motion (that is, a series of unit motions) is segmented into words or meaningful units. Therefore, unlike the conventional segmentation method, the user does not need to intentionally present the segment position.
[0050]
Now, in FIG. 2, the program storage unit 203 stores program data for realizing processing as shown in the flowchart of FIG. The CPU 201 executes the process shown in FIG. 1 according to the program data stored in the program storage unit 203. The RAM 202 stores data necessary for the processing of the CPU 201, work data generated during the processing, and the like.
[0051]
The input unit 204 includes a keyboard and a mouse, and inputs various instructions and data to the CPU 201 according to the operation of the operator. The output unit 205 includes a display and a speaker, and outputs the processing result of the CPU 201 in the form of video or audio.
[0052]
The imaging unit 206 includes at least one television camera and images a user's operation. Here, the number of television cameras may be one when capturing the movement of the user two-dimensionally, but two are necessary when capturing three-dimensionally.
The image storage unit 207 stores the output image of the imaging unit 206 for a plurality of frames. The sign language hand motion storage unit 208 stores sign language feature data indicating the features of the hand motion for sign language. The transition operation storage unit 209 stores transition feature data indicating the characteristics of the transition operation.
[0053]
Here, there are the following three methods for storing program data in the program storage unit 203. The first method is to read the program data from the recording medium on which the program data is recorded and store it in the program storage unit 203. The second is a method of receiving program data transmitted through a communication line and storing it in the program storage unit 203. A third method is to store program data in the program storage unit 203 in advance when the computer apparatus is shipped.
The sign language feature data and the transition feature data can also be stored in the sign language hand motion storage unit 208 and the transition motion storage unit 209 by the same method as the first to third methods.
[0054]
The operation of the computer device configured as described above will be described below with reference to the flowchart of FIG.
First, imaging of the user is started by the imaging unit 206 (step S101). The image data output from the imaging unit 206 is stored in the image storage unit 207 at a predetermined sampling period (for example, at intervals of 1/30 second) (step S102). Each frame of image data stored in the image storage unit 207 is assigned a serial number (frame number) in time series.
[0055]
Next, the CPU 201 extracts data corresponding to the user's hand from each frame of the image data stored in the image storage unit 207 in step S102 (step S103). Then, based on the data extracted in step S103, the movement of the user's hand is detected (step S104). Details of these steps S103 and S104 will be described later.
[0056]
Next, the CPU 201 extracts data corresponding to a specific part of the user's body from the image data stored in the image storage unit 207 in step S102 (step S105). Here, the specific part is a part where the above-mentioned transition operation appears in the user's body, for example, eyes, mouth, face (contour), torso, and the like. In step S105, data corresponding to each part is extracted for at least one part among these specific parts, preferably a plurality of parts. In the following description, it is assumed that data corresponding to eyes, mouth, face, and body are extracted.
[0057]
Next, the CPU 201 detects the movement of each part based on the data extracted in step S105 (step S106). Note that the transition operation appears not only in the eyes, mouth, face, and torso, but also in the hand. For the hand movement, the detection result in step S104 is diverted.
[0058]
Here, the data extraction process in steps S103 and S105 and the motion detection process in steps S104 and S106 will be specifically described.
The data extraction process in steps S103 and S105 is performed as follows, for example.
First, the CPU 201 divides the image data stored in the image storage unit 207 into a plurality of regions associated with each part of the user's body. Here, it is divided into three areas: a hand area including a hand, a face area including a face, and a torso area including a torso. This area division is performed as follows, for example.
[0059]
Next, the user inputs the color of the part to be extracted to the CPU 201 through the input unit 204. That is, in step S103, the hand color (eg, skin color) is input, while in step S105, the white color of the eye (eg, white), the color of the lips (eg, dark red), and the face color (eg, skin color). , And the color of the clothes (eg, blue).
[0060]
In response, the CPU 201 determines whether or not the color indicated by each pixel data matches or is close to the color designated by the user for the plurality of pixel data constituting the image data in each region, and the determination result Only pixel data for which is positive is selected.
That is, in step S103, since only the data indicating the skin color is selected from the image data belonging to the hand region, data corresponding to the hand can be extracted.
On the other hand, in step S105, since only data indicating white color is selected from the face area, data corresponding to the eyes (white eye portion) can be extracted. Similarly, since only data indicating dark red is selected from the face area, data corresponding to the mouth (lip part) can be extracted, and only data indicating skin color is selected from the face area, so that it corresponds to the face. Since data can be extracted and only the data showing blue color is selected from the body region, data corresponding to the body (clothing) can be extracted.
[0061]
The motion detection process in step S104 is performed as follows.
The CPU 201 detects the movement of the hand in each frame by comparing the data extracted from each frame in step S103 with each other. Then, the detected motion is coded by a predetermined procedure.
Therefore, the hand movement detected in step S104 has the form of a code string composed of a plurality of action codes determined for the hand. This operation code string is temporarily stored in the RAM 202.
[0062]
The motion detection process in step S106 is performed as follows.
The CPU 201 detects the movements of the eyes, mouth, face and torso in each frame by comparing the data extracted from each frame in step S105. Then, the detected motion is coded by a predetermined procedure.
Therefore, the movement of each part (eyes, mouth, face, and torso) detected in step S106 has the form of a code string composed of a plurality of operation codes determined for each part. These operation code strings are temporarily stored in the RAM 202.
[0063]
Returning to FIG. 2 again, the processing after step S107 will be described.
Next, the CPU 201 reads the transition feature data from the transition operation storage unit 209 and compares it with the movement of each part detected in step S106. Here, the transition feature data is described using a plurality of operation codes used to express the movement of each part of the user's body in steps S104 and S106. Whether or not there is a part that matches or approximates the transition action (blinking action, closing action, whispering action, hand rest and torso stillness) in the movement of each part (eyes, mouth, face, hand and torso) Is determined (step S107).
Specifically, the CPU 201 searches the operation code string of each part stored in the RAM 202 and determines whether there is a part that matches the operation code or the operation code string of the transition feature data.
If the determination result of step S107 is negative, the CPU 201 proceeds to step S109.
[0064]
If the determination result of step S107 is affirmative, the CPU 201 determines a segment position for segmenting the hand movement detected in step S104 into words (step S108). This segment position determination process is performed as follows.
First, the CPU 201 picks up a part that matches or approximates the transition operation in the movement of each part as a segment position candidate. Specifically, the operation code string of each part stored in the RAM 202 is searched to detect a part that matches or resembles the action code or action code string of the transition feature data, and the time position of these matches or similar parts Is specified using the frame number. The time position specified in this way is hereinafter referred to as a segment position candidate.
[0065]
Next, the CPU 201 compares the segment position candidates picked up for each part as described above, and based on the comparison result, the segment position in the manual operation (a series of unit operations) detected in step S104. To decide.
[0066]
Specifically, in the case of the blinking operation, for example, the moment when the eyelid is closed (that is, the white-eye region disappears) is set as the segment position candidate. In the case of the closing operation, for example, the moment when the lips are closed is set as the segment position. In the case of a whispering motion, for example, the moment when the movement of the lower end of the face turns from downward to upward (the moment when the tip of the jaw reaches the lowest point) is set as a segment position candidate. When the hand is stationary, for example, the moment when the hand movement stops is set as a segment position candidate. In the case where the body is stationary, for example, the moment when the movement of the body stops is set as a segment position candidate.
The segment position candidates for each part thus picked up are compared with each other. For example, in two or more parts, if the picked segment position candidates match each other or are closer than a predetermined interval, the position Is determined as the segment position. More precisely, when the segment position candidates of two or more parts match each other, the matched position is set as the segment position. When segment position candidates of two or more parts are close to each other, the average position of the two or more adjacent positions is set as the segment position (or any one of the two or more adjacent positions is set as the segment position) Also good).
[0067]
In step S109, referring to the segment position determined in step S108, a process of translating the hand movement detected in step S104 is performed.
That is, the CPU 201 segments the hand movement detected in step S104 at the segment position determined in step S108, and the sign language feature data stored in the sign language hand movement storage unit 208 for each sign language word obtained thereby. Compare and translate. Here, the sign language feature data is described using a plurality of operation codes used for expressing the hand operation in step S104.
Thereafter, it is determined whether or not to end the operation (step S110). If the determination result is negative, the process returns to step S101 and the same processing is repeated. If the determination is affirmative, the processing ends.
[0068]
As described above, according to the present embodiment, it does not appear on the user's body in the middle of an action indicating one word, but appears when a transition from an action indicating a word to an action indicating another word is made. Since the manual motion is segmented based on the transition motion, even if the user does not intentionally present the segment position, the computer device automatically detects the detected manual motion in word units or meaningful units consisting of multiple words. Can be segmented automatically.
[0069]
In the first embodiment, the image data includes a hand region including a hand, a face region including a face, and a torso in association with a process of extracting data corresponding to each part of the user's body from the image data. Although it was divided into three regions of the body region, it may be divided into four regions obtained by adding a dead region to these three regions instead. Here, the dead hand area is an area corresponding to the vicinity of the bottom of the screen of the output unit 205 such that the hand is in the area with the arm lowered.
In this case, it is determined that the user is not performing sign language during the period in which the hand is in the dead hand area, and the moment when the hand goes out of the dead hand area is determined to be the start of hand movement. As a result, the computer apparatus can clearly recognize that the user has started the manual operation. Furthermore, it is also possible to detect the entry / exit of the hand into the dead-hand area and use it as a transition operation for segmentation.
[0070]
In the first embodiment, blinking operation, closing operation, whispering operation, hand rest and torso rest (at least one of them) are detected as transition operations for determining the segment position. It is not limited to the operation. For example, a contact operation between a hand and a face may be added to the transition operation. This is because, in sign language, an operation of bringing a hand and face that are separated from each other into contact with each other or separating a hand and face that are in contact with each other is often performed at the beginning or end of a word.
[0071]
In the first embodiment, the duration of the transition operation may be taken into account when determining the segment position. For example, the duration of hand rest is measured and compared with a predetermined threshold value. If the duration is longer than the threshold, the hand rest is determined as a transition action and used for determining the segment position. If the duration is shorter than the threshold, the hand rest is not a transition action. Judge and ignore. Thereby, the accuracy of the segment processing can be increased.
[0072]
In the first embodiment, in addition to the transition operation, a non-transition operation may be further stored, and the segment position may be determined based on the transition operation and the non-transition operation. Here, a non-transition action is performed during a manual movement corresponding to one word, and rarely occurs when moving from one word to the next. Operation. For example, an operation of bringing both hands close to each other, an operation of changing the mouth shape, and the like.
[0073]
Specifically, the computer apparatus of FIG. 2 is further provided with a non-transition action storage unit (not shown), in which non-transition characteristic data indicating the characteristics of the non-transition action is stored. In addition, in step S106 of FIG. 1, in addition to the transition operation, a non-transition operation is also detected. The non-transition operation can be detected in the same manner as the detection of the transition operation. In step S108, segmentation of the manual operation is performed based on the transition operation and the non-transition operation detected in step S106.
[0074]
That is, in the first embodiment, segmented position candidates for each part that are picked up are compared with each other. For example, in two or more parts, the picked segment position candidates coincide with each other or more than a predetermined interval. If they are close to each other, the segment position is determined based on the segment position candidates (specifically, the matched position or the average value of two or more adjacent positions is determined as the segment position). On the other hand, when considering non-transitional motions, non-transitional motions are detected at the same time even if the picked segment position candidates match each other or are closer than a predetermined interval in two or more parts. If so, the segment position is not determined based on the segment position candidates. That is, during the period in which the non-transition operation is performed, even if the transition operation is detected, the segment is not executed. Thereby, the precision of segmentation processing can be improved.
[0075]
In the first embodiment, in order to allow the computer device to accurately detect the transition operation, the correct transition operation (that is, the computer device can easily recognize) is performed through the screen of the output unit 205. An animation image for guiding the user may be presented.
[0076]
Specifically, in the computer apparatus of FIG. 2, animation image data indicating each transition operation is stored in advance in an animation storage unit (not shown). Then, the transition motion detection status (for example, the detection frequency of a specific transition motion is extremely low) and the manual motion recognition status (when manual motion is segmented based on the detected transition motion, the hand motion is recognized. The CPU 201 determines which transition operation should be presented to the user based on whether or not it has been completed, reads out the animation image data indicating the transition operation from the animation storage unit, and provides the data to the output unit 205. Thus, an animation indicating the transition operation is displayed on the screen of the output unit 205, and the user corrects the transition operation while referring to the displayed animation.
[0077]
(Second Embodiment)
FIG. 3 is a block diagram showing a configuration of a sign language action segmentation apparatus according to the second embodiment of the present invention.
In FIG. 3, the sign language action segmentation device includes an image input unit 301, a body feature extraction unit 302, a feature motion tracking unit 303, a segment position determination unit 304, and a segment element storage unit 305.
[0078]
The sign language motion segmentation device is provided in a sign language recognition device (not shown), for example. In addition, it is also provided in computer devices such as home appliances and station ticket machines.
An image captured through an image input device such as a television camera is given to the image input unit 301. Here, unless otherwise noted, since the signer's movement is captured two-dimensionally, only one image input device is required.
[0079]
The image input unit 301 inputs a signer's body image. An image input from the image input unit 301 (hereinafter referred to as an input image) is assigned a number for each frame and is sent to the body feature extraction unit 302. In the segment element storage unit 305, body features and motion features are stored in advance as elements for performing segmentation (hereinafter referred to as segment elements).
The body feature extraction unit 302 extracts an image corresponding to the body feature stored in the segment element storage unit 305 from the input image. The feature motion tracking unit 303 calculates the motion of the body feature based on the extracted image, and sends motion information indicating the calculation result to the segment position determination unit 304.
The segment position determination unit 304 determines the segment position based on the received motion information and the motion feature stored in the segment element storage unit 305, and outputs a frame number indicating the segment position.
[0080]
Here, the image input unit 301, the body feature extraction unit 302, the feature motion tracking unit 303, and the segment position determination unit 304 can be realized by one or a plurality of computers. The segment element storage unit 305 can be realized by a recording device such as a hard disk, a CD-ROM, or a DVD connected to a computer.
[0081]
Hereinafter, a processing procedure of the sign language action segmentation apparatus configured as described above will be described.
FIG. 4 is a flowchart illustrating an example of a process flow of the sign language operation segmentation apparatus of FIG.
In each step of FIG. 4, the following processing is performed.
[0082]
[Step S401]
If there is an input image to the image input unit 301, the image input unit 301 captures an input image for one frame. Then, “1” is added to the frame number i, and then the input image is sent to the body feature extraction unit 302. Thereafter, the sign language action segmentation apparatus proceeds to step S402.
If there is no input image, “0” is added to the frame number i, and then “1” is added to the determination code number j. Thereafter, in the sign language action segmentation apparatus, step S401 is repeated.
[0083]
[Step S402]
The body feature extraction unit 302 divides the spatial region according to the signer's body. The space area is divided by a method similar to the method described in, for example, “Operation Start Position Detection Method” (Japanese Patent Laid-Open No. 9-44668).
That is, the body feature extraction unit 302 first detects a person area based on the color difference between the background and the person in the image data, and then detects the sign language along the detected outline of the person area. Divide the surrounding space area. Then, an area code is set for each area obtained by division.
FIG. 5 is a diagram illustrating an example of an area code set by the body feature extraction unit 302.
In FIG. 5, an input image 501 (spatial region) is divided by a human region outline 502, a head circumscribed rectangle 503, a neck line 504, a torso left line 505, a torso right line 506, and a dead hand determination line 507. .
[0084]
Specifically, the body feature extraction unit 302 first detects the position of the neck from the contour line 502 of the person area, and draws the neck line 504 parallel to the X axis at the height of the neck.
Next, from the lower end of the screen, a dead hand determination line 507 is drawn horizontally to the X axis at a position obtained by multiplying the height of the neck line 504 by the dead hand determination ratio. Here, the dead hand determination ratio is a parameter for determining the effectiveness of the hand, and when the hand enters below the dead hand determination line 507, even if the hand movement is performed, the hand movement is performed. Is considered to be invalid, that is, the same as when hands are not used. In the present embodiment, the dead hand determination ratio is set to approximately 1/5.
[0085]
Next, an area code is set in each area obtained by dividing as described above. The numbers in the circles in the figure are area codes. In the present embodiment, the area code is set as shown in FIG. 5, for example. That is, the area outside the head circumscribed rectangle 503 and above the neck line 504 is (1), the area inside the head circumscribed rectangle 503 is (2), and the neck line 504 and the dead hand determination line 507 are The region left between the fuselage left line 505 is (3), and the region between the neck line 504 and the dead hand determination line 507 and sandwiched between the fuselage left line 505 and the fuselage right line 506 is (4). ▼, the area between the neck line 504 and the dead hand determination line 507 and the right side of the trunk right line 506 is set as (5), and the area under the dead hand determination line 507 is set as (6). .
Thereafter, the sign language action segmentation apparatus proceeds to step S403.
[0086]
[Step S403]
The body feature extraction unit 302 extracts an image corresponding to the body feature stored in the segment element storage unit 305 from the input image. The image thus extracted is hereinafter referred to as an extracted body feature.
FIG. 6 is a diagram illustrating an example of segment element data stored in the segment element storage unit 305.
In FIG. 6, the segment element data includes a body feature 601 and a motion feature 602. One or more body features, here, a face region, eyes, a mouth, a hand region and a torso, a hand region and a face region, and a hand region are set in the body feature 601.
[0087]
On the other hand, in the movement feature 602, a movement feature corresponding to each body feature set in the body feature 601 is set. That is, a blinking vertex is set corresponding to the face area, a blink is set corresponding to the eye, a mouth shape change is set corresponding to the mouth, a motion stationary is set corresponding to the hand area and the torso, A hand-face contact is set corresponding to the region and the face region, and a hand effectiveness change point is set corresponding to the hand region.
The body feature extraction unit 302 detects the body feature set in the body feature 601 as the extracted body feature. For example, when “face region” is set in the body feature 601, the body feature extraction unit 302 extracts a face region as the extracted body feature.
[0088]
Here, a method for extracting a face area will be described.
The body feature extraction unit 302 first extracts a skin color area from the input image based on the RGB color information. Next, from the extracted skin color region, a portion that is obtained by dividing in step S402 and that overlaps with the region (head region) where the region code is (2) is taken out and set as a face region.
[0089]
FIG. 7 is a diagram illustrating an example of a skin color region extracted by the body feature extraction unit 302.
As shown in FIG. 7, the skin color area includes a face skin color area 702 and a hand skin color area 703. Therefore, if the skin color area is simply extracted based on the RGB color information, the skin color area 702 of the face and the skin color area 703 of the hand are extracted and cannot be distinguished from each other. Therefore, as shown in FIG. 5, the input image is divided into regions {circle around (1)} to {circle around (6)} in advance and overlaps the head region 701 (region {circle around (2)} in FIG. 5) among the extracted skin color regions. Take out only the part. Thereby, a skin color area 702 of the face is obtained.
[0090]
Next, the body feature extraction unit 302 generates face area information. That is, for the extracted face area, the center of gravity, area, X-axis direction maximum length, and Y-axis direction maximum length are set in the i-th face area information face [i].
FIG. 8 is a diagram illustrating an example of face area information generated by the body feature extraction unit 302.
In FIG. 8, the face area information includes the center of gravity coordinates 801 of the face area, the area 802 of the face area, the maximum length 803 of the face area in the X-axis direction, and the maximum length 804 of the face area in the Y-axis direction.
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0091]
[Step S404]
If the frame number i is 1, the sign language action segmentation apparatus goes to step S401. On the other hand, if the frame number i is not 1, the process proceeds to step S405.
[0092]
[Step S405]
The feature motion tracking unit 303 uses the <expression 1> to calculate the feature motion code of the face region from the i-th face region information face [i] and the i-1th face region information face [i-1]. Ask for. Further, the i-th face area is calculated from the centroid g_face [i] of the i-th face area information face [i] and the centroid g_face [i-1] of the i-1th face area information face [i-1]. The face motion direction vector V_face [i] is obtained.
[Expression 1]

[0093]
Next, the feature motion tracking unit 303 determines a feature motion code from the i-th face motion direction vector V_face [i].
FIG. 9 is a diagram illustrating a feature motion determination condition by a face for the feature motion tracking unit 303 to determine a feature motion code.
In FIG. 9, the feature motion determination condition by the face includes a motion code 901 and a condition 902. Numbers “1” to “8” are set in the motion code 901, and a feature motion determination condition based on the face is set in the condition 902 corresponding to each number set in the motion code 901.
That is, the feature motion tracking unit 303 compares the i-th face motion direction vector V_face [i] with the condition 902 in FIG. 9 and uses a face that matches the i-th face motion direction vector V_face [i]. Select feature motion determination conditions. Then, a feature motion code is determined by extracting a number corresponding to the feature motion determination condition for the selected face from the motion code 901 in FIG.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0094]
[Step S406]
The segment position determination unit 304 compares the feature motion code with the segment element data (see FIG. 6) stored in the segment element storage unit 305, and checks whether the feature motion code matches the motion feature 602. To do. In the motion feature 602, a parameter (motion feature parameter) indicating a motion feature (motion feature) for segment determination is set.
[0095]
FIG. 10 is a diagram illustrating an example of a motion feature parameter set in the motion feature 602.
In FIG. 10, the motion feature parameters include a motion feature 1001, a determination code 1002, a time 1003, and a segment position 1004. A motion feature 1001 indicates the type of motion feature. The determination code 1002 is a code string that is a condition for determining a motion feature. The time 1003 is a time that is a condition for determining a motion feature. The segment position 1004 indicates the segment position in the motion feature.
[0096]
In the code string included in the determination code 1002, each code uses the same numbers “1” to “8” as the motion code 901 (characteristic motion code) in FIG. 9 and the number “0” indicating motion rest. It is expressed and the code is connected with “-”.
Whether or not the feature motion code determined in step S405 matches, for example, “1-0-2”, is whether or not the code continues in the order of “0” and “2” after “1”. If the codes continue in that order, it is determined that they match.
[0097]
Note that the codes in () are considered to match even without them. For example, “7- (0) -3” has the condition that the code “7” is followed by “0” and “3”, and the code “7” is followed by “3”. Considered a match.
Further, a code with “/” in between does not show “/” indicating that either code may be used. For example, in the case of “0/3”, “0” or “3” is regarded as a condition match.
On the other hand, “*” represents any code.
[0098]
When a whirling motion is detected, the body feature 601 in FIG. 6 is a “face region”, and “whisker vertex” is set as the motion feature 602. At this time, the segment position determination unit 304 determines whether or not the facial feature motion code determined in step S405 matches the code string “7- (0) -3” corresponding to the “whisker vertex” in FIG. judge.
The sign language action segmentation apparatus determines whether j is 1 or not. If j = 1, the process proceeds to step S407.
If j> 1, the process proceeds to step S409.
[0099]
[Step S407]
The sign language action segmentation device determines whether or not the feature motion code matches the first code of the determination code 1002, and if they match, the process proceeds to step S408. If not, the process proceeds to S401.
[0100]
[Step S408]
The segment position determination unit 304 creates determination code data. That is, the feature motion code is set to the code number of the first determination code data Code_data [1], and i is set to the code start frame number of the first determination code data Code_data [1].
FIG. 11 is a diagram illustrating an example of determination code data created by the segment position determination unit 304.
In FIG. 11, the determination code data includes a code number 1101, a code start frame number 1102, and a code end frame number 1103.
[0101]
For example, in the case of the example of FIG. 10, if the feature motion code is “7”, “7” is set to the code number of the first determination code data Code_data [1], and the first determination code data Code_data [1 ] Is set to the code start frame number of].
Thereafter, 2 is set to j, and the sign language action segmentation apparatus proceeds to step S401.
[0102]
[Step S409]
It is determined whether or not the feature motion code matches the code number of the (j-1) th determination code data Code # data [j-1]. If they match, the sign language action segmentation apparatus proceeds to step S401. .
If the feature motion code does not match the code number of the j−1th determination code data Code # data [j−1], the process proceeds to step S410.
[0103]
[Step S410]
The segment position determination unit 304 sets i−1 to the code end frame number of the (j−1) th determination code data Code # data [j−1]. Thereafter, the sign language action segmentation apparatus proceeds to step S411.
[0104]
[Step S411]
It is determined whether or not the number of codes of the determination code 1002 is j or more. If the number of determination codes 1002 is j or more, the sign language action segmentation apparatus proceeds to step S412.
If the number of codes of the determination code 1002 is j−1, the process proceeds to step S417.
[0105]
[Step S412]
It is determined whether or not the j-th code of the determination code 1002 matches the feature movement code. If they do not match, the sign language action segmentation apparatus proceeds to step S413.
If they match, the process proceeds to step S416.
[0106]
[Step S413]
It is determined whether or not the j-th code of the determination code 1002 is attached with (). If it is attached with (), the sign language action segmentation apparatus goes to step S414.
If not (), the process proceeds to step S415.
[0107]
[Step S414]
It is determined whether or not the j + 1-th code of the determination code 1002 matches the feature movement code. If they do not match, the sign language action segmentation apparatus proceeds to step S415.
If they match, 1 is added to j, and then the process proceeds to step S416.
[0108]
[Step S415]
j is set to 1, and then the sign language action segmentation apparatus proceeds to step S401.
[0109]
[Step S416]
The feature motion code is set to the code number of the jth determination code data Code # data [j]. Also, i is set to the code start frame number of the jth determination code data Code # data [j]. Then, 1 is added to j. Thereafter, the sign language action segmentation apparatus proceeds to step S401.
[0110]
[Step S417]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the motion feature is “swing vertex”, the segment position corresponding to “swing vertex” is the Y coordinate lowest point, and the segment position determination unit 304 obtains the frame number corresponding to the Y coordinate lowest point.
That is, for each frame corresponding to the code end frame number of the j−1th determination code data Code # data [j−1] from the code start frame number of the first determination code data Code # data [1], The Y coordinates of the center of gravity of the face area are compared with each other. Then, the frame number of the frame having the minimum Y coordinate (that is, the center of gravity of the face region is at the lowest point) is set as the segment position in the motion feature.
When a plurality of frame numbers corresponding to the lowest Y coordinate point appear, the first frame number (the smallest number) is set as the segment position.
Thereafter, the sign language action segmentation apparatus proceeds to step S418.
[0111]
[Step S418]
The sign language motion segmentation device outputs the segment position. Then, it returns to step S401 and repeats the same process as the above.
With the above procedure, a sign language motion segmentation method by detecting a whispering motion can be realized.
[0112]
Next, a sign language motion segmentation method by detecting blinking motion will be described.
In the sign language motion segmentation method based on the detection of blinking motion, the processing in step S403 is changed as follows in the sign language motion segmentation method based on the detection of the whispering motion (see FIG. 4).
[Step S403a]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When the blinking motion is detected, “eye” is set in the body feature 601, and the body feature extraction unit 302 extracts the eye as the extracted body feature.
[0113]
Here, an eye extraction method will be described.
First, a face area is extracted by the same method as in step S403. Next, eyes are extracted from the extracted face area as follows.
FIG. 12 is a diagram illustrating an example of a face area extracted by the body feature extraction unit 302.
In FIG. 12, the extracted face area 1201 includes a hole area 1202 by eyebrows, a hole area 1203 by eyes, and a hole area 1204 by mouth (the shaded area is a skin color area).
[0114]
In the figure, a straight line indicated by reference numeral 1205 is a face vertical division line. The face vertical division line 1205 is a line that divides the extracted face area 1201 vertically.
First, the face vertical division line 1205 is drawn at a position designated by the face vertical division ratio between the upper end of the face and the lower end of the face. Here, the face vertical division ratio is a parameter, and is set so that the eye hole area 1203 falls within the area above the face vertical division line 1205. In the present embodiment, for example, the face vertical division ratio is set to “½”.
[0115]
Next, a hole area in the face area above the face vertical division line 1205 is detected.
If there are two detected hole areas, the hole areas are determined as eyebrows and it is determined that the eyes are closed.
If there are three detected hole areas, it is determined that one eye is closed, and the lower one of the three hole areas is determined as the eye.
When the detected hole areas are four, it is determined that both eyes are open, and two lower ones of the four hole areas are determined as eyes.
In the case of the example of FIG. 12, since there are four hole regions, the lower two hole regions can be recognized as the hole region 1203 by the eyes.
[0116]
Next, the body feature extraction unit 302 generates eye area information. That is, the number of extracted eyes and the area of the eyes are set in the i-th eye area information eye [i].
FIG. 13 is a diagram illustrating an example of eye region information generated by the body feature extraction unit 302.
In FIG. 13, the eye area information includes an eye number 1301, a first (eye) area 1302, and a second area 1303.
The body feature extraction unit 302 first sets the number of eyes extracted to the number 1301 of eyes. Then, according to the number of extracted eyes, the area of the eyes is set as follows.
[0117]
When the number of extracted eyes is 0, 0 is set to the first area 1302 and the second area 1303, respectively.
When the number of extracted eyes is 1, the area of the eyes (the hole region 1203 by the eyes) is calculated and set to the first area 1302. 0 is set in the second area 1303.
When the number of extracted eyes is 2, the area of the second eye is calculated, and the area of the left eye (the left side of the hole region 1203 by the eye) is set in the first area 1302, and the second area 1302 In the area 1303, the area of the right eye is set.
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0118]
In the sign language motion segmentation method based on the detection of blinking motion, the processing in step S405 is changed as follows.
[Step S405a]
Using the <Expression 2>, the feature motion tracking unit 303 calculates the feature motion code of the eye from the i-th eye region information eye [i] and the i-1 th eye region information eye [i-1]. Ask. Also, from the first area s1_eye [i] of the i-th eye region information eye [i] and the first area s1_eye [i-1] of the i-1th eye region information eye [i-1]. , The eye area area change d1_eye [i] of the first eye of the i-th eye is obtained. Also, from the second area s2_eye [i] of the i-th eye region information eye [i] and the second area s2_eye [i-1] of the i-1th eye region information eye [i-1]. The second area area change d2_eye [i] of the i-th eye is obtained.
[Expression 2]

[0119]
FIG. 14 is a diagram illustrating the feature motion determination condition by the eye for the feature motion tracking unit 303 to determine the feature motion code.
In FIG. 14, the characteristic motion determination condition by the eye includes a motion code 1401 and a condition 1402. Numbers “0” to “6” are set in the movement code 1401, and the characteristic movement determination condition by the eye is set in the condition 1402 corresponding to each number set in the movement code 1401.
Α in the condition 1402 is an eye area threshold value for determining whether or not the eyes are closed. For example, “1” is set. β is an eye size change threshold value for determining whether or not the size of the eye is changing. For example, “5” is set.
[0120]
That is, the feature motion tracking unit 303 calculates the i-th eye region information eye [i], the i-th first eye region area change d1_eye [i], and the second eye region area change d2_eye [i]. Compared to the 14th condition 1402, the i-th eye region information eye [i], the i-th first eye region area change d1_eye [i], and the second eye region area change d2_eye [i] are matched. The feature motion determination condition by the eyes is selected. Then, a feature motion code is determined by taking out the number corresponding to the feature motion determination condition by the selected eye from the motion code 1401 of FIG.
For example, when both eyes are closed, s1_eye [i] ≦ α and s2_eye [i] ≦ α are satisfied, and at this time, the feature motion code is 0.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0121]
In the sign language motion segmentation method based on the detection of blinking motion, the process of step S417 is changed as follows.
[Step S417a]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the motion feature is “blink”, the segment position corresponding to “blink” is the eye region vanishing point, and the segment position determination unit 304 obtains the frame number corresponding to the eye region vanishing point.
That is, the code start frame number of the second determination code data Code # data [2] is set as the segment position.
Thereafter, the sign language action segmentation apparatus proceeds to step S418.
With the above procedure, a sign language motion segmentation method by detecting blinking motion can be realized.
[0122]
Next, a sign language motion segmentation method based on detection of mouth shape change (closed mouth) will be described.
In this case, in the sign language motion segmentation method (see FIG. 4) based on the detection of the whispering motion described above, the process in step S403 is changed as follows.
[Step S403b]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When detecting a mouth shape change (closed), “mouth” is set in the body feature 601, and the body feature extraction unit 302 extracts the mouth as the extracted body feature.
[0123]
Here, a mouth extraction method will be described.
First, a face area is extracted by the same method as in step S403. Next, the mouth is extracted from the extracted face area as follows.
In FIG. 12, the face vertical division line 1205 is first drawn as in step S403. Next, a hole area in the face area below the face vertical division line 1205 is detected.
[0124]
When there are two or more detected hole areas, the hole area whose distance from the lower end of the face area is closest to the mouth height condition is defined as the mouth. Here, the mouth height condition is a parameter, and represents the position of a standard human mouth by the distance from the lower end of the face area. In the present embodiment, the mouth height condition is set to “10”.
When the detected hole area is one, the hole area is used as a mouth.
If no hole area is detected, the mouth is considered closed.
In the example of FIG. 12, since there is only one hole area below the face vertical dividing line 1205, the hole area can be understood as a hole area 1204 by the mouth.
[0125]
Next, the body feature extraction unit 302 generates mouth area information. That is, the extracted mouth area and the maximum length in the Y-axis direction of the mouth are set in the i-th mouth area information mouth [i].
FIG. 15 is a diagram illustrating an example of mouth area information generated by the body feature extraction unit 302.
In FIG. 15, the mouth area information includes a mouth area 1501 and a mouth maximum length 1502 in the Y-axis direction.
The body feature extraction unit 302 calculates the extracted mouth area, sets it to the mouth area 1501, calculates the maximum length of the mouth in the Y-axis direction, and calculates the maximum length of the mouth in the Y-axis direction. Set to 1502.
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0126]
In the sign language motion segmentation method based on the detection of the mouth shape change, the process in step S405 is changed as follows.
[Step S405b]
The feature movement tracking unit 303 uses <Equation 3> to obtain a mouth characteristic movement code from the i-th mouth area information mouth [i] and the i-1th mouth area information mouth [i-1]. Ask. Further, from the area s_mout [i] of the i-th mouth region information mouth [i] and the area s_mouth [i−1] of the i−1th mouth region information mouth [i−1], Mouth area change d_mouth [i] is obtained.
[Equation 3]

[0127]
Further, using <Expression 4>, the Y-axis direction maximum length h_mouth [i] of the i-th mouth area information mouth [i] and the i-th mouth area information mouth [i-1] in the Y-axis direction From the maximum length h_mouth [i−1], the Y-axis direction change y_mouth [i] of the i-th mouth is obtained.
[Expression 4]

[0128]
FIG. 16 is a diagram illustrating a condition motion determination condition by mouth for the feature motion tracking unit 303 to determine a feature motion code.
In FIG. 16, the characteristic movement determination condition by the mouth includes a movement code 1601 and a condition 1602. Numbers “0” and “1” are set in the movement code 1601, and the characteristic movement determination condition by the mouth is set in the condition 1602 corresponding to each number set in the movement code 1601.
Γ in the condition 1602 is a mouth area change threshold value for determining whether or not there is a change in mouth shape. In this embodiment, for example, “5” is set. λ is the mouth Y-axis direction change threshold, and for example, “3” is set.
[0129]
That is, the feature motion tracking unit 303 compares the i-th mouth region area change d_mouth [i] and the Y-axis direction maximum length h_mouth [i] with the condition 1602 in FIG. The feature motion determination condition by the mouth is selected so as to coincide with [i] and the maximum Y-axis direction length h_route [i]. Then, a feature motion code is determined by taking out the number corresponding to the feature motion determination condition by the selected mouth from the motion code 1601 of FIG.
For example, when the mouth is closed, s_mouth [i] ≦ γ, and the characteristic motion code is “0”.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0130]
In the sign language operation segmentation method based on the detection of the mouth shape change, the processing in step S417 is changed as follows.
[Step S417b]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the motion feature is “mouth shape change”, the segment position corresponding to “mouth shape change” is the change start point end point, and the segment position determination unit 304 obtains the frame number corresponding to the change start point end point.
That is, the code start frame number of the second determination code data Code # data [2] and the code end frame number of the second determination code data Code # data [2] are output as segment positions.
Thereafter, the sign language action segmentation apparatus proceeds to step S418.
With the above procedure, a sign language motion segmentation method based on detection of mouth shape change can be realized.
[0131]
Next, a sign language motion segmentation method based on detection of still motions of the hand and the torso will be described.
In this case, in the sign language motion segmentation method (see FIG. 4) based on the detection of the whispering motion described above, the processing in step S403 is changed as follows.
[Step S403c]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When detecting the motions of the hand and the torso, “hand region, torso” is set in the body feature 601, and the body feature extraction unit 302 extracts the hand region and the torso as the extracted body feature.
[0132]
  Here, a method for extracting the hand region and the trunk will be described.
  First, the body feature extraction unit 302 extracts a hand region by the same method as in step S403. That is, a skin color region is extracted from the input image, a portion that does not overlap with the head region is extracted from the extracted skin color region, and is used as a hand region.
  In the case of FIG. 7, the skin color area does not overlap with the head area, that is, the hand.Skin colorA region 703 is extracted.
  For the trunk, the person area detected in step S402 is used as the trunk as it is.
[0133]
Next, the body feature extraction unit 302 generates hand region information. That is, regarding the extracted hand region, the center of gravity, the area, the maximum length in the X-axis direction, and the maximum length in the Y-axis direction are set in the i-th hand region information hand [i]. Next, the center of gravity, area, X-axis direction maximum length, and Y-axis direction maximum length of the extracted body are set in the i-th body information body [i].
FIG. 17 is a diagram illustrating an example of hand region information generated by the body feature extraction unit 302.
In FIG. 17, the hand region information includes the number of hands 1701, the center of gravity coordinates 1702 of the first hand, the area 1703 of the first hand, the center of gravity coordinates 1704 of the second hand, and the area 1705 of the second hand. included.
The body feature extraction unit 302 first sets the number of extracted hands to the number of hands 1701. Then, according to the number of extracted hands, the center-of-gravity coordinates and the area of the hand are set as follows.
[0134]
  When the number of extracted hands 1701 is 0, (0, 0) is set for the center of gravity coordinates 1702 of the first hand and the center of gravity coordinates 1704 of the second hand, respectively, and the area 1703 of the first hand is set. And the area of the second hand1705Is set to 0.
  When the number of extracted hands 1701 is “1”, the barycentric coordinates and area of the hand region are calculated and set to the barycentric coordinates 1702 of the first hand and the area 1703 of the first hand. Also, set the center of gravity coordinate 1704 of the second hand to (0, 0), and the area of the second hand1705Set to 0.
  When the number of extracted hands 1701 is “2”, the center-of-gravity coordinates and area of the left hand side of the two hand regions are calculated and set to the center-of-gravity coordinates 1702 of the first hand and the area 1703 of the first hand. To do. In addition, the center-of-gravity coordinates and the area of the right region of the two hand regions are calculated and set to the center-of-gravity coordinates 1704 of the second hand and the area 1705 of the second hand.
  The body information body [i] can be realized with the configuration of FIG. 8 as with the face area information face [i].
  Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0135]
In the sign language motion segmentation method by detecting the still motion of the hand and the torso, the process in step S405 is changed as follows.
[Step S405c]
Using the <Expression 5>, the feature motion tracking unit 303 uses the i-th hand region information hand [i], the i-1th hand region information hand [i-1], and the i-th body information body [ i] and the (i-1) th body information body [i-1], the characteristic motion codes of the hand region and the body are obtained. Further, the center of gravity g1_hand [i] of the first hand of the i-th hand region information hand [i] and the center of gravity g1_hand [i− of the first hand of the i-th hand region information hand [i-1]. 1], the movement amount m1_hand [i] of the first hand in the i-th hand region is obtained. Also, the center of gravity g2_hand [i] of the second hand of the i-th hand region information hand [i] and the center of gravity g2_hand [i− of the second hand of the i−1th hand region information hand [i−1]. 1], the movement amount m2_hand [i] of the second hand in the i-th hand region is obtained.
[Equation 5]

[0136]
Further, using <Expression 6>, the area s1_hand [i] of the first hand of the i-th hand region information hand [i] and the first of the i-1th hand region information hand [i-1]. The area change amount d1_hand [i] of the first hand of the i-th hand region is obtained from the area s1_hand [i-1] of the hand. In addition, the area s2_hand [i] of the second hand of the i-th hand region information hand [i] and the area s2_hand [i− of the second hand of the i-th hand region information hand [i−1]. 1], an area change amount d2_hand [i] of the second hand of the i-th hand region is obtained.
[Formula 6]

[0137]
Further, using <Expression 7>, the centroid g_body [i] of the i-th body information body [i] and the centroid g_body [i-1] of the i-1th body information body [i-1]. , The moving amount m_body [i] of the i-th body is obtained.
[Expression 7]

[0138]
FIG. 18 is a diagram illustrating conditions for determining characteristic motions based on the trunk and the hand region.
In FIG. 18, the characteristic motion determining condition based on the torso and hand region includes a motion code 1801 and a condition 1802. Numbers “0” and “1” are set in the motion code 1801, and characteristic motion determination conditions based on the body and the hand region are set in the condition 1802 corresponding to each number set in the motion code 1801. Is done.
Χ in the condition 1802 is a stop determination threshold value for the hand region. In the present embodiment, for example, “5” is set. δ is a shape change determination threshold value of the hand region, and for example, “10” is set. ε is a torso stop determination threshold, for example, “5” is set.
[0139]
That is, the feature motion tracking unit 303 moves the first hand movement amount m1_hand [i] in the i-th hand region, the second hand movement amount m2_hand [i] in the i-th hand region, and the i-th hand region. The area change amount d1_hand [i] of the first hand of the first hand, the area change amount d2_hand [i] of the second hand of the i-th hand region, and the movement amount m_body [i] of the i-th body are shown in FIG. Compared to the condition 1802, the movement amount m1_hand [i] of the first hand in the i-th hand region, the movement amount m2_hand [i] of the second hand in the i-th hand region, and 1 in the i-th hand region. The hand region that matches the area change amount d1_hand [i] of the first hand, the area change amount d2_hand [i] of the second hand of the i-th hand region, and the movement amount m_body [i] of the i-th body And torso characteristic movement To select a constant conditions. Then, from the motion code 1801 of FIG. 18, the feature motion code is determined by taking out the number corresponding to the feature motion determination condition by the selected hand region and torso.
For example, when the hand region is moving left and right, the movement amount m_hand [i]> χ of the i-th hand region is satisfied, and at this time, the characteristic motion code is “1”.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0140]
In the sign language motion segmentation method by detecting the still motion of the hand and the torso, the process in step S417 is changed as follows.
[Step S417c]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the motion feature is “motion stationary”, the segment position corresponding to “motion stationary” is the motion stationary start point, and the segment position determination unit 304 obtains the frame number corresponding to the motion stationary start point.
Or you may obtain | require the frame number corresponding to the intermediate point of a stationary area. In this case, the code start frame number of the first determination code data Code_data [1] and the code end frame number of the first determination code data Code_data [1] are obtained, and an intermediate value thereof may be calculated.
Thereafter, the sign language action segmentation apparatus proceeds to step S418.
Through the above procedure, a sign language motion segmentation method based on detection of still motion of the hand and the torso can be realized.
[0141]
Next, a sign language motion segmentation method based on detection of a contact motion between a hand and a face will be described.
In this case, in the sign language motion segmentation method (see FIG. 4) based on the detection of the whispering motion described above, the processing in step S403 is changed as follows.
[Step S403d]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When detecting a contact operation between a hand and a face, “face area, hand area” is set in the body feature 601, and the face area and the hand area are extracted as extracted body features.
[0142]
Here, a method of extracting the face area and the hand area will be described.
First, the face area is extracted by the same method as in step S403, and the hand area is extracted by the same method as in step S403c.
Next, the center of gravity, area, X-axis direction maximum length, and Y-axis direction maximum length of the extracted face area are set in the i-th face area information face [i]. For the extracted hand region, the center of gravity, the area, the maximum length in the X-axis direction and the maximum length in the Y-axis direction are set in the i-th hand region information hand [i].
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0143]
In the sign language motion segmentation method based on the detection of the contact motion between the hand and the face, the process in step S405 is changed as follows.
[Step S405d]
The feature motion tracking unit 303 uses <Equation 8> to calculate the feature motion codes of the hand region and the face region from the i-th hand region information hand [i] and the i-th face region information face [i]. Ask. Further, from the centroid g1_hand [i] of the first hand of the i-th hand area information hand [i] and the centroid g_face [i] of the i-th face area information face [i], the i-th hand area information The distance l1_fh [i] between the first hand and the face is obtained. Further, the i-th hand from the centroid g2_hand [i] of the second hand of the i-th hand area information hand [i] and the centroid g_face [i-1] of the i-th face area information face [i]. The distance l2_fh [i] between the second hand of the region and the face is obtained.
[Equation 8]

[0144]
However, if the area s1_hand [i] of the first hand of the i-th hand region information hand [i] is 0, l1_fh [i] = 0 if l1_fh [i−1] ≦ Φ. If l1_fh [i-1]> Φ, l1_fh [i] = 1000.
Similarly, if the area s2_hand [i] of the second hand of the i-th hand region information hand [i] is 0, l2_fh [i] = 0 if l2_fh [i−1] ≦ Φ. If l2_fh [i-1]> Φ, then l2_fh [i] = 1000. Here, Φ is a distance threshold between the face and the hand. In this embodiment, for example, “20” is set.
[0145]
FIG. 19 is a diagram illustrating a feature motion determination condition based on a contact operation between a hand and a face for the feature motion tracking unit 303 to determine a feature motion code.
In FIG. 19, the characteristic motion determination condition based on the contact operation between the hand and the face includes a motion code 1901 and a condition 1902. Numbers “0” and “1” are set in the movement code 1901, and the characteristic movement determination based on the contact operation between the hand and the face is associated with each number set in the movement code 1901 in the condition 1902. A condition is set.
In the condition 1902, ω is a contact threshold between the hand area and the face area. In this embodiment, for example, “5” is set.
[0146]
That is, the feature motion tracking unit 303 calculates the distance l1_fh [i] between the first hand and the face in the i-th hand region and the distance l2_fh [i] between the second hand and the face in the i-th hand region. 19, the distance l1_fh [i] between the first hand and the face of the i-th hand region and the distance l2_fh [i] of the second hand and the face of the i-th hand region as compared with the condition 1902 of FIG. ] Is selected such that the feature motion determination condition matches. Then, by extracting the number corresponding to the selected feature motion determination condition from the motion code 1901 in FIG. 19, the feature motion code is determined.
For example, when the right hand overlaps the face, the distance between the first hand of the i-th hand region and the face is l1_fh [i] = 0, and at this time, the feature motion code is “0”.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0147]
In the sign language motion segmentation method based on the detection of the contact motion between the hand and the face, the process in step S417 is changed as follows.
[Step S417d]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the movement feature is “contact operation between hand and face”, the segment position corresponding to “hand-face contact” is “contact start point end point”, and segment position determination unit 304 starts and ends the hand face contact section. And the corresponding frame number is obtained.
That is, the code start frame number of the first determination code data Code # data [1] and the code end frame number of the first determination code data Code # data [1] are set as segment positions.
Thereafter, the sign language action segmentation apparatus proceeds to step S401.
With the above procedure, a sign language motion segmentation method can be realized by detecting the contact motion between the hand and the face.
[0148]
Next, a sign language motion segmentation method based on detection of a change in hand effectiveness will be described.
In this case, in the sign language motion segmentation method (see FIG. 4) based on the detection of the whispering motion described above, the processing in step S403 is changed as follows.
[Step S403e]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When detecting a change in effectiveness of the hand, a “hand region” is set in the body feature 601, and the body feature extraction unit 302 extracts the hand region as the extracted body feature.
[0149]
The hand region is extracted by the same method as in step S403c.
Next, the body feature extraction unit 302 sets the center of gravity, area, X-axis direction maximum length, and Y-axis direction maximum length for the extracted hand region in the i-th hand region information hand [i].
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0150]
In the sign language operation segmentation method by detecting the change in the effectiveness of the hand, the process in step S405 is changed as follows.
[Step S405e]
The feature motion tracking unit 303 obtains a feature motion code based on hand effectiveness and motion from the i-th hand region information hand [i] using the above-described <Expression 5>.
In addition, a plurality of regions obtained by dividing the space of the first hand in step S402 from the center of gravity g1_hand [i] of the first hand of the i-th hand region information hand [i] (see FIG. 5). The region code is determined, and the region code is obtained and set in the hand region space code sp1_hand [i] of the first hand. If the area s1_hand [i] of the first hand of the i-th hand region information hand [i] is 0, “6” is set to the hand region space code sp1_hand [i] of the first hand.
[0151]
Also, an area code is obtained from the center of gravity g2_hand [i] of the second hand of the i-th hand area information hand [i], and set to the hand area space code sp2_hand [i] of the second hand. When the area s2_hand [i] of the second hand of the i-th hand area information hand [i] is 0, “6” is set to the hand area space code sp2_hand [i] of the second hand.
[0152]
Further, the center of gravity g1_hand [i] of the first hand of the i-th hand region information hand [i] and the center of gravity g1_hand [i− of the first hand of the i-th hand region information hand [i-1]. 1], the movement amount m1_hand [i] of the first hand in the i-th hand region is obtained.
Also, the center of gravity g2_hand [i] of the second hand of the i-th hand region information hand [i] and the center of gravity g2_hand [i− of the second hand of the i−1th hand region information hand [i−1]. 1], the movement amount m2_hand [i] of the second hand in the i-th hand region is obtained.
[0153]
FIG. 20 is a diagram illustrating a feature motion determination condition based on a change in hand effectiveness for the feature motion tracking unit 303 to determine a feature motion code.
In FIG. 20, the characteristic motion determining condition based on the change in the effectiveness of the hand includes a motion code 2001 and a condition 2002. Numbers “0” to “5” are set in the movement code 2001, and the characteristic movement determination by the contact action of the hand and the face is associated with each number set in the movement code 2001 in the condition 2002. A condition is set.
Χ in the condition 2002 is a stop determination threshold value for the hand region. In this embodiment, for example, “5” is set.
[0154]
That is, the feature motion tracking unit 303 includes the hand region space code sp1_hand [i] of the first hand of the i-th hand region, the movement amount m1_hand [i] of the first hand of the i-th hand region, the i-th hand region. The hand region space code sp2_hand [i] of the second hand of the hand region and the movement amount m2_hand [i] of the second hand of the i-th hand region are compared with the condition 2002 of FIG. Hand region space code sp1_hand [i] of the first hand of the hand region, movement amount m1_hand [i] of the first hand of the i-th hand region, hand region space code of the second hand of the i-th hand region A feature motion determination condition is selected based on the contact motion between the hand and the face that matches sp2_hand [i] and the movement amount m2_hand [i] of the second hand in the i-th hand region.
For example, when the right hand is moving and the left hand is lowered to the lowest position of the input image 501 (see FIG. 5), the movement amount m1_hand [i]> of the first hand in the i-th hand region> χ, and the hand area space code sp2_hand [i] = 7 of the second hand of the i-th hand area, and the feature motion code is “2”.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0155]
In the sign language operation segmentation method by detecting the change in the effectiveness of the hand, the process in step S417 is changed as follows.
[Step S417e]
The segment position determination unit 304 obtains the segment position in the motion feature based on the motion feature 1001 and the segment position 1004 (see FIG. 10).
When the motion feature is “hand effectiveness change point”, the segment position corresponding to “hand effectiveness change point” is “code change point”, and the segment position determination unit 304 has a frame corresponding to the code change point. Ask for a number.
That is, the code start frame number of the first determination code data Code_data [1] and the code end frame number of the first determination code data Code_data [1] are set as segment positions.
Thereafter, the process proceeds to step S418.
With the above procedure, a sign language motion segmentation method by detecting a change in hand effectiveness can be realized.
[0156]
Next, a sign language motion segmentation method based on a combination of the above motion detections will be described.
In this case, step S403 is changed as follows in the sign language motion segmentation method (see FIG. 4) based on the detection of the whispering motion described above.
[Step S403f]
The body feature extraction unit 302 extracts an image corresponding to the body feature 601 (see FIG. 6) stored in the segment element storage unit 305 from the input image.
When each of the above motions is detected, “face area”, “eyes”, “mouth”, “hand area, torso”, “hand area, face area”, and “hand area” are set in the body feature 601. The feature extraction unit 302 extracts a face region, eyes, mouth, hand region, and trunk as extracted body features.
The face area is extracted by the same method as in step S403. The eyes are extracted by the same method as in step S403a. The mouth is extracted by the same method as in step S403b. The hand region and the trunk are extracted by the same method as in step S403c.
[0157]
Next, the body feature extraction unit 302 uses the facial area information face [i], the eye area information eye [i], and the mouth area information “mouth [ i], hand area information hand [i], and body information body [i].
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0158]
Then, the sign language action segmentation apparatus processes step S405 to step S417, and then processes step S405b to step S417b. Thereafter, the processing from step S405c to step S417c, the processing from step S405d to step S417d, and the processing from step S405e to step S417d are processed in order.
Thereby, the sign language action segmentation method by the combination of the detection of each action described above can be realized.
[0159]
Next, a sign language motion segmentation method that performs segmentation in consideration of the duration of the detected motion in each of the above methods will be described.
FIG. 21 is a flowchart showing the flow of processing when segmentation is performed in consideration of the duration of the detected motion in the sign language motion segmentation method (see FIG. 4) by detecting the whispering motion.
The method of FIG. 21 is obtained by changing step S411 as follows and adding step S2101 to the method of FIG.
[Step S411a]
It is determined whether or not the number of codes of the determination code 1002 is j or more. If it is j or more, the process proceeds to step S412.
If j−1, the process proceeds to step S2101.
[0160]
[Step S2101]
First, the number of frames from the frame corresponding to the code start frame number of the first determination code data Code_data [1] to the frame corresponding to the code end frame number of the j−1th determination code data Code_data [j−1]. Is set to the feature duration.
Next, it is determined whether or not a value is set at the time 1003 of the motion feature parameter (see FIG. 10), and it is further determined whether or not the feature duration is smaller than the value set at the time 1003. The
If a value is set for the time 1003 and the feature duration is smaller than the value set for the time 1003, the process proceeds to step S415.
If nothing is set for the time 1003 or if the feature duration is greater than or equal to the value set for the time 1003, the process proceeds to step S417.
With the above procedure, a sign language motion segmentation method that performs segmentation in consideration of the duration of the detected motion can be realized.
[0161]
Next, a sign language operation segmentation method in which non-segment elements are detected in addition to segment elements to perform segmentation will be described.
(Third embodiment)
FIG. 22 is a block diagram showing a configuration of a sign language action segmentation apparatus according to the third embodiment of the present invention.
The apparatus of FIG. 22 further includes a non-segment element storage unit 2201 in the apparatus of FIG. The non-segment element storage unit 2201 stores in advance non-segment elements that are conditions for not performing a segment. The other components are the same as those in FIG.
That is, the apparatus of FIG. 22 executes a sign language action segmentation method that detects non-segment elements in addition to segment elements and performs segmentation of sign language actions based on the non-segment elements.
[0162]
Hereinafter, a processing procedure of the sign language action segmentation apparatus configured as described above will be described.
First, a case where the approach of both hands is detected as a non-segment element will be described.
23 and 24 are flowcharts showing an example of the processing flow of the sign language action segmentation device of FIG.
In the method of FIGS. 23 and 24, step S2401 is added after step S403, steps S2402 to S2405 are added after step S405, and step S418 is changed to step S418a in the method of FIG. Is.
In each of these steps (S2401 to S2405, S418a), the following processing is performed.
[0163]
[Step S2401]
The body feature extraction unit 302 extracts an image corresponding to the body feature stored in the non-segment element storage unit 2201 from the input image.
FIG. 25 is a diagram illustrating an example of non-segment element data stored in the non-segment element storage unit 2201.
In FIG. 25, the non-segment element data includes a body feature 2501 and a non-segment motion feature 2502.
For example, when detecting the approach of both hands, “hand region” is set in the body feature 2501.
The body feature extraction unit 302 extracts the hand region as a non-segment body feature. The hand area can be extracted by the procedure of step S403c.
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0164]
[Step S2402]
In the following procedure, non-segment feature motion codes are determined.
When the number of hands in the i-th hand region information hand [i] is 2, the feature motion tracking unit 303 uses <Equation 9> to calculate the first hand in the i-th hand region information hand [i]. From the centroid g1_hand [i] and the centroid g2_hand [i] of the second hand, an inter-hand distance l_hand [i] of the i-th hand region is obtained.
[Equation 9]

[0165]
Next, using <Equation 10>, from the labor distance l_hand [i] of the i-th hand area and the labor distance l_hand [i-1] of the i-1th hand area, the labor distance change amount dl_hand [ i].
[Expression 10]

[0166]
The number of hands in the i-th hand area information hand [i] is not 2, or the number of hands in the i-th hand area information hand [i] and the i-1th hand area information hand [i-1]. ], The feature motion tracking unit 303 sets a non-negative value, for example, 1000 to the inter-hand distance change amount dl_hand [i].
When the inter-hand distance change amount dl_hand [i] ≦ −θ, the non-segment feature motion code is “1”. When the inter-hand distance change amount dl_hand [i]> − θ, the non-segment feature motion code is “0”. Here, θ is a manual distance change threshold, and in this embodiment, for example, “5” is set.
When nothing is set in the non-segment code number k, “1” is set in the non-segment code number k, and “0” is set in the number of non-segment feature frames.
Here, the non-segment code number k indicates the number of codes constituting the non-segment feature motion code, and the non-segment feature frame number is the number of frames corresponding to the non-segment motion feature detection period, that is, the detection end from the detection start frame. Indicates the number of frames up to the frame.
Thereafter, the sign language action segmentation apparatus goes to Step 3003.
[0167]
[Step S2403]
The segment position determination unit 304 compares the non-segment feature motion code with the non-segment element data stored in the non-segment element storage unit 2201 (see FIG. 25). Check if it matches 2502. In the non-segment motion feature 2502, a parameter (non-segment motion feature parameter) indicating a motion feature (non-segment motion feature) for determining a non-segment is set.
[0168]
FIG. 26 is a diagram illustrating an example of a non-segment motion feature parameter set in the non-segment motion feature 2502.
In FIG. 26, the non-segment motion feature parameters include a non-segment motion feature 2601, a determination code 2602, and a time 2603. The non-segment motion feature 2601 indicates the type of non-segment motion feature. The determination code 2602 is a code string that is a condition for determining a non-segment motion feature. Time 2603 is a time that is a condition for determining the non-segment motion feature.
The determination code 2602 is described in the same manner as the determination code 1002 in the motion feature parameter of FIG. At time 2603, the minimum duration of the non-segment motion feature 2601 is set.
[0169]
If the determination code 2602 is different from the k-th code of the non-segment feature motion code determined in step S2402, that is, the last code constituting the non-segment feature motion code, the sign language action segmentation apparatus goes to step S2404.
If they are the same, the process proceeds to step S2405.
[0170]
[Step S2404]
“0” is set to the number of non-segment feature frames, and “1” is set to the non-segment code number k.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0171]
[Step S2405]
“1” is added to the number of non-segment feature frames.
Since k> 2 and the (k−1) th code in the non-segment defined code string condition is different from the non-segment feature motion code, “1” is added to k.
Thereafter, the sign language action segmentation apparatus proceeds to step S406.
[0172]
[Step S418a]
If no value is set at time 2603 in the non-segment motion feature parameter (see FIG. 26), 0 is set as the minimum non-segment time.
When a value is set for the time 2603, the value of the time 2603 is set to the minimum non-segment time.
If the number of non-segment feature frames is smaller than the number of frames corresponding to the non-segment time minimum value, the segment position set in step S417 is output.
Thereafter, the sign language action segmentation apparatus proceeds to step S401.
According to the above procedure, a sign language motion segmentation method can be realized in which non-segment elements (two-hand approach) are detected in addition to segment elements, and segmentation is performed based on them.
[0173]
Next, a case where a mouth shape change is detected as a non-segment element will be described.
In this case, the above step S2401 is changed as follows.
[Step S2401a]
The body feature extraction unit 302 extracts an image corresponding to the body feature stored in the non-segment element storage unit 2201 from the input image.
In FIG. 25, when detecting a mouth shape change, “mouth” is set in the body feature 2501.
The body feature extraction unit 302 extracts the mouth as a non-segment body feature. The mouth can be extracted by the procedure of step S403b.
Thereafter, the sign language action segmentation apparatus proceeds to step S404.
[0174]
Further, the above step S2402 is changed as follows.
[Step S2402a]
In the following procedure, non-segment feature motion codes are determined.
The feature motion tracking unit 303 first performs the same procedure as step S405b above, the mouth area area change d_mout [i] of the i-th mouth area information, and the Y-axis direction change y_mouth [i] of the i-th mouth area information. ].
Next, the mouth area area change d_mout [i] of the i-th mouth area information and the Y-axis direction change y_mouth [i] of the i-th mouth area information are compared with the condition 1602 in FIG. The feature motion determination condition by the mouth is selected so as to coincide with the mouth region area change d_mouth [i] of the mouth region information and the Y-axis direction change y_mouth [i] of the i-th mouth region information. Then, a non-segment feature motion code is determined by taking out the number corresponding to the feature motion determination condition by the selected mouth from the motion code 1601 of FIG.
For example, when the mouth is not moved, the area of the mouth and the maximum length in the Y-axis direction do not change, and at this time, the non-segment feature motion code is “0”.
When nothing is set in the non-segment code number k, “1” is set in the non-segment code number k, and “0” is set in the number of non-segment feature frames.
Thereafter, the sign language operation segmentation apparatus goes to step S2403.
According to the above procedure, a sign language operation segmentation method can be realized in which non-segment elements (mouth shape change) are detected in addition to segment elements, and segmentation is performed based on them.
[0175]
Next, a description will be given of the case where the symmetry of the movement of the right hand and the left hand is detected as a non-segment element.
In this case, the above step S2402 is changed as follows.
[Step S2402b]
In the following procedure, non-segment feature motion codes are determined.
The feature motion tracking unit 303 first determines whether or not the number of hands in the i-th hand region information hand [i] is 1 or less, and if it is 1 or less, sets 0 in the non-segment feature motion code. Thereafter, the sign language operation segmentation apparatus goes to step S2403.
When the number of hands in the i-th hand region information hand [i] is 2, the feature motion tracking unit 303 uses <Equation 11> to determine the first hand in the i-th hand region information hand [i]. The center of gravity g1_hand [i], the center of gravity g2_hand [i] of the second hand, the center of gravity g1_hand [i-1] of the first hand of the (i-1) th hand region information hand [i-1], and the second From the centroid g2_hand [i-1] of the hand of the i-th, the i-th first hand region motion direction vector vh [1] [i], and the i-th second hand region motion direction vector vh [2] [i ].
## EQU11 ##

[0176]
Next, the feature motion tracking unit 303 calculates the movement amount dvh [1] [i] of the i-th first hand region and the movement amount dvh [2] of the i-th second hand region according to <Equation 12>. ] [I].
[Expression 12]

[0177]
FIG. 27 shows the non-segment feature motion determination condition based on the symmetry of the sign language action for the feature motion tracking unit 303 to determine the non-segment feature motion code.
In FIG. 27, the non-segment feature motion determination condition based on the symmetry of the sign language operation includes a motion code 2701 and a condition 2702. Numbers “0” to “8” are set in the motion code 2701, and the non-segment feature motion determination based on the symmetry of the sign language motion is performed in the condition 2702 corresponding to each number set in the motion code 2701. A condition is set.
Next, the feature motion tracking unit 303 determines the i-th first hand region motion code Ch [1] [i] and the i-th 2 in accordance with the non-segment feature motion determination condition based on the symmetry of the sign language motion of FIG. The finger movement region code Ch [2] [i] is obtained.
When the number of non-segment feature frames is 0, the centroid g1_hand [i-1] of the first hand of the i-1th hand region information hand [i-1] is added to the first non-segment condition start point Psh [1]. And the center of gravity g2_hand [i-1] of the second hand of the (i-1) th hand region information hand [i-1] is set to the second non-segment condition start point Psh [2].
[0178]
Here, the non-segment element storage unit 2201 stores in advance non-segment code determination conditions based on the symmetry of sign language action.
FIG. 28 is a diagram illustrating an example of the non-segment code determination condition based on the symmetry of the sign language operation stored in the non-segment element storage unit 2201.
In the non-segment code determination condition of FIG. 28, contrast that appears in an operation (sign language operation) that is a recognition target by a sign language recognition device (not shown) is set as a condition of numbers 1 to 10.
For example, in the case of a sign language operation, the left and right hands often take a symmetrical locus with respect to a plane perpendicular or parallel to the trunk. It should be noted that the same conditions can be set for operations other than manual operations that are to be recognized.
[0179]
Next, the segment position determination unit 304 determines that the first non-segment condition start point Psh [1] = (Xps1, Yps1), the second segment condition start point Psh [2] = (Xps2, Yps2), and the i th Based on the first hand region motion code Ch [1] [i] and the i-th second hand region motion code Ch [2] [i], a symmetrical feature motion code (ie, sign language motion) , The i-th first hand region motion code Ch [1] [i] and the i-th second hand region motion code Ch [2] [i]) are the conditions (numbers 1 to 10) of FIG. It is determined whether or not any one of the conditions) is met. If they match, 1 is set in the non-segment feature code. If they do not match, 0 is set in the non-segment feature code.
Thereafter, the sign language action segmentation apparatus goes to Step 2403.
According to the above procedure, a sign language motion segmentation method can be realized in which non-segment elements (symmetry of motion of the right hand and left hand) are detected in addition to the segment elements, and segmentation is performed based on the detected non-segment elements.
[0180]
By the way, in the sign language motion segmentation method that detects non-segment elements (symmetry of motion of the right hand and left hand) in addition to the segment elements and performs segmentation based on them, the motion of the right hand and the left hand When detecting the symmetry of the sign language, the signer's motion is captured in a plane, and therefore the detectable symmetry of the right and left hand motions is limited to two-dimensional.
Therefore, a method for detecting the three-dimensional symmetry of the right and left hand movements by capturing the movement of the sign language in three dimensions will be described next.
[0181]
In FIG. 22, an image input unit 301 includes two television cameras and inputs a three-dimensional image. As a result, the signer's movement is captured three-dimensionally.
Also in this case, the processing operation of the apparatus of FIG. 22 is basically the same as that of FIGS. 23 and 24, but the following points are changed.
That is, in step S403 in FIG. 23, the body feature extraction unit 302 extracts a body feature, here, an image of a hand region from a three-dimensional input image by two television cameras.
In order to extract a hand region from a three-dimensional image, a skin color region may be extracted based on RGB color information as in the case of extracting a hand region from a two-dimensional image. In this case, however, the RGB color information describes, for example, the RGB color information of each pixel constituting the three-dimensional image as a function of the three-dimensional coordinates.
In addition, “Face Detection System from Color Images Using Fuzzy Pattern Matching” (Kure, Chen, Taniuchida; IEICE Transactions D-II Vol. J80-D-II No. 7 pp. 1774 to 1785, 1997) .7) may be used.
[0182]
When the hand region is extracted, the body feature extraction unit 302 extracts the three-dimensional coordinates h [1] [i] of the i-th first hand region and the three-dimensional coordinates h [2] of the i-th second hand region. [I] is obtained.
In order to obtain the three-dimensional position coordinates of the hand region extracted from the three-dimensional images by the two television cameras, the parallax generated between the two-dimensional image by one television camera and the two-dimensional image by the other television camera is calculated. Use it.
[0183]
Further, the above step S2402b is changed as follows.
[Step S2402c]
Using the information of the hand area calculated from the image of one of the two television cameras, for example, the left television camera, the same processing as step S2402b is performed.
However, the feature motion tracking unit 303 performs the three-dimensional movement vector Vth [1] [i] of the i-th first hand region and the three-dimensional movement vector Vth [2] [i] of the i-th second hand region. Is obtained using <Equation 13>.
[Formula 13]

If the number of non-segment feature frames is smaller than 3, the sign language action segmentation apparatus goes to step S2403.
With the above procedure, the three-dimensional symmetry of the movement of the right hand and the left hand can be detected.
[0184]
Next, in the sign language motion segmentation method in which non-segment elements (symmetry of the motion of the right hand and the left hand) are detected in addition to the segment elements described above, and segmentation is performed based on them, the right hand and the left hand How to detect the change in the symmetry of the operation will be described.
[0185]
The change in the symmetry of the operation can be detected, for example, by capturing the change in the operation surface. Here, the operation surface refers to a surface including an operation trajectory.
For example, the motion surface of the hand is a surface that includes a locus of manual motion. When at least one of the left and right hand motion surfaces changes, it is considered that a change in the symmetry of the motion has occurred.
[0186]
In order to detect a change in the operation surface, for example, a change in the normal vector of the operation surface may be detected.
Therefore, a method for detecting that the operating surface has changed by detecting a change in the normal vector of the operating surface will be described next.
When it is detected that the operating surface has changed by changing the normal vector of the operating surface, step S2402 described above is changed as follows.
[0187]
[Step S2402d]
The feature motion tracking unit 303 uses the expression 14 to calculate the three-dimensional movement vector Vth [1] [i] of the i-th first hand region and the three-dimensional of the i−1 first hand region. From the movement vector Vth [1] [i-1], a normal vector Vch [1] [i] of the i-th first hand region motion surface is obtained, and the i-th second hand region is three-dimensionally moved. From the vector Vth [2] [i] and the three-dimensional movement vector Vth [2] [i-1] of the (i-1) th second hand region, the normal line of the i-th second hand region motion surface. Vector Vch [2] [i] is obtained.
[Expression 14]

[0188]
Also, using <Expression 15>, the normal vector Vch [1] [i] of the i-th first hand region motion surface and the normal vector Vch of the i-1st first hand region motion surface. From [1] [i−1], the i-th first hand region motion cosine CosΘh [1] [i] is obtained, and the normal vector Vch [2] [1] of the i-th second hand region motion surface is obtained. i] and the normal vector Vch [2] [i-1] of the (i-1) th second hand region motion plane, the i-th second hand region motion cosine CosΘh [2] [i] Ask.
[Expression 15]

[0189]
The i-th first hand region motion cosine CosΘh [1] [i] and the i-th second hand region motion cosine CosΘh [2] [i] do not satisfy at least one of the conditions of Equation 16. In this case, 0 is set in the non-segment feature code. Here, α_vc is a normal direction change threshold, for example, 0.1 is set.
[Expression 16]

Thereafter, the sign language operation segmentation apparatus goes to step S2403.
With the above procedure, it is possible to detect that the operating surface has changed by detecting a change in the normal vector of the operating surface.
[0190]
In addition to the above method, there is a method using an operation code vector to detect a change in the operation surface.
Therefore, a method for detecting the change of the operation plane using the operation code vector will be described next.
When it is detected using the operation code vector that the operation surface has changed, the above step S2402 is changed as follows.
[0191]
[Step S2402e]
The feature motion tracking unit 303 calculates the i-th first hand from the i-th first three-dimensional coordinate h1 [i] and the (i-1) th first three-dimensional coordinate position h1 [i-1]. The region three-dimensional motion code Code_h1 [i] is obtained. From the i-th second three-dimensional coordinate h2 [i] and the (i−1) -th second three-dimensional coordinate position h2 [i−1], the i-th second hand region three-dimensional motion code Code_h2 [ i].
[0192]
Here, the calculation method of the three-dimensional motion code is described in, for example, “motion recognition device” (Japanese Patent Laid-Open No. 7-282235). In this method, the movement of the hand region is represented by 27 codes from 0 to 26. These 27 codes correspond to three-dimensional vectors each having a different direction.
On the other hand, the non-segment element storage unit 2201 stores the same operation surface table in advance.
[0193]
FIG. 29 is a diagram illustrating an example of the same operation surface table stored in the non-segment element storage unit 2201.
In FIG. 29, nine identical operation surfaces (operation surface numbers “1” to “9”) are described in the same operation surface table. Each identical operation plane is expressed using the same code as the 27 codes.
Based on the i-th first three-dimensional coordinates h1 [i], the feature motion tracking unit 303 calculates the motion surface number including the i-th first hand region and the i-th 2nd from the table of FIG. The motion surface number including the finger region is extracted.
[0194]
If nothing is set in the first hand region motion surface candidate Move_plane1, all the motion surface numbers of the extracted first hand region are set in the first hand region motion surface candidate Move_plane1 and the extracted 2 All the motion surface numbers of the second hand region are set in the second hand region motion surface candidate Move_plane2. Thereafter, the sign language operation segmentation apparatus goes to step S2403.
[0195]
Next, the feature motion tracking unit 303 determines whether or not the extracted motion surface number of the first hand region matches the motion surface number set in Move_plane1, and the extracted second motion region number. It is determined whether or not there is an action surface number in the hand region that matches the operation surface number set in Move_plane2.
[0196]
As a result of the determination, if the motion surface number of the extracted first hand region does not match the motion surface number set in Move_plane1, or the motion surface number of the extracted second hand region is If none of the motion plane numbers set in Move_plane2 matches, the feature motion tracking unit 303 deletes all the motion plane numbers set in Move_plane1, and the motion set in Move_plane2 All face numbers are deleted, and 0 is set to the non-segment feature code. Thereafter, the sign language operation segmentation apparatus goes to step S2403.
[0197]
On the other hand, if there is an action face number that matches the action face number set in Move_plane 1 among the action face numbers of the extracted first hand area, only the matching number among the action face numbers set in Move_plane 1 Set to Move_plane1 and delete the non-matching numbers from Move_plane1.
Also, if there is an action surface number that matches the action surface number set in Move_plane1 among the extracted action surface numbers of the second hand area, one or more actions are included in the second hand area action surface candidate Move_plane2 If the face number is set, only the matching number among the action face numbers of the second hand region is set in Move_plane2, and the unmatched number is deleted from Move_plane2. Thereafter, the sign language operation segmentation apparatus goes to step S2403.
By the above procedure, it is possible to detect that the operation surface has changed using the operation code vector.
[0198]
Next, a sign language recognition device (not shown) and the sign language motion segmentation device shown in FIG. 3 or 22 are provided in association with the sign language motion segmentation device so that the sign language motion segmentation device can easily perform a segmentation operation through an animation screen. A segment element guidance device for guiding a person will be described.
(Fourth embodiment)
FIG. 30 is a block diagram showing a configuration of a segment element guidance apparatus according to the fourth embodiment of the present invention.
The segment element guidance device of FIG. 30 is provided in association with a sign language recognition device (not shown) and the sign language motion segmentation device of FIG. 3 or FIG.
30, the segment element guidance device includes a recognition result input unit 3001, a segment result input unit 3002, a guidance control information generation unit 3003, an output unit 3004, and a guidance rule storage unit 3005.
[0199]
Current recognition status information is input to the recognition result input unit 3001 from the sign language recognition device connected to the recognition result input unit 3001. The current segment status information is input to the segment result input unit 3002 from the sign language action segmentation device connected to the segment result input unit 3002.
[0200]
  The recognition result input unit 3001 sends the input recognition status information to the guidance control information generation unit 3003. The segment result input unit 3002 guides the input segment status information.informationThe data is sent to the generation unit 3003. The guidance control information generation unit 3003 generates guidance control information using the guidance rules stored in the guidance rule storage unit 3005 based on the recognition status information and the segment status information, and sends the guidance control information to the output unit 3004. The output unit 3004 outputs guidance control information to a sign language animation device or the like (not shown) connected to the output unit 3004.
[0201]
Hereinafter, processing of the segment element guidance device configured as described above will be described.
FIG. 31 is a flowchart showing a process flow of the segment element guidance apparatus of FIG.
In each step of FIG. 31, the following processing is performed.
[Step S3101]
The recognition result input unit 3001 checks recognition status information input from a sign language recognition device connected to the recognition result input unit 3001.
FIG. 32 is a diagram illustrating an example of recognition status information input to the recognition result input unit 3001.
In FIG. 32, the recognition status information includes a frame number 3201 and a status flag 3202. The frame number 3201 is set to the current frame, that is, the frame number of the frame being processed by the sign language recognition device when the recognition status information is created. The status flag 3202 is set to “0” if the recognition has succeeded and “1” if the recognition has failed.
When the recognition status information is input, the recognition result input unit 3001 sends the recognition status information to the guidance control information generation unit 3003.
Thereafter, the segment element guidance apparatus proceeds to step S3102.
[0202]
[Step S3102]
The segment result input unit 3002 checks the segment status information input from the sign language action segmentation device.
FIG. 33 is a diagram illustrating an example of segment status information input to the segment result input unit 3002.
In FIG. 33, the segment status information includes a frame number 3301 and an unsegmented frame number 3302. The frame number 3301 is set to the frame number of the current frame, that is, the frame being processed by the sign language action segmentation device when the segment status information is created. The number of unsegmented frames 3302 is set to the number of frames from the last frame targeted for segmenting to the current frame.
When the segment status information is input, the segment result input unit 3002 sends the segment information to the guidance control information generation unit 3003.
Thereafter, the segment element guidance apparatus proceeds to step S3103.
[0203]
[Step S3103]
The guidance control information generation unit 3003 creates guidance control information using the guidance rules stored in the guidance rule storage unit 3005.
FIG. 34 is a diagram illustrating an example of guidance control information created by the guidance control information generation unit 3003.
In FIG. 34, the guidance control information includes a control part number 3401, a control part 3402, and a control operation 3403. In the number of control parts 3401, the number of parts to be controlled in the CG character (animation) is set. In the control part 3402, a part to be controlled in the CG character is set. In the control operation 3403, the operation of the part to be controlled is set. It should be noted that the control part 3402 and the control operation 3403 are continuously set a number of times equal to the number of parts set in the control part number 3401.
[0204]
Next, the guidance control information generation unit 3003 extracts guidance rules from the guidance rule storage unit 3005 according to the currently input recognition status information and segment status information.
FIG. 35 is a diagram illustrating an example of guidance rules stored in the guidance rule storage unit 3005.
In FIG. 35, the guidance rule includes a recognition state 3501, a non-segment frame number 3502, a control part 3503, and a control operation 3504.
For example, when the recognition status information in FIG. 32 and the segment status information in FIG. 33 are input, the recognition status and the segment status are described in the second row of the recognition status 3501 and the number of non-segment frames 3502 in FIG. It matches the condition of. Therefore, in the guidance control information of FIG. 34, “1” is set for the number of control parts 3401, “head” is set for the control part 3402, and “whisper” is set for the control action 3403.
The guidance control information generated in this way is sent to the output unit 3004.
Thereafter, the segment element guidance apparatus proceeds to step S3104.
[0205]
[Step S3104]
The output unit 3004 outputs the guidance control information sent from the guidance control information generation unit 3003 to an animation generation device or the like. At that time, the output unit 3004 transforms the guidance control information into a format required by the animation generation device or the like as necessary.
Thereafter, the segment element guidance apparatus proceeds to step S3101.
The segment element guiding method can be realized by the above procedure.
[0206]
Next, in the segment element guidance method described above, a case where the animation speed is changed according to the recognition rate of the sign language action will be described.
That is, the sign language motion recognition rate of the sign language recognition device is given to the segment element guidance device side. The segment element guidance device is provided with an animation speed adjustment device for guiding the signer so as to reduce the speed of the animation to be presented when the sign language motion recognition rate is low and thereby perform the transition motion slowly.
[0207]
FIG. 36 is a block diagram showing a configuration of an animation speed adjusting device provided in the segment element guiding device of FIG.
36, the animation speed adjustment apparatus includes a recognition result input unit 3601, a segment result input unit 3602, a speed adjustment information generation unit 3603, a speed adjustment rule storage unit 3604, and an output unit 3605.
Recognition result information from a sign language recognition device (not shown) is input to the recognition result input unit 3601. The segment result input unit 3602 receives segmentation result information from the sign language action segmentation apparatus of FIG. 3 or FIG. The speed adjustment rule storage unit 3604 stores a speed adjustment rule in advance. The speed adjustment information generation unit 3603 refers to the speed adjustment rule and controls information (animation speed adjustment information) for controlling the speed of the animation based on at least the recognition result information, preferably the recognition result information and the segmentation result information. ) Is generated.
Here, a case where the speed adjustment information generation unit 3603 generates animation speed adjustment information based on the recognition result information will be described.
[0208]
In the segment element guidance device provided with the animation speed adjustment device configured as described above, the same processing as in FIG. 31 is performed. However, the following points are different.
Step S3103 in FIG. 31 is changed as follows.
[Step S3103a]
The speed adjustment information generation unit 3603 sets 0 when nothing is set in the misrecognition flag flag_rec. When the status flag included in the recognition result information is 1, 1 is added to the misrecognition flag flag_rec. When the status flag is 0 and the misrecognition flag flag_rec> 0, 1 is subtracted from the misrecognition flag flag_rec.
[0209]
FIG. 37 is a diagram illustrating an example of a speed adjustment rule stored in the speed adjustment rule storage unit 3604.
In FIG. 37, the speed adjustment rule includes a speed adjustment amount 3701 and a condition 3702. Condition 3702 is a condition for determining the speed adjustment amount. D_spd in the condition 3702 is a speed adjustment parameter, for example, 50 is set.
The speed adjustment information generation unit 3603 obtains a speed adjustment amount d_spd corresponding to the erroneous recognition flag flag_rec while referring to the speed adjustment rule stored in the speed adjustment rule storage unit 3604.
The speed adjustment amount obtained in this way is sent to the output unit 3605.
Processing other than the above is the same as that in step S3103, and will be omitted.
[0210]
Step S3104 is changed as follows.
[Step S3104a]
The output unit 3605 sends the speed adjustment amount d_spd to an animation generation device (not shown). The animation generation apparatus adjusts the animation speed to be slower from the default animation speed Spd_def by the speed adjustment amount d_spd.
According to the above procedure, when the sign language motion recognition rate is poor, the signer can be guided to reduce the speed of the animation to be presented and thereby perform the transition motion slowly.
[0211]
Next, in the segment element guiding device (see FIG. 22; the animation speed adjusting device in FIG. 36 may or may not be provided), the television camera is prevented from being seen by the sign language person. A case where a concealing unit is provided will be described.
If the TV camera is exposed, the sign language may become nervous while being aware of the TV camera, and the sign language operation may become awkward. As a result, segmentation may not be performed correctly, and the recognition rate of the sign language recognition device may deteriorate.
[0212]
FIG. 38 is a schematic diagram illustrating an example of a configuration of a television camera concealment unit provided in the segment element guidance apparatus of FIG.
In FIG. 38, the television camera 3802 is installed facing the sign language 3801, and the monitor 3803 is installed vertically below the straight line connecting the television camera 3802 and the sign language 3801.
The TV camera concealment unit includes a half mirror 3804 that transmits light from the forward direction and reflects light from the reverse direction. The half mirror 3804 is arranged on a straight line connecting the sign language person 3801 and the TV camera 3802. In addition, this is realized by installing the monitor 3803 at a position vertically above the straight line at an angle of 45 degrees with respect to the straight line.
[0213]
That is, light from the monitor 3803 is reflected by the half mirror 3804 and reaches the sign language person 3801, so that the sign language person 3801 can see the animation displayed on the monitor 3803.
On the other hand, light traveling from the sign language 3801 to the television camera 3802 is transmitted through the half mirror 3804, whereas light traveling from the television camera 3802 to the sign language 3801 is reflected by the half mirror. Therefore, although the sign language 3801 can be photographed by the television camera 3802, the sign camera 3801 cannot see the television camera 3802 at the time of photographing.
By providing such a TV camera concealing section, it is possible to prevent the TV camera from being seen from the sign language person.
[Brief description of the drawings]
FIG. 1 is a flowchart showing a hand motion recognition method using a hand motion segmentation method according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing an example of the configuration of a computer device that implements the method of FIG. 1;
FIG. 3 is a block diagram showing a configuration of a sign language action segmentation apparatus according to a second embodiment of the present invention.
4 is a flowchart showing an example of a processing flow of the sign language action segmentation device of FIG. 3; FIG.
FIG. 5 is a diagram illustrating an example of a region code set by a body feature extraction unit 302;
6 is a diagram showing an example of segment element data stored in a segment element storage unit 305. FIG.
7 is a diagram illustrating an example of a skin color region extracted by a body feature extraction unit 302. FIG.
FIG. 8 is a diagram illustrating an example of face area information generated by the body feature extraction unit 302;
FIG. 9 is a diagram showing a feature motion determination condition by a face for the feature motion tracking unit 303 to determine a feature motion code.
10 is a diagram illustrating an example of a motion feature parameter set in a motion feature 602. FIG.
FIG. 11 is a diagram illustrating an example of determination code data created by a segment position determination unit 304;
12 is a diagram illustrating an example of a skin color area of a face extracted by a body feature extraction unit 302. FIG.
13 is a diagram showing an example of eye area information created by a body feature extraction unit 302. FIG.
FIG. 14 is a diagram showing a feature motion determination condition by an eye for the feature motion tracking unit 303 to determine a feature motion code.
15 is a diagram illustrating an example of mouth area information generated by a body feature extraction unit 302. FIG.
FIG. 16 is a diagram illustrating a condition motion determining condition by mouth for the feature motion tracking unit 303 to determine a feature motion code;
17 is a diagram illustrating an example of hand region information generated by a body feature extraction unit 302. FIG.
FIG. 18 is a diagram illustrating a feature motion determination condition based on a torso and a hand region for the feature motion tracking unit 303 to determine a feature motion code.
FIG. 19 is a diagram illustrating a feature motion determination condition based on a contact operation between a hand and a face for the feature motion tracking unit 303 to determine a feature motion code.
FIG. 20 is a diagram illustrating a feature motion determination condition based on a change in the effectiveness of a hand for the feature motion tracking unit 303 to determine a feature motion code.
FIG. 21 is a flowchart showing the flow of processing when segmentation is performed in consideration of the duration of the detected motion in the sign language motion segmentation method (see FIG. 4) by detecting the whispering motion;
FIG. 22 is a block diagram showing a configuration of a sign language action segmentation apparatus according to a third embodiment of the present invention.
FIG. 23 is a flowchart illustrating an example of a process flow of the sign language operation segmentation apparatus of FIG. 22;
24 is a flowchart showing an example of a process flow of the sign language operation segmentation device of FIG. 22;
25 is a diagram showing an example of non-segment element data stored in a non-segment element storage unit 2201. FIG.
FIG. 26 is a diagram illustrating an example of a non-segment motion feature parameter set in a non-segment motion feature 2502.
FIG. 27 is a non-segment feature motion determination condition based on the symmetry of sign language action for the feature motion tracking unit 303 to determine a feature motion code.
FIG. 28 is a diagram illustrating an example of a non-segment code determination condition based on symmetry of sign language action stored in the non-segment element storage unit 2201.
29 is a diagram illustrating an example of the same operation surface table stored in the non-segment element storage unit 2201. FIG.
30 is a block diagram showing a configuration of a segment element guidance device according to a fourth exemplary embodiment of the present invention (provided with a sign language recognition device (not shown) and a sign language action segmentation device of FIG. 3 or FIG. 22); ).
FIG. 31 is a flowchart showing a process flow of the segment element guidance apparatus of FIG. 30;
32 is a diagram illustrating an example of recognition status information input to a recognition result input unit 3001. FIG.
33 is a diagram illustrating an example of segment status information input to a segment result input unit 3002. FIG.
34 is a diagram illustrating an example of guidance control information created by a guidance control information generation unit 3003. FIG.
35 is a diagram illustrating an example of guidance rules stored in a guidance rule storage unit 3005. FIG.
36 is a block diagram showing a configuration of an animation speed adjusting device provided in the segment element guiding device of FIG. 30. FIG.
FIG. 37 is a diagram illustrating an example of a speed adjustment rule stored in a speed adjustment rule storage unit 3604.
38 is a schematic diagram illustrating an example of a configuration of a television camera concealment unit provided in the segment element guidance apparatus in FIG. 22. FIG.
[Explanation of symbols]
201 CPU
202 RAM
203 Program storage
204 Input section
205, 3004, 3605 Output section
206 Imaging unit
207 Image storage unit
208 Hand movement memory for sign language
209 Transition operation storage unit
301 Image input unit
302 Body feature extraction unit
303 Feature Motion Tracking Unit
304 Segment position determination unit
305 Segment element storage unit
2201 Non-segment element storage unit
3001, 3601 Recognition result input section
3002, 3602 Segment result input section
3003 Guide control information generation unit
3005 Guide rule storage unit
3603 Speed adjustment information generator
3604 Speed regulation rule storage unit
3801 sign language
3802 TV camera
3803 monitor
3804 half mirror

Claims (11)

Blink, which is a feature of transitional motion that appears unconsciously when transitioning from motion indicating one word to motion indicating another word, which does not appear in the user's body during the motion indicating one word, Transition action storage means, photographing means, image storage means, extraction means, detection means, and hand for preliminarily storing transition feature data describing, as action codes, characteristics relating to at least one of whispering, closing, torso stillness, and hand-face contact In a manual motion segmentation device comprising motion segmentation means, when recognizing a manual motion performed by a user, a manual motion that is executed to automatically segment the manual motion into word units or meaningful units composed of a plurality of words A segmentation method ,
Before SL photographing means, by photographing the user, an imaging step of storing in the image storage means the image data in frame units,
A transition action extracting step in which the extraction means extracts the image data from the image storage means and extracts image data corresponding to the color of the body part in which the transition action appears from the image data for each frame;
The detection means compares the image data extracted in the transition motion extraction step with each other, detects the motion of the body part in which the transition motion appears, and performs a transition motion detection step for motion coding,
The manual action segmenting means takes out the action code stored in the transition action storing step from the transition action storing means, compares the action code with the action code converted into the action code in the transition action detecting step, and A manual motion segmentation method including: a manual motion segmentation step of obtaining a time position where a transition motion appears and determining a segment position for segmenting the manual motion based on the time position. The manual motion segmentation device further describes the characteristics of the non-transitional motion that does not appear in the user's body during the transition from the motion indicating one word to the motion indicating another word, but appears in the middle of the motion indicating one word. Non-transition action storage means for storing the non -transition feature data in advance ,
The manual segmentation method is :
Before Symbol extracting means retrieves the image data from said image storage means, from the image data, and a non-transition operation extraction step of extracting image data corresponding to the body parts of appearance of non-transition operations colors per frame,
The detection means compares the image data extracted in the non-transition motion extraction step with each other, detects the motion of the body part in which the non-transition motion appears, and non-transition motion detection step for coding the motion;
The manual motion segmenting means retrieves the motion code stored in the non-transition motion storage step from the non-transition motion storage means, and compares the motion code with the motion code converted into the motion code in the non-transition motion detection step. A non-transition action segmentation step for obtaining a time position at which the non-transition action appears,
The manual motion segment according to claim 1, wherein, in the manual motion segmentation step, the segmentation is not performed at the time position where the non-transition motion appears even at the time position where the transition motion appears. Method. In the previous SL photographing step, sterically photographing the user, and stores the 3-dimensional image data in the image storage means,
A non-transition operation extracting step in which the extracting means extracts the three-dimensional image data from the image storage means and extracts three-dimensional image data corresponding to the right hand and the left hand from the three-dimensional image data;
A non-transitional motion detection step in which the detecting means detects a three-dimensional vector indicating the movement of the right hand and the left hand based on the three-dimensional image data;
Based on the three-dimensional vector, the manual motion segmentation means detects a change in the motion surface of the right hand and the motion surface of the left hand, and when neither the motion surface of the right hand or the motion surface of the left hand changes, A non-transition action segmentation step for determining that a non-transition action indicating that it is in the middle of the action to be exhibited has occurred and obtaining a time position thereof,
The manual motion segment according to claim 1, wherein, in the manual motion segmentation step, the segmentation is not performed at the time position where the non-transition motion appears even at the time position where the transition motion appears. Method.   The manual motion segment according to claim 3, wherein in the non-transitional motion segment step, a change in the motion surface of the right hand and the motion surface of the left hand is detected based on a change in a normal vector of the motion surfaces. Method. The manual motion segmentation device further includes the same motion surface table creation means and a three-dimensional code string conversion means,
The same operation surface in which the same operation surface table creation means describes combinations of three-dimensional operation codes that are included in one surface with respect to a plurality of three-dimensional operation codes corresponding to three-dimensional vectors having mutually different directions. Creating a table in advance;
The three-dimensional code sequence conversion means further comprising the step of converting the movement of the right hand and the left hand into a three-dimensional operation code sequence expressed by the plurality of three-dimensional operation codes,
The non-transitional motion segmenting step detects changes in the motion surface of the right hand and the motion surface of the left hand based on the three-dimensional motion code sequence and the same motion surface table. Manual motion segmentation method. Blink, which is a feature of transitional motion that appears unconsciously when transitioning from motion indicating one word to motion indicating another word, which does not appear in the user's body during the motion indicating one word, A computer comprising transition motion storage means for preliminarily storing transition feature data describing features relating to at least one of whispering, closing, torso stillness, and hand-face contact as motion codes;
Photographing means for photographing a user and storing the image data in the image storage means in units of frames;
Extracting the image data from the image storage means, extracting from the image data image data corresponding to the color of the body part in which the transition action appears, for each frame;
Detection means for comparing the image data extracted by the extraction means with each other, detecting the motion of the body part in which the transition motion appears, and encoding the motion;
The operation code is extracted from the transition operation storage unit, the operation code is compared with the operation code converted into the operation code by the detection unit, a time position where the transition operation appears is obtained, and based on the time position A computer-readable recording medium on which a program for functioning as a manual motion segmenting means for segmenting the manual motion is recorded. A manual motion segmentation device for automatically segmenting a manual motion into meaningful units consisting of words or a plurality of words when recognizing a manual motion performed by a user,
Blink, which is a feature of transitional motion that appears unconsciously when transitioning from motion indicating one word to motion indicating another word, which does not appear in the user's body during the motion indicating one word, Transition action storage means for preliminarily storing transition feature data describing, as an action code, characteristics relating to at least one of whispering, closing, torso stillness, and hand-face contact;
Photographing means for photographing the user and storing the image data in the image storage means in units of frames;
Extracting the image data from the image storage means, extracting from the image data image data corresponding to the color of the body part in which the transition action appears, for each frame;
Detection means for comparing the image data extracted by the extraction means with each other, detecting the motion of the body part in which the transition motion appears, and encoding the motion;
The operation code is extracted from the transition operation storage unit, the operation code is compared with the operation code converted into the operation code by the detection unit, a time position where the transition operation appears is obtained, and based on the time position A manual motion segmentation device comprising: manual motion segmentation means for segmenting the manual motion. The manual motion segmentation device further describes the characteristics of the non-transitional motion that does not appear in the user's body during the transition from the motion indicating one word to the motion indicating another word, but appears in the middle of the motion indicating one word. Non-transition action storage means for storing the non-transition feature data in advance,
The extraction means further extracts the image data from the image storage means, and extracts, from the image data, image data corresponding to the color of the body part where the non-transition motion appears for each frame,
The detection means further compares the image data corresponding to the body part in which the non-transition action appears, extracted by the extraction means, detects the movement of the body part in which the non-transition action appears, and encodes the action,
The manual motion segmenting means further takes out the motion code of the non-transition feature data from the non-transition motion storage means, and corresponds to the motion code and the body part where the non-transition motion appears as the motion code by the detection means. Is compared with an operation code indicating the motion of the image to obtain a time position at which the non-transition operation appears, and even at a time position at which the non-transition operation appears, The manual motion segmentation device according to claim 7, wherein segmentation is not performed. The photographing means photographs the user in three dimensions, stores the three-dimensional image data in the image storage means,
The extraction means further extracts the three-dimensional image data from the image storage means, extracts from the three-dimensional image data the three-dimensional image data corresponding to the right hand and the left hand in which the non-transition action appears,
The detection means further detects a three-dimensional vector indicating the movement of the right hand and the left hand based on the three-dimensional image data extracted by the extraction means,
The manual motion segmenting means further includes:
Based on the three-dimensional vector, a change in the motion surface of the right hand and the motion surface of the left hand is detected,
If neither the right-hand action plane nor the left-hand action plane changes, it is determined that a non-transition action indicating that it is in the middle of the action indicating one word has occurred, and the time position is obtained.
8. The manual motion segmentation device according to claim 7, wherein no segmentation is performed at the time position where the non-transition motion appears even at the time position where the transition motion appears.   10. The manual motion segmentation device according to claim 9, wherein the manual motion segmentation unit detects a change in the motion surface of the right hand and the motion surface of the left hand based on a change in a normal vector of the motion surfaces. . The manual segmentation device further includes:
Means for previously creating a single motion surface table that describes combinations of three-dimensional motion codes that are included in one surface with respect to a plurality of three-dimensional motion codes corresponding to three-dimensional vectors having different directions from each other; ,
Means for converting right and left hand movements into a three-dimensional motion code sequence expressed by the plurality of three-dimensional motion codes;
10. The manual motion segmentation device according to claim 9, wherein the manual motion segmentation unit detects a change in the motion surface of the right hand and the motion surface of the left hand based on the same motion surface table.

Images