JP3944841B2 - Object tracking method - Google Patents

Description

【００２３】
【数１】

【００２４】
【外２】

【００２５】
【数２】

【００２６】
【外３】

【００２７】
【数３】

【００２８】
【外４】

【００２９】
【外５】

【００３０】
【外６】

【００３１】
【数４】

【００３２】
【外７】

【００３３】
模様のモデルを３Ｄとして与えておけば、基本的にはどのような運動にも対応できる。今回の実験では、牛は飲水器までの移動と水を飲む行為が中心となっている。そのため、胴部の動きは平面的であり、寝転ぶなど、胴部が傾く動きはない。このため、胴部の白い模様の見え方の変化は、画像上の２次元的な平行移動と画像光軸周りの回転のみである。
【００３４】
また、上から見た画像からみえる頭部上面は平面的と仮定できる。このため、この頭部上面が画像面と平行になった時の観測パターンをこの平面上の２Ｄパターンと捕らえ、その見え方変化から、画像面上の平行移動、及び任意軸周りの回転を求めることができる。求める運動を以上述べたものに限定すれば、モデル点は２次元座標でよく、新たな設備を必要とすることなく、観測用の一台のカメラだけから対象の牛の白模様モデルを作成することができる長所がある。
【００３５】
【外８】

【００３６】
（初期位置の探索）
牛の位置の検出は、入力画像より牛の胴部にある白い模様を探すことから始める。牛の胴部は頭部や首部に比べて領域が大きく、動きも遅いため、検出が容易に行える。通常、牛が立っている状態や歩いている状態においては、ほとんど胴部は地面に平行であり、白い模様の見え方は図１（ａ）におけるＸ−Ｙ平面上の平行移動とＺ軸まわりの回転のみと見なせる。そのため、２次元テンプレートの照合問題として扱える。入力画像内に大きな白い領域を順に、あらかじめ登録した胴部の輪郭情報とを比較して、個体の特定と胴部の位置・姿勢（Ｚ軸周りの回転）を推測する。
【００３７】
胴部の位置・姿勢が推測できると、首部と頭部の位置の探索を行う。首部と頭部の動きうる範囲は胴部の周辺に限定されるため、その限定した範囲で首部・頭部の白い模様を探索する。図３に探索領域の例を示す。図３の白枠内にある白い領域を順に探す。大きな白い領域のある枠を頭部のある位置とし、白い領域の重心にあらかじめ登録した頭部の輪郭情報を配置する。頭部の位置が決まれば、その向きを推定する。向きは胴部と頭部の位置関係から推定する。各位置に対する頭部の向きを推定した例を図３の白枠内に点線で示す。登録した輪郭情報から変形拘束付きｄｅｆｏｒｍａｂｌｅ ｍｏｄｅｌを用いて白枠内の観測された輪郭を抽出する。その結果として得られた輪郭はトラッキングモードの初期値となる。
【００３８】
（輪郭の追随）
前述したように、一度、牛の位置・姿勢の初期状態が推定できるとその輪郭を次のフレームにおける初期値として、フレーム内の画像のエッジに適合させ、模様を抽出する一連のプロセスを繰り返すことで模様のトラッキングを行う。拘束条件付きｄｅｆｏｒｍａｂｌｅ ｍｏｄｅｌを用いて、模様を抽出することで、トラッキングしながら、同時に動きパラメタを求めることができる。各部分のトラッキングに用いる白い模様は必ずしも１つの領域でなく、同じ動きで表せる部分上の複数の領域の模様を１つの単位として扱う。
【００３９】
（実画像における実験）
牛舎内の飲水器を監視した時系列画像に対して本方法を適用した。カメラを牛舎の天井に設置することで、水を飲みに飲水器に接近する牛の動きを明瞭に観察できる。この実験では、頭部と胴部に白い模様を持つ２頭の牛についてトラッキングを行った。また、首部については前述したとおり実装していないのでトラッキングは行っていない。この実験では牛が水を飲むときなど頭部の大きな動きを含んだ時系列画像を用いており、トラッキングの評価に適している。
【００４０】
図４は探索モードにおける、トラッキングモードのための初期状態の探索過程を示している。この状態は、すでに、牛の個体識別が済み、胴部の位置および姿勢が推定された状態である。図４（ａ）の黒い実線は、拘束条件付きｄｅｆｏｒｍａｂｌｅ ｍｏｄｅｌをあらかじめ登録しておいた頭部の模様の輪郭情報で初期化したもので、画像上の配置は図４（ａ）に示したように、２値化して得た白い模様の重心を通り、かつ首の付け根との位置関係から推定した角度で配置したものである。探索モードによって得られた位置・姿勢を元にしているため、白い模様と一致していない。図４（ｂ）にある中央に黒点を配した白丸は図２に示した方法で求めた観測対象のエッジである。図４（ｃ）は５０回繰り返し処理を行った後のｄｅｆｏｒｍａｂｌｅ ｍｏｄｅｌの状態である。白い模様の輪郭を抽出していることがわかる。
【００４１】
異なる２頭の牛に対するトラッキングの結果を図５に示す。胴部の位置・姿勢は地面に対してほとんど平行運動と見なせるので、画像上に楕円で近似して表示した。また、頭部の動きは推定した動きパラメタを用いて台形を剛体として変形させて表示した。現在、首部は頭部と胴部を単純に結んだ曲線で表示しており、その動きを正しく示してはいない。この実験では、頭部の高さ（Ｚ軸方向）を算出していないが、首部の長さなどを考慮するなどして推定する方法を検討している。
【００４２】
推定した動きパラメタのうち、頭部の角度の評価方法について述べる。評価に用いた画像の例を図６に示す。図６（ａ）と（ｂ）はそれぞれ上から見た画像と横から見た画像であり、同期がとれている。Ｚ軸、Ｘ軸周りの回転角度をそれぞれ図６（ａ）（ｂ）より、マニュアルで計測し、これを正解として、本方法により上からの観測画像だけから算出された解と比較した。
【００４３】
算出される解は、図１（ｃ）に示す、頭部物体座標のＸ軸周りの解で、この画像からマニュアルで計測される世界座標系のＸ軸周りの解と厳密に等しくないが、このことがＸ軸周りの回転角度の正解に与える影響は、この例（Ｚ軸周りに約１０度回転）では、Ｘ軸周りの回転角度が４５度のとき、約０．５度であるため無視している。また、Ｙ軸周りの回転については正解を得る画像が得られなかったので評価していない。
【００４４】
図７に牛Ｂの時系列画像における頭部の角度の変化を示す。示した角度は図１（ｃ）に示した座標軸のＸ軸とＺ軸周りの回転の変化でＺ軸周りの変化を’○’、Ｘ軸周りの変化を’□’で示した。図８には本方法によって推定した角度と評価用に上から見た画像と同時に横から見た画像から直接計測した角度との差（推定誤差）を示す。図８で示した角度は図７と同様にＸ軸とＺ軸周りの回転角度である。Ｚ軸周りの変化を’○’、’●’、Ｘ軸周りの変化を’＋’、’×’で示した。Ｘ軸周りの回転角度が大きくなると白い模様が非常に小さくなるため、トラッキングができなくなる。
【００４５】
実験の結果、Ｘ軸周りの回転角度が約７０度を越えるとトラッキングが難しくなった。この場合には、白い模様が小さく見える状態になるとトラッキングを中止し、「探索」モードにして再度、頭部の位置を探すこととする。Ｘ軸周りの角度が０〜３０度付近で大きな誤差を示しているのはこの角度での見え方の変化が小さいために起こると考えられる。しかしながら、その誤差は多くても１５度となっている。全体的には、Ｘ軸周りの平均誤差は６．９度、Ｚ軸周りの平均誤差は５．５度であった。これらの結果は牛の動きを認識するためには許容できる範囲と考える。
【００４６】
（まとめ）
本願発明者は拘束条件付きdeformable modelを用いて、牛体表面にある白い模様をトラッキングする事で、牛の動きを監視する方法をここに提案する。上から見た画像を用いることで、牛の動きの推定に必要な胴部、首部、頭部のそれぞれの動きを重なりなしに明瞭に観測できる。胴部と頭部、各部分の白い模様の見え方とその動きの関係から得られる結果を用いて、動きパラメタを推定しながら、良好にトラッキングが行えた。
【００４７】
今回の実験において、もっとも動きの大きい頭部のトラッキングが可能なことを確認した。頭部の姿勢の推定誤差はおおむね１０度以下であった。今後は、あらかじめ登録する輪郭情報を３Ｄ化し、かつ、首部の曲がりを考慮したより詳細な動きパラメタを得るように本方法を拡張していく予定である。あわせて、現在、得られる動きパラメタを用いて、監視装置への適用を進めてゆきたい。
【００４８】
参考文献
［１］ Ｄ． Ｍａｇｅｅ ａｎｄ Ｒ． Ｂｏｙｌｅ： ”Ｆｅａｔｕｒｅ ｔｒａｃｋｉｎｇ ｉｎ ｒｅａｌ ｗｏｒｌｄ ｓｃｅｎｅｓ （ｏｒ ｈｏｗ ｔｏ ｔｒａｃｋ ａ ｃｏｗ）”， Ｉｎ ＩＥＥ Ｃｏｌｌｏｑｕｉｕｍ ｏｎ Ｍｏｔｉｏｎ Ａｎａｌｙｓｉｓ ａｎｄ Ｔｒａｃｋｉｎｇ， ｐｐ． ２／１２／７， １９９９
［２］ Ｍ． Ｋａｓｓ， Ａ． Ｗｉｔｋｉｎ， ａｎｄ Ｄ． Ｔｅｒｚｏｐｏｕｌｏｓ： ”Ｓｎａｋｅｓ： ａｃｔｉｖｅ ｃｏｎｔｏｕｒ ｍｏｄｅｌｓ”， Ｉｎ Ｉｎｔｅｒｎａｔｉｏｎａｌ Ｊｏｕｒｎａｌ ｏｆ Ｃｏｍｐｕｔｅｒ Ｖｉｓｉｏｎ， Ｖｏｌ． １， Ｎｏ． ４， ｐｐ． ３２１３３１， １９８８
［３］ Ｔ． Ｆ． Ｃｏｏｔｅｓ， Ｃ． Ｊ． Ｔａｙｌｏｒ， Ｄ． Ｈ． Ｃｏｏｐｅｒ， ａｎｄ Ｊ． Ｇｒａｈａｍ： ”Ａｃｔｉｖｅ Ｓｈａｐｅ Ｍｏｄｅｌｓ Ｔｈｅｉｒ Ｔｒａｉｎｉｎｇ ａｎｄ Ａｐｐｌｉｃａｔｉｏｎ”， Ｉｎ Ｃｏｍｐｕｔｅｒ Ｖｉｓｉｏｎ ａｎｄ Ｉｍａｇｅ Ｕｎｄｅｒｓｔａｎｄｉｎｇ， Ｖｏｌ． ６１， Ｎｏ． １， ｐｐ． ３８５９， １９９５
［４］ 口田ら、“肥育牛舎内における非接触型自動増体管理システムの構築”、ホクサイテック財団北海道の研究開発、ｐｐ．１１９１２４， ２０００
上述の説明は学術論文として発表されているので、説明が技術的に高度であり、通常の技術者にとっては難解と思われる個所がある。そこで、上記説明の補足説明を行う。
【００４９】
本発明に係わる対象物の動作追跡方法は、生体に代表されるような複雑な動きをする対象物を、比較的単純な動きで記述できる各部分に分割し、各部分の動きを並行して追跡することにより全体の動きを頑健、高速に把握する。
【００５０】
各部分の基本状態における形状情報から、その部分がとり得る動きによって観測画像上での見え方がどのように変形しうるかを定式化して表し、その式に従ってしか変形しない変形可能輪郭モデルを用いて、観測画像上の対象部分の抽出およびその動きの特定を同時に行う。
【００５１】
また、本実施形態の方法を適用した対象物の動作追跡システムは、時系列画像を監視入力するカメラと、上述した画像処理を実行する画像処理装置により構築することができる。画像処理装置は、上記カメラにより撮影した対象物の時系列画像の中の対象物部分を比較的単純な動きに記述できる部分に分割して、その１つの部分を基本形状部分として、監視入力画像中の基本形状部分に関する模様候補を検出する。
【００５２】
次に画像処理装置は、対象物毎に予め登録された基本形状部分の模様に基づき、前記検出された基本形状部分の模様により基本形状部分の位置及び姿勢の検出を行なう。
【００５３】
さらに画像処理装置は、前記基本形状部分の位置及び姿勢を元に、その他の部分の位置及び姿勢の候補を算出し、それに基づき、各部分に関する模様の検出を行ない、拘束条件付き変形可能モデルを配置する。
【００５４】
画像処理装置は、連続する時系列画像で、各模様をその模様毎の拘束条件付き変形可能モデルで並列に追跡することにより、各部分の位置及び姿勢を追跡しかつ算出する。また、画像処理装置は、各部分の動きを並行して追跡することにより全体の動きを把握する。
【００５５】
本発明を、牛の動作の追跡・把握のために実施した例を用いて説明する。図９に、画像処理装置により牛の動作を追跡・認識する処理手順を示す。
【００５６】
ステップ1:
牛舎内の天井に設置したTVカメラで時系列画像を取得する。図６（ａ）はその一例である。
【００５７】
ステップ2:
監視入力画像中に、胴体部分の白い模様候補が検出されれば、その模様をもとに、どの個体の牛がどのような姿勢で存在するかの検出を画像処理装置により行なう。白い模様の画像またはその特徴は、あらかじめ各牛ごとに画像処理装置に登録しておくが、その前処理については、以下の通りである。
【００５８】
[前処理]
まず、撮影された対象物の時系列画像を画像処理装置の表示画面に表示し、手動操作により位置指定により対象物の画像を比較的単純な動きに記述できる部分で分割する。牛を上部から観測する場合、胴体、首、頭の部分に分けることによって、胴体部の動きは(そのしなりを無視すれば)剛体運動、頭部の動きも剛体運動で記述できる。首部に関しては、剛体運動とその首軸に添った曲がりの運動で記述できる。
【００５９】
各牛の白い模様の輪郭情報を点の座標集合として頭部、首部、胴部の各部分ごとにあらかじめ画像処理装置に登録しておく。登録するデータは、頭部の模様のある鼻面が天井と平行になるような頭の状態の時に、同じ観測カメラより入力した画像より取得する。
【００６０】
ステップ3:
図３に示すように、胴体画像の位置・姿勢を元に、首部画像、頭部画像の位置・姿勢の候補を算出し、それに基づき、各部分画像上の白い模様の検出を行ない、拘束条件付き変形可能モデルを配置する。配置した各モデルを用いて、時系列画像中の白い模様を追跡し、その見え方の変化から各部分の位置・姿勢を決定する。この部分の処理の詳細に関しては、後述の[拘束条件付き変形可能モデル] で述べる。
【００６１】
図４（ａ）の黒い実線は、頭部の模様の輪郭情報に基づく拘束条件付き変形可能モデルで、首の付け根との位置関係から推定した角度で配置したものである。図４（ｂ）にある中央に黒点を配した白丸は図５に示した方法で求めた観測対象のエッジである。具体的に、変形可能モデル上の各点(図中黒丸)の法線方向（図中矢印方向）に、画像上で明るさの大きく変化する点(図中白丸)を探索する。図４ｃ）は収束後の状態で、白い模様の輪郭を抽出していることがわかる。この状態のモデルのパラメータより、頭部の位置・姿勢がわかる。
【００６２】
ステップ4:
連続する時系列画像で、各模様をその模様毎の拘束条件付き変形可能モデルで並列に追跡することにより、各部分の位置・姿勢を追跡・算出する。図５はこのようにして、頭部と胴体の模様を追跡して得られた、牛の姿勢変化を表している。
【００６３】
【発明の効果】
従来、単一の観測画像からは動物の活動のような複雑な動きの追跡、把握は、不可能であった。本発明は、対象物の動きをその動きの拘束に従った変形可能輪郭モデルを用いることにより、ビデオカメラのような簡易な一台の撮像装置で追跡・把握できるようになる。これにより、たとえば、家畜の日常動作を休みなく自動監視する技術が実現され、一日のうち数回の震えにしか現れないような初期病状や、突発的な異常事態の発生などの検出が可能となる。
【００６４】
また、画像特徴として対象物の模様を使用することで、撮影画像の中に他の物体が含まれていても対象物を確実に検出することができる。
【００６５】
また、対象物の複数の部位ごとにその動きを追跡することにより動作の追跡精度がさらに高まる。
【００６６】
さらに複数の部位が重りにくい角度から撮像装置よる撮影を行うことにより、画像追跡に使用するデータの欠落を防止し、さらに追跡精度を高めることができる。
【図面の簡単な説明】
【図１】（ａ）、（ｂ）、（ｃ）は対象物の部位の位置を検出するための座標系を示す説明図である。
【図２】拘束条件付変形可能モデルを説明するための説明図である。
【図３】探索領域を説明するための説明図である。
【図４】（ａ）、（ｂ）、（ｃ）は探索モードにおける、トラッキングモードのための初期状態の探索過程を説明するための説明図である。
【図５】（ａ）および（ｂ）はトラッキングの結果を示す説明図である。
【図６】（ａ）および（ｂ）は対象物の撮影画像を模式的に示す説明図である。
【図７】時系列画像における頭部の角度の変化を示す説明図である。
【図８】推定した角度と評価用に上から見た画像と同時に横から見た画像から直接計測した角度との差（推定誤差）を示す説明図である。
【図９】本発明の実施形態の処理手順を示す説明図である。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a motion tracking method for an object in which the movement of the object in the moving image is analyzed by the image processing device based on the moving image (time-series image) of the object captured by the imaging device.
[0002]
[Prior art]
In monitoring livestock such as cattle, it is important to check the movement daily to detect abnormalities due to accidents and diseases. Abnormal behaviors such as trembling and other abnormal behaviors such as few movements are difficult to judge at each moment, and it is necessary to analyze them from continuous motion information. To that end, it is important to continue monitoring for a certain amount of time, and it is required to construct a monitoring system that automatically collects these data, detects abnormalities at an early stage, and notifies dairy farmers and the like. Magee et al. [1] described below are investigating methods of detecting abnormalities by monitoring the walking of cattle.
[0003]
[Problems to be solved by the invention]
In this method, it is necessary to acquire the contour of the whole cow body and to acquire a plurality of contours that change due to movement. When used for motion estimation, an outline corresponding to the motion to be estimated has to be acquired, which is not always convenient.
[0004]
Accordingly, an object of the present invention is to provide a motion tracking method for an object that can track (track) the movement of a cow using a time-series image that can be easily obtained on a daily basis.
[0005]
[Means for Solving the Problems]
In order to achieve such an object, the invention according to claim 1 shoots an object to be tracked by an imaging device , and the time series in the image processing device based on a time series image obtained by the imaging. In the object motion tracking method for tracking the movement of the object in the image, an image indicating the basic shape of the object is registered in the image processing device in advance, and the image processing device stores the basic shape in the image. Based on the image shown, a relational expression between a plurality of parameters representing the motion of the object and a change in appearance on the image caused by the motion is determined in advance, a deformable model with constraints is arranged based on the relationship, and the imaging Tracking the movement of the object by tracking the image features of the time-series image obtained by the apparatus with the deformation model with the constraint condition, and tracking the movement of the object In order, the contour pattern of the plurality of sites as the basic shape the object is used, the equation is more parameters vector q representing the movement of the object, the appearance of the contour on the image When the change is expressed by the coordinate position u of the contour image of the pattern of the part, it is expressed in the form of a function of u = F (q). In the step of tracking the movement of the object, the pattern of the plurality of parts The movement of the contour of the contour is tracked as the motion of the deformable model with constraint conditions, and the relational expression u = F (q) is obtained from the coordinate position u of the contour image representing the change in the appearance of the contour on the tracked image. The vector q is calculated as a plurality of parameters representing the motion of the object.
[0007]
According to a second aspect of the present invention, in the object tracking method according to the first aspect, in the step of tracking the movement of the target, the movement of the position and orientation is tracked for each part of the target. It is characterized by.
[0008]
A third aspect of the present invention, in the operation method of tracking an object according to claim 2, wherein the imaging by the imaging device, wherein a plurality of said sites is performed from the most overlap difficult angles.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described in detail with reference to the drawings.
[0010]
The present embodiment aims to establish a method for tracking cow movements using time series images that can be easily obtained on a daily basis. In this task, the following things make the problem difficult.
1) Complicate how the cows overlap.
2) In natural situations, the contrast between the cow and the background may not be very large.
3) Tracking non-rigid motion.
[0011]
First, in order to prevent the cows from overlapping, a camera was installed on the ceiling in the barn, and an image viewed from above as shown in FIG. 6 (a) was used. In this image, the movement of the legs cannot be observed well, but the main parts other than the legs can be observed well, and sufficient information can be obtained to identify the abnormal movement. The image seen from above is used in a study of estimating the weight of a cow by Kuchida et al. In the image viewed from above, the background is the ground or the floor, and in general, it is often dark.
[0012]
On the other hand, a white pattern has a high contrast with a dark background and can be easily extracted. In Japan, more than 90% of dairy cows and 50% of all domestic cattle are Holstein, and the body has a black and white pattern. This time, we used a white pattern that was easy to extract without using the contour of the cow body to track the movement of the cow. Although this method cannot be applied to cows without a pattern, it can be said that many cows have a pattern and are practical enough.
[0013]
Since the cow body is non-rigid, the change in the appearance of the cow body on the image is complicated, and it is difficult to obtain the motion parameters. However, a method for solving this problem by learning the appearance in advance has been proposed [see Documents 1 and 3].
[0014]
On the other hand, in the cow image seen from above, the body, head and neck of the cow can be separated relatively easily. These three parts can be observed without overlapping. We considered the movement of each part independently, and estimated the movement of the cow easily without the need for learning. In particular, the movement of the head and torso can be treated as a rigid body except for the detailed movement, and the movement of the cow can be expressed simply. The change in the appearance of the white pattern on each part is brought about by the movement, so the movement of the cow is estimated by observing the change in the appearance.
[0015]
Next, consider extraction of a white pattern. A deformable model such as Snakes [see Reference 2 described later] has been proposed as one method for extracting a boundary from an arbitrary region from an observed image. However, in general, it is difficult to match Snakes' “hardness” and “smoothness” parameters to the target to be extracted.
[0016]
As a method for solving the problems related to parameter adjustment, Active Shape Models [see Reference 3] have been proposed. In this method, an object to be deformed is extracted by learning by giving an example of a deformed contour in advance. Since the deformable model is deformed according to the deformation rule obtained by learning, it is effective for searching for a similar contour in a new image. Although it may be a problem that learning is necessary, it is effective when it has characteristics that are robust against noise and overlap, and when it is difficult or unknown to formulate deformation.
[0017]
The method of the present inventor uses the fact that the movement of the cow and the resulting change in the appearance of the cow pattern are expressed by a simple relational expression, and obtains the movement parameter from the change. Therefore, once a pattern is registered, a motion parameter can be obtained without requiring a learning process. By using a deformable model with a constraint that changes the appearance of a registered pattern, it is possible to make the pattern extraction and tracking robust. In this way, it is possible to obtain a motion parameter that gives a change in the appearance while tracking the pattern, and as a result, it is possible to estimate the motion of the cow.
[0018]
(Monitoring method)
Time series images are acquired with a camera installed on the ceiling in the barn. FIG. 6A is an example of the time-series image. The movement of the cow is estimated by tracking a white pattern on each of the head, neck and torso. Each part of the cow is described in the respective coordinate system shown in FIG. By integrating the movement information of each part divided into three, the movement of the whole cow can be grasped in detail. It is also possible to know the abnormality with only a part of the movements such as “tremor”.
[0019]
The contour information of the white pattern of each cow is registered in advance for each part of the head, neck, and trunk as a set of points. The data to be registered is obtained from the image seen from above. Currently, pattern registration is performed manually, but we are considering a method of automatic registration.
[0020]
Monitoring can be divided into two modes, “search mode” and “tracking mode”. In the search mode, the position of the cow is obtained from the input image, the individual is identified, and the initial position and posture necessary for tracking are obtained. In the tracking mode, the target is tracked from the initial position obtained in the search mode. These processes are performed for each individual part, and even if there are a plurality of cows on the input image, they can be processed in parallel.
[0021]
(Deformable model with deformation constraint)
[0022]
[Outside 1]

[0023]
[Expression 1]

[0024]
[Outside 2]

[0025]
[Expression 2]

[0026]
[Outside 3]

[0027]
[Equation 3]

[0028]
[Outside 4]

[0029]
[Outside 5]

[0030]
[Outside 6]

[0031]
[Expression 4]

[0032]
[Outside 7]

[0033]
If you give the pattern model as 3D, you can basically handle any movement. In this experiment, cattle are mainly moving to the drinking water and drinking water. Therefore, the movement of the torso is flat, and there is no movement in which the torso tilts, such as lying down. For this reason, the change in the appearance of the white pattern on the body is only two-dimensional translation on the image and rotation around the image optical axis.
[0034]
Further, it can be assumed that the upper surface of the head seen from the image seen from above is planar. For this reason, the observation pattern when the upper surface of the head is parallel to the image plane is regarded as a 2D pattern on this plane, and the translation on the image plane and the rotation about an arbitrary axis are obtained from the change in appearance. be able to. If the desired movement is limited to the one described above, the model point may be a two-dimensional coordinate, and a white pattern model of the target cow is created from only one camera for observation without requiring new equipment. There is an advantage that can be.
[0035]
[Outside 8]

[0036]
(Search for initial position)
The detection of the position of the cow is started by searching for a white pattern on the body of the cow from the input image. The cow's torso has a larger area than the head and neck and moves slowly, so detection is easy. Normally, when the cow is standing or walking, the torso is almost parallel to the ground, and the white pattern is visible on the XY plane in FIG. It can be considered only the rotation of. Therefore, it can be treated as a two-dimensional template matching problem. A large white area in the input image is sequentially compared with body contour information registered in advance, and the identification of the individual and the position / posture (rotation around the Z axis) of the body are estimated.
[0037]
When the position / posture of the torso can be estimated, the positions of the neck and head are searched. Since the range in which the neck and head can move is limited to the periphery of the torso, the white pattern of the neck and head is searched for in the limited range. FIG. 3 shows an example of the search area. The white areas in the white frame in FIG. 3 are searched in order. A frame with a large white area is set as the position of the head, and the head outline information registered in advance is arranged at the center of gravity of the white area. If the position of the head is determined, the direction is estimated. The direction is estimated from the positional relationship between the torso and the head. An example in which the orientation of the head with respect to each position is estimated is indicated by a dotted line in the white frame of FIG. An observed contour in a white frame is extracted from the registered contour information using a deformable model with deformable constraints. The resulting contour is the initial value for the tracking mode.
[0038]
(Following contour)
As described above, once the initial state of the cow's position / posture can be estimated, the outline is used as the initial value in the next frame, and the process is repeated to match the edges of the image in the frame and extract the pattern. To track the pattern. By extracting a pattern using a deformable model with constraint conditions, it is possible to simultaneously obtain a motion parameter while tracking. The white pattern used for tracking each part is not necessarily one area, but a pattern of a plurality of areas on the part that can be expressed by the same movement is handled as one unit.
[0039]
(Experiment with real images)
The method was applied to time-series images of a drinking device in a barn. By installing the camera on the ceiling of the cowshed, it is possible to clearly observe the movement of the cow approaching the drinker for drinking water. In this experiment, two cows with white patterns on the head and torso were tracked. Further, since the neck is not mounted as described above, tracking is not performed. This experiment uses time-series images that include large head movements, such as when a cow drinks water, and is suitable for tracking evaluation.
[0040]
FIG. 4 shows an initial state search process for the tracking mode in the search mode. This state is a state where the individual identification of the cow has already been completed and the position and posture of the torso have been estimated. The black solid line in FIG. 4A is obtained by initializing the deformable model with constraint conditions with the contour information of the head pattern registered in advance, and the arrangement on the image is as shown in FIG. 4A. In addition, the white pattern obtained by binarization passes through the center of gravity and is arranged at an angle estimated from the positional relationship with the base of the neck. Since it is based on the position and orientation obtained by the search mode, it does not match the white pattern. A white circle with a black dot in the center in FIG. 4B is an edge to be observed obtained by the method shown in FIG. FIG. 4C shows the state of a deformable model after 50 iterations. It can be seen that the outline of the white pattern is extracted.
[0041]
The tracking results for two different cattle are shown in FIG. Since the position and posture of the torso can be regarded as almost parallel motion with respect to the ground, it is displayed as an ellipse on the image. In addition, the movement of the head was displayed by deforming the trapezoid as a rigid body using the estimated movement parameters. Currently, the neck is displayed as a curve that simply connects the head and torso, and the movement is not shown correctly. In this experiment, the height of the head (in the Z-axis direction) is not calculated, but a method of estimation is considered by considering the length of the neck and the like.
[0042]
Among the estimated motion parameters, a method for evaluating the angle of the head will be described. An example of an image used for evaluation is shown in FIG. 6A and 6B are an image viewed from above and an image viewed from the side, respectively, and are synchronized. The rotation angles around the Z-axis and the X-axis were manually measured from FIGS. 6 (a) and 6 (b), respectively, and this was taken as the correct answer and compared with the solution calculated from only the observed image from above by this method.
[0043]
The calculated solution is a solution around the X axis of the head object coordinates shown in FIG. 1C, and is not exactly equal to the solution around the X axis of the world coordinate system manually measured from this image. The effect of this on the correct answer of the rotation angle around the X axis is about 0.5 degrees when the rotation angle around the X axis is 45 degrees in this example (rotation about 10 degrees around the Z axis). Ignoring. Further, the rotation around the Y axis is not evaluated because an image for obtaining a correct answer was not obtained.
[0044]
FIG. 7 shows a change in the angle of the head in the time series image of cow B. The angle shown is a change in rotation around the Z-axis by a change in rotation around the X-axis and the Z-axis of the coordinate axis shown in FIG. FIG. 8 shows the difference (estimation error) between the angle estimated by this method and the angle measured directly from the image viewed from the side at the same time as the image viewed from above for evaluation. The angle shown in FIG. 8 is the rotation angle around the X axis and the Z axis as in FIG. Changes around the Z axis are indicated by “◯” and “●”, and changes around the X axis are indicated by “+” and “×”. When the rotation angle around the X-axis increases, the white pattern becomes very small and tracking becomes impossible.
[0045]
As a result of the experiment, tracking became difficult when the rotation angle around the X axis exceeded about 70 degrees. In this case, when the white pattern appears to be small, the tracking is stopped and the “search” mode is set to search for the position of the head again. It is considered that the reason why the angle around the X axis shows a large error in the vicinity of 0 to 30 degrees is due to the small change in appearance at this angle. However, the error is at most 15 degrees. Overall, the average error around the X axis was 6.9 degrees and the average error around the Z axis was 5.5 degrees. These results are considered acceptable for recognizing cow movement.
[0046]
(Summary)
The present inventor proposes a method for monitoring the movement of a cow by tracking a white pattern on the surface of the cow using a deformable model with a constraint condition. By using the image seen from above, it is possible to clearly observe the movements of the trunk, neck, and head necessary for estimating the movement of the cow without overlapping. Using the results obtained from the relationship between the movement of the body and head and the white pattern on each part and the movement, we were able to track well while estimating the motion parameters.
[0047]
In this experiment, it was confirmed that the head with the most movement can be tracked. The estimation error of the head posture was approximately 10 degrees or less. In the future, we plan to expand this method to 3D contour information registered in advance and to obtain more detailed motion parameters considering the bending of the neck. At the same time, we would like to proceed with the application to monitoring devices using the motion parameters that are currently available.
[0048]
Reference [1] Magee and R.M. Boyle: “Feature tracking in real world scenes (or how to track a cow)”, In IEEE Colloquium on Motion Analysis and Tracking, pp. 2/12/7, 1999
[2] M.M. Kass, A.M. Witkin, and D.W. Terzopoulos: “Snakes: active control models”, International Journal of Computer Vision, Vol. 1, no. 4, pp. 321331, 1988
[3] T.M. F. Cootes, C.I. J. et al. Taylor, D.D. H. Cooper, and J.M. Graham: "Active Shape Models Training and Application", In Computer Vision and Image Understanding, Vol. 61, no. 1, pp. 3859, 1995
[4] Kuchida et al., “Construction of a non-contact automatic body weight gain management system in fattening cattle barns”, research and development of Hokusaitec Foundation Hokkaido, pp. 119124, 2000
Since the above explanation is published as an academic paper, the explanation is technically advanced, and there are places that are considered difficult for ordinary engineers. Therefore, a supplementary explanation of the above explanation will be given.
[0049]
The object motion tracking method according to the present invention divides an object that moves in a complex manner as typified by a living body into parts that can be described by relatively simple movements, and moves the movements of the parts in parallel. By tracking, the whole movement is robust and grasped at high speed.
[0050]
From the shape information in the basic state of each part, formulate how the appearance on the observed image can be deformed by the movement that the part can take, and use a deformable contour model that deforms only according to that formula The target part on the observed image is extracted and its movement is specified at the same time.
[0051]
An object motion tracking system to which the method of the present embodiment is applied can be constructed by a camera that monitors and inputs a time-series image and an image processing apparatus that executes the above-described image processing. The image processing apparatus divides an object part in a time-series image of an object photographed by the camera into parts that can be described in a relatively simple motion, and uses that one part as a basic shape part as a monitoring input image A pattern candidate relating to the inside basic shape portion is detected.
[0052]
Next, the image processing apparatus detects the position and orientation of the basic shape portion based on the detected pattern of the basic shape portion based on the pattern of the basic shape portion registered in advance for each object.
[0053]
Furthermore, the image processing apparatus calculates candidates for the positions and orientations of the other parts based on the positions and orientations of the basic shape parts, and based on that, detects a pattern for each part, and creates a deformable model with constraint conditions. Deploy.
[0054]
The image processing apparatus tracks and calculates the position and orientation of each part by tracking each pattern in parallel with a deformable model with a constraint condition for each pattern in continuous time-series images. The image processing apparatus grasps the entire movement by tracking the movement of each part in parallel.
[0055]
The present invention will be described using an example implemented for tracking and grasping the behavior of a cow. FIG. 9 shows a processing procedure for tracking and recognizing the behavior of the cow by the image processing apparatus.
[0056]
step 1:
A time series image is acquired with a TV camera installed on the ceiling in the barn. FIG. 6A shows an example.
[0057]
Step 2:
If a white pattern candidate for the body part is detected in the monitoring input image, the image processing apparatus detects which individual cow is present in what posture based on the pattern. The white pattern image or its features are registered in advance in the image processing apparatus for each cow. The preprocessing is as follows.
[0058]
[Preprocessing]
First, a time-series image of the photographed object is displayed on the display screen of the image processing apparatus, and the image of the object is divided into portions that can be described in a relatively simple motion by specifying a position by manual operation. When observing a cow from the upper part, the movement of the torso can be described as rigid body movement and the movement of the head can also be described as rigid body movement by dividing it into the body, neck and head parts. As for the neck, it can be described by rigid body motion and bending motion along the neck.
[0059]
The contour information of the white pattern of each cow is registered in advance in the image processing apparatus for each part of the head, neck, and trunk as a coordinate set of points. Data to be registered is obtained from an image input from the same observation camera when the head is in a state where the nose surface with the head pattern is parallel to the ceiling.
[0060]
Step 3:
As shown in FIG. 3, based on the position / posture of the torso image, the candidates for the position / posture of the neck image and the head image are calculated, and based on this, the white pattern on each partial image is detected, and the constraint condition Place a deformable model. Using each placed model, the white pattern in the time-series image is tracked, and the position / posture of each part is determined from the change in appearance. Details of the processing of this part will be described later in [Deformable model with constraint conditions].
[0061]
The black solid line in FIG. 4A is a deformable model with constraint conditions based on the contour information of the pattern of the head, and is arranged at an angle estimated from the positional relationship with the base of the neck. A white circle with a black dot in the center in FIG. 4B is an edge to be observed obtained by the method shown in FIG. Specifically, a point (white circle in the figure) where the brightness changes greatly on the image is searched in the normal direction (arrow direction in the figure) of each point (black circle in the figure) on the deformable model. FIG. 4c) shows that the outline of the white pattern is extracted in the state after convergence. The position and orientation of the head can be determined from the model parameters in this state.
[0062]
Step 4:
By tracking each pattern in parallel with a deformable model with constraint conditions for each pattern in a continuous time-series image, the position and orientation of each part are tracked and calculated. FIG. 5 shows the change in the posture of the cow obtained in this way by tracing the pattern of the head and the torso.
[0063]
【The invention's effect】
Conventionally, it has been impossible to track and grasp complicated movements such as animal activities from a single observation image. According to the present invention, by using a deformable contour model in accordance with the movement constraint, the movement of an object can be tracked and grasped by a simple imaging device such as a video camera. This enables, for example, a technology to automatically monitor the daily movement of livestock, enabling detection of initial medical conditions that appear only during a few tremors of the day, or sudden abnormal situations. It becomes.
[0064]
Further, by using the pattern of the object as an image feature, the object can be reliably detected even if another object is included in the captured image.
[0065]
In addition, by tracking the movement of each part of the object, the tracking accuracy of the action is further enhanced.
[0066]
Furthermore, by performing imaging using an imaging device from an angle at which a plurality of parts are unlikely to overlap, it is possible to prevent data used for image tracking from being lost and to further improve tracking accuracy.
[Brief description of the drawings]
FIGS. 1A, 1B, and 1C are explanatory views showing a coordinate system for detecting the position of a part of an object.
FIG. 2 is an explanatory diagram for explaining a deformable model with constraint conditions;
FIG. 3 is an explanatory diagram for explaining a search area;
FIGS. 4A, 4B, and 4C are explanatory diagrams for explaining an initial state search process for the tracking mode in the search mode; FIGS.
FIGS. 5A and 5B are explanatory diagrams showing tracking results. FIGS.
FIGS. 6A and 6B are explanatory diagrams schematically showing a captured image of an object.
FIG. 7 is an explanatory diagram showing a change in head angle in a time-series image.
FIG. 8 is an explanatory diagram showing a difference (estimation error) between an estimated angle and an angle directly measured from an image viewed from the side simultaneously with an image viewed from above for evaluation;
FIG. 9 is an explanatory diagram illustrating a processing procedure according to the embodiment of this invention.

Claims (3)

An object motion tracking method of capturing an object to be tracked by an imaging device and tracking the motion of the object in the time-series image in an image processing device based on a time-series image obtained by the imaging In
An image showing the basic shape of the object is registered in the image processing apparatus in advance,
The image processing apparatus includes:
Based on the image showing the basic shape, a relational expression between a plurality of parameters representing the movement of the object and a change in appearance on the image caused by the movement is determined in advance, and a deformable model with constraints is arranged based on the relational expression. And
Performing the step of tracking the motion of the object by tracking the image characteristics of the time-series image obtained by the imaging device with the deformation model with a constraint condition;
In order to track the movement of the object, contours of patterns of a plurality of parts of the object are used as the basic shape, and the relational expression includes a plurality of parameters representing the movement of the object as a vector q, A step of tracking the movement of the object expressed in the form of a function of u = F (q) when the change in the appearance of the contour on the image is expressed by the coordinate position u of the contour image of the pattern of the part. Then, the movement of the contour of the pattern of the plurality of parts is tracked as the movement of the deformable model with the constraint condition, and the relationship is determined from the coordinate position u of the contour image representing the change in the appearance of the contour on the tracked image. A method for tracking an object motion , comprising: calculating the vector q as a plurality of parameters representing the motion of the object by an equation u = F (q) .   The object tracking method according to claim 1, wherein in the step of tracking the movement of the object, the movement of the position and orientation of each part of the object is tracked. Behavior tracking method.   3. The motion tracking method for an object according to claim 2, wherein the imaging by the imaging device is performed from an angle at which the plurality of parts are least likely to overlap.

Images