JP5452307B2

JP5452307B2 - Tracking device

Info

Publication number: JP5452307B2
Application number: JP2010066672A
Authority: JP
Inventors: 功青山; 洋志亀田; 親房野中
Original assignee: Mitsubishi Electric Corp
Current assignee: Mitsubishi Electric Corp
Priority date: 2010-03-23
Filing date: 2010-03-23
Publication date: 2014-03-26
Anticipated expiration: 2030-03-23
Also published as: JP2011196940A

Description

この発明は、連続観測された画像中の目標を追尾する追尾装置に関する。 The present invention relates to a tracking device that tracks a target in continuously observed images.

連続観測された複数フレームの画像より特定の目標物を追尾する追尾装置では、目標検出処理により最新フレームの入力画像から目標物の位置の検出を行った後、検出された目標物の位置と過去の目標物の動きとに基づいて、次の入力画像における目標物の位置や速度等の状態を予測する追尾処理を実行する。
追尾処理による目標物の状態予測は、目標検出処理による目標物の検出位置に基づいて行われるので、目標検出処理の検出精度が追尾処理の予測精度に影響を与える。
従来の目標検出処理としては、目標物のテンプレート画像を用いて入力画像とのマッチングを行い、最もテンプレート画像に類似している領域(すなわち、テンプレート画像との相関が最も高い領域)を目標物が存在する領域として抽出する方式が一般的である。
例えば、下記特許文献１や特許文献２では、テンプレートマッチングによる目標検出に関する発明が開示されている。 In a tracking device that tracks a specific target from continuously observed images of multiple frames, the target position is detected from the input image of the latest frame by target detection processing, and then the detected target position and past Based on the movement of the target, a tracking process for predicting a state such as the position and speed of the target in the next input image is executed.
Since the target state prediction by the tracking process is performed based on the detection position of the target by the target detection process, the detection accuracy of the target detection process affects the prediction accuracy of the tracking process.
In conventional target detection processing, a target template image is used for matching with an input image, and the target is the region that is most similar to the template image (i.e., the region that has the highest correlation with the template image). A method of extracting as an existing area is common.
For example, the following Patent Document 1 and Patent Document 2 disclose inventions related to target detection by template matching.

特開２００５−１１４５８８号公報JP 2005-114588 A 特開２００９−８７２３８号公報JP 2009-87238 A

テンプレートマッチングによる目標検出における問題点は、複雑背景下において、背景にテンプレートと類似した領域が存在することによる誤検出の発生や、画像上の追尾目標の大きさや向きの時間的な変化にテンプレートが適応できないことによる誤検出の発生などが挙げられる。
上記特許文献１では、背景データベースを保持し、データベース内に保持する背景と一致する画像の背景を除去することで背景の誤検出を低減している。しかし、画像背景が背景データベースに無い場合、背景除去の効果は低く、誤検出が発生する可能性がある。
特許文献２では、エッジ情報とエッジ周辺の輝度情報をテンプレートとして用いることで背景の誤検出を防いでいる。しかし、追尾目標の形状や大きさが変化した場合、テンプレートと入力画像内の追尾目標に差が生じ、誤検出が発生する可能性がある。 The problems with target detection by template matching are that the template may be affected by the occurrence of false detection due to the presence of a region similar to the template in the background under a complex background, or the temporal change in the size and orientation of the tracking target on the image. For example, the occurrence of false detection due to inability to adapt.
In the above-mentioned Patent Document 1, a background database is held, and background misdetection is reduced by removing the background of an image that matches the background held in the database. However, when the image background is not in the background database, the effect of background removal is low, and erroneous detection may occur.
In Patent Document 2, background information is prevented from being erroneously detected by using edge information and luminance information around the edge as a template. However, when the shape or size of the tracking target changes, a difference may occur between the template and the tracking target in the input image, and erroneous detection may occur.

さらに、テンプレートマッチングは、目標の位置を１点の座標として求めることはできるが、最新画像上の目標とテンプレートが完全に一致することは無いので、目標画素あるいは目標領域を正確に抽出することはできない。このため、精度良く目標画素あるいは目標領域を抽出するためには別途精度の高い特徴抽出等の画像処理が必要となる。
さらに、最新画像から抽出した目標領域をテンプレートの更新に反映する場合は、目標領域の抽出精度が低いと、テンプレートマッチングの精度が低下してしまうという問題もある。 Furthermore, in template matching, the target position can be obtained as the coordinates of one point. However, since the target on the latest image and the template do not completely match, it is not possible to accurately extract the target pixel or target area. Can not. For this reason, in order to extract a target pixel or target area with high accuracy, image processing such as feature extraction with high accuracy is required separately.
Furthermore, when the target area extracted from the latest image is reflected in the template update, there is a problem that the accuracy of template matching is lowered if the extraction accuracy of the target area is low.

この発明は、上記のような課題を解決するためになされたものであり、最新画像上の目標位置の検出および目標画素あるいは目標領域の抽出を行う際に、テンプレートマッチングや特徴抽出等の画像処理を用いることなく、背景画素の誤抽出を低減し、追尾目標の画像上の大きさや向きの変化に影響を受けない高精度の追尾装置を提供することを目的とする。 The present invention has been made to solve the above-described problems, and performs image processing such as template matching and feature extraction when detecting a target position on the latest image and extracting a target pixel or target area. An object of the present invention is to provide a high-accuracy tracking device that reduces erroneous extraction of background pixels and is not affected by changes in the size and orientation of a tracking target image without using the.

この発明は、観測センサで同時に観測された同一画角の距離画像と強度画像がフレーム単位で入力され、最新フレーム画像の追尾点情報を入力とし該追尾点の距離画像より求める３次元位置座標と強度画像より得られる強度値を観測ベクトルとし、追尾フィルタにより次フレームにおける追尾目標の状態予測値を出力する追尾フィルタ装置と、前記追尾フィルタ装置からの追尾目標の状態予測値に基づく最新フレーム画像における残差２次形式と、追尾予測位置との距離差分と、追尾予測強度との強度差分により各画素が目標を構成する画素である確からしさを表す目標画素指標を算出し、さらに目標画素指標から最新フレーム画像における目標を構成する目標領域を抽出し、前記目標領域に含まれる画素から前記追尾フィルタ装置への新たな前記追尾点情報を算出して出力する目標検出装置と、を備え、前記追尾フィルタ装置は、追尾点情報の該追尾点の距離画像より求める３次元位置座標と強度画像より得られる強度値を観測ベクトルとし、追尾フィルタにより次フレームにおける追尾点の３次元位置座標、３次元速度、強度の７次元の状態予測値を求め、前記目標検出装置は、前記追尾フィルタ装置が出力する前記状態予測値と距離画像より求める各画素の３次元位置と強度画像より得られる各画素の強度値とにより求める各画素の残差２次形式値と、前記状態予測値の追尾予測位置と前記各画素の３次元位置との距離差分値と、前記状態予測値の追尾予測強度と前記各画素の強度値との強度差分値とを求める第１の処理手段と、前記残差２次形式値と前記距離差分値と前記強度差分値とをそれぞれ正規化する第２の処理手段と、距離差分正規化値と強度差分正規化値にそれぞれ重み係数を掛け、前記残差２次形式正規化値との積和により各画素の目標画素指標を算出し、前記目標画素指標の平均と標準偏差より目標画素検出閾値を定め、目標画素指標値が前記目標画素検出閾値を超える画素を目標画素として抽出し、さらに抽出した前記目標画素の分布より目標領域を抽出する第３の処理手段と、抽出した前記目標領域の中心位置を最新フレーム画像における追尾点とし、前記追尾点情報として追尾フィルタ装置へ入力する第４の処理手段と、を含む、ことを特徴とする追尾装置にある。 According to the present invention, a distance image and an intensity image of the same angle of view simultaneously observed by an observation sensor are input in units of frames, and tracking point information of the latest frame image is input, and a three-dimensional position coordinate obtained from the distance image of the tracking point In the latest frame image based on the state prediction value of the tracking target from the tracking filter device that outputs the state prediction value of the tracking target in the next frame by the tracking filter, using the intensity value obtained from the intensity image as an observation vector A target pixel index representing the probability that each pixel is a pixel constituting a target is calculated based on an intensity difference between the distance difference between the residual quadratic format, the tracking prediction position, and the tracking prediction intensity. A target area constituting a target in the latest frame image is extracted, and a new pixel from the pixel included in the target area to the tracking filter device is extracted. And a target detection device calculates and outputs the tracking point information, Bei example, said tracking filter apparatus, the intensity values obtained from the three-dimensional position coordinates and intensity images obtained from the distance image of該追tail point of the tracking point information Using the tracking filter as the observation vector, the three-dimensional position coordinates of the tracking point in the next frame, the three-dimensional velocity, and the seven-dimensional state prediction value of the intensity are obtained, and the target detection device outputs the state prediction value output by the tracking filter device And the residual quadratic form value of each pixel obtained from the three-dimensional position of each pixel obtained from the distance image and the intensity value of each pixel obtained from the intensity image, the tracking predicted position of the state prediction value, and 3 of each pixel A first processing means for obtaining a distance difference value with respect to a dimension position, an intensity difference value between the tracking prediction intensity of the state prediction value and the intensity value of each pixel, the residual quadratic form value, and the distance difference Value and above A second processing means for normalizing the degree difference value, a distance difference normalized value and an intensity difference normalized value, respectively multiplied by a weighting factor, and each pixel by a product sum of the residual quadratic form normalized value A target pixel index is calculated, a target pixel detection threshold is determined from the average and standard deviation of the target pixel index, a pixel whose target pixel index value exceeds the target pixel detection threshold is extracted as a target pixel, and the extracted target Third processing means for extracting a target area from the distribution of pixels, and fourth processing means for setting the center position of the extracted target area as a tracking point in the latest frame image and inputting the tracking point information to the tracking filter device; , Including a tracking device.

この発明では、テンプレートマッチングや特徴抽出等の画像処理を用いることなく、背景画素の誤抽出を低減し、追尾目標の画像上の大きさや向きの変化に影響を受けない高精度の追尾装置を提供できる。 In this invention, without using image processing such as template matching or feature extraction, a high-accuracy tracking device is provided that reduces erroneous extraction of background pixels and is not affected by changes in the size or orientation of the tracking target image. it can.

この発明の実施の形態１による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 1 of this invention. この発明の実施の形態２による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 2 of this invention. この発明の実施の形態３による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 3 of this invention. この発明の実施の形態４による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 4 of this invention. この発明の実施の形態５による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 5 of this invention. この発明の実施の形態６による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 6 of this invention. この発明の実施の形態７による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 7 of this invention. この発明の実施の形態８による追尾装置の機能ブロック図である。It is a functional block diagram of the tracking apparatus by Embodiment 8 of this invention.

以下、この発明による追尾装置を各実施の形態に従って図面を用いて説明する。なお、各実施の形態において、同一もしくは相当部分は同一符号で示し、重複する説明は省略する。 A tracking device according to the present invention will be described below with reference to the drawings according to each embodiment. In each embodiment, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

実施の形態１．
図１はこの発明の実施の形態１による追尾装置の機能ブロック図である。図１において、この発明における追尾装置は、目標検出装置１００と追尾フィルタ装置２００により構成され、同時に取得された、同一画角の距離画像と強度画像が入力として与えられる。ここで、距離画像とは観測面各点の距離および反射強度を取得する観測センサにより生成される、センサから観測面各点までの距離を画素値とする画像であり、強度画像とは前記観測センサにより生成される観測面各点の反射強度を画素値とする画像である。 Embodiment 1 FIG.
FIG. 1 is a functional block diagram of a tracking device according to Embodiment 1 of the present invention. In FIG. 1, the tracking device according to the present invention includes a target detection device 100 and a tracking filter device 200, and a distance image and an intensity image having the same angle of view, which are simultaneously acquired, are given as inputs. Here, the distance image is an image generated by an observation sensor that obtains the distance and reflection intensity of each point on the observation surface, and has a pixel value as the distance from the sensor to each point on the observation surface. It is an image which uses the reflection intensity of each point of the observation surface generated by the sensor as a pixel value.

目標検出装置１００は、追尾フィルタ装置２００が出力する追尾目標の状態予測値に基づいて最新画像上の目標画素および目標領域を抽出し、追尾フィルタ装置２００の入力となる最新画像上の追尾点を決定する。目標検出装置１００は、初期目標領域情報取得部１０１、追尾点決定部１０２、残差算出部１０３、ゲート処理部１０４、残差正規化部１０５、距離差分算出部１０６、距離差分正規化部１０７、強度差分算出部１０８、強度差分正規化部１０９、目標画素指標算出部１１０、目標画素抽出部１１１を有している。 The target detection device 100 extracts a target pixel and a target region on the latest image based on the state prediction value of the tracking target output from the tracking filter device 200, and sets a tracking point on the latest image to be input to the tracking filter device 200. decide. The target detection apparatus 100 includes an initial target area information acquisition unit 101, a tracking point determination unit 102, a residual calculation unit 103, a gate processing unit 104, a residual normalization unit 105, a distance difference calculation unit 106, and a distance difference normalization unit 107. , An intensity difference calculation unit 108, an intensity difference normalization unit 109, a target pixel index calculation unit 110, and a target pixel extraction unit 111.

追尾フィルタ装置２００は、目標検出装置１００より入力される追尾点に基づき追尾処理を実行し、次に入力される画像上の追尾点の位置、速度、強度の予測を行い、追尾処理結果として追尾目標の状態予測値を目標検出装置１００に出力する。追尾フィルタ装置２００は、追尾処理部２０１、遅延処理部２０２を有している。 The tracking filter device 200 executes a tracking process based on the tracking point input from the target detection device 100, predicts the position, speed, and intensity of the tracking point on the next input image, and tracks as the tracking processing result. The target state predicted value is output to the target detection device 100. The tracking filter device 200 includes a tracking processing unit 201 and a delay processing unit 202.

なお、実線矢印は処理またはデータの流れ、破線矢印はデータの流れを示す。 A solid line arrow indicates a process or data flow, and a broken line arrow indicates a data flow.

初期目標領域情報取得部１０１は、この発明の追尾装置とは別の装置において実行される、初期入力画像中の追尾目標を含む最小の矩形領域抽出結果を、初期目標領域として取得する。
追尾点決定部１０２は、抽出された目標領域内に含まれる画素から追尾点を算出し、追尾フィルタ装置２００に算出した追尾点を出力する。 The initial target area information acquisition unit 101 acquires, as an initial target area, a minimum rectangular area extraction result including a tracking target in the initial input image, which is executed by an apparatus different from the tracking apparatus of the present invention.
The tracking point determination unit 102 calculates a tracking point from the pixels included in the extracted target area, and outputs the calculated tracking point to the tracking filter device 200.

残差算出部１０３は、追尾フィルタ装置２００の処理結果である追尾目標の状態予測値と、最新の入力画像の各画素値との残差２次形式の算出を行う。
ゲート処理部１０４は、χ(カイ)２乗分布に基づく閾値(これをゲートサイズと呼ぶ)を用いて、残差算出部１０３において算出した残差２次形式の値がゲートサイズ以下となる画素を、ゲート内画素として抽出する。
残差正規化部１０５は、ゲート処理部１０４において抽出されたゲート内画素を対象として、残差算出部１０３において算出された残差２次形式の値を、最小値が１、最大値が０となるよう正規化を行う。 The residual calculation unit 103 calculates a quadratic residual format between the state prediction value of the tracking target that is the processing result of the tracking filter device 200 and each pixel value of the latest input image.
The gate processing unit 104 uses a threshold value based on a χ (chi) square distribution (referred to as a gate size), and a pixel whose residual quadratic value calculated by the residual calculation unit 103 is equal to or smaller than the gate size. Are extracted as in-gate pixels.
The residual normalization unit 105 targets the in-gate pixel extracted by the gate processing unit 104 as a target and calculates the residual quadratic form value calculated by the residual calculation unit 103 with a minimum value of 1 and a maximum value of 0. Normalization is performed so that

距離差分算出部１０６は、追尾フィルタ装置２００の処理結果である追尾目標の状態予測値を用いて、センサ位置から追尾点予測位置までの距離(これを追尾点予測距離と呼ぶ)を算出し、さらに、最新の距離画像における各画素距離と追尾点予測距離との距離差分値を算出する。
距離差分正規化部１０７は、ゲート処理部１０４において抽出されたゲート内画素を対象として、距離差分算出部１０６において算出した距離差分値を、最小値が１、最大値が０となるように正規化を行う。 The distance difference calculation unit 106 calculates the distance from the sensor position to the tracking point prediction position (referred to as the tracking point prediction distance) using the state prediction value of the tracking target that is the processing result of the tracking filter device 200. Further, a distance difference value between each pixel distance and the tracking point predicted distance in the latest distance image is calculated.
The distance difference normalization unit 107 normalizes the distance difference value calculated by the distance difference calculation unit 106 for the in-gate pixel extracted by the gate processing unit 104 so that the minimum value is 1 and the maximum value is 0. To do.

強度差分算出部１０８は、追尾フィルタ装置２００の処理結果である追尾目標の状態予測値を用いて、最新の強度画像における各画素強度値と追尾点予測強度値との差分を算出する。
強度差分正規化部１０９は、ゲート処理部１０４において抽出されたゲート内画素を対象として、強度差分算出部１０８において算出した強度差分値を、最小値が１、最大値が０となるように正規化を行う。 The intensity difference calculation unit 108 calculates the difference between each pixel intensity value and the tracking point predicted intensity value in the latest intensity image using the tracking target state predicted value that is the processing result of the tracking filter device 200.
The intensity difference normalization unit 109 normalizes the intensity difference value calculated by the intensity difference calculation unit 108 so that the minimum value is 1 and the maximum value is 0 for the in-gate pixel extracted by the gate processing unit 104. To do.

目標画素指標算出部１１０は、残差正規化部１０５において算出した残差２次形式正規化値と、距離差分正規化部１０７において算出した距離差分正規化値と、強度差分正規化部１０９において算出した強度差分正規化値とを用いて、各画素が追尾目標を構成するか否かの判定指標となる目標画素指標を算出する。
目標画素抽出部１１１は、目標画素指標算出部１１０において算出された各目標画素指標値より各画素が目標を構成するか否かを判断し、目標を構成すると判断した画素を目標画素として抽出する。さらに、抽出した全ての目標画素を含む最小矩形領域を目標領域として抽出する。 The target pixel index calculation unit 110 includes a residual quadrature normalization value calculated by the residual normalization unit 105, a distance difference normalization value calculated by the distance difference normalization unit 107, and an intensity difference normalization unit 109. Using the calculated intensity difference normalized value, a target pixel index that is a determination index of whether or not each pixel constitutes a tracking target is calculated.
The target pixel extraction unit 111 determines whether each pixel constitutes a target based on each target pixel index value calculated by the target pixel index calculation unit 110, and extracts the pixel determined to constitute the target as a target pixel. . Further, a minimum rectangular area including all the extracted target pixels is extracted as a target area.

追尾処理部２０１は、追尾点決定部１０２が出力する追尾点を入力とし、カルマンフィルタ等の追尾フィルタを用いて、追尾点の３次元直交座標上の位置および速度と、追尾点の強度の予測を行う。
遅延処理部２０２は、最新の画像が目標検出装置１００に入力されると同時に、追尾処理部２０１の処理結果である追尾目標の状態予測値が目標検出装置１００に入力されるように、遅延処理を行う。 The tracking processing unit 201 receives the tracking point output from the tracking point determination unit 102 as an input, and uses a tracking filter such as a Kalman filter to predict the position and speed of the tracking point on the three-dimensional orthogonal coordinates and the intensity of the tracking point. Do.
The delay processing unit 202 performs delay processing so that the latest image is input to the target detection device 100 and, at the same time, the state prediction value of the tracking target that is the processing result of the tracking processing unit 201 is input to the target detection device 100. I do.

ここで、この発明の追尾装置は、演算処理部(ＣＰＵ)、記憶部(ＲＯＭ、ＲＡＭ、ハードディスク等)および信号入出力部をハードウェアとして備えるコンピュータ(図示せず)により構成することができる。記憶部には、各機能ブロック１０１〜２０２の機能を実現するためのプログラムが格納され、また追尾処理に必要な後述する観測センサの画角情報等の各種データが予め格納される。 Here, the tracking device of the present invention can be configured by a computer (not shown) provided with hardware as an arithmetic processing unit (CPU), a storage unit (ROM, RAM, hard disk, etc.) and a signal input / output unit. The storage unit stores a program for realizing the functions of the functional blocks 101 to 202, and stores various data such as later-described observation sensor angle-of-view information necessary for the tracking process.

また追尾処理に必要な各種データが外部機器から逐次入力される。例えば、後述する観測センサの３次元直交座標上の位置情報に関し、観測センサが固定の場合、観測センサの３次元直交座標上の位置は一定であるが、観測センサが移動体上に搭載されて移動する場合には可変となる。そこで観測センサが移動する場合には(追尾装置も一緒に移動する構成の場合もある)、観測センサと一緒に移動するＧＰＳ装置等の外部機器から観測センサの３次元直交座標上の位置情報が逐次入力される(いずれも図示省略)。 Various data necessary for the tracking process are sequentially input from an external device. For example, regarding the position information of the observation sensor on the three-dimensional orthogonal coordinates described later, when the observation sensor is fixed, the position of the observation sensor on the three-dimensional orthogonal coordinates is constant, but the observation sensor is mounted on the moving body. When moving, it becomes variable. Therefore, when the observation sensor moves (the tracking device may move together), the position information on the three-dimensional orthogonal coordinates of the observation sensor is obtained from an external device such as a GPS device that moves with the observation sensor. Input sequentially (both not shown).

次に、実施の形態１による追尾装置の一連の動作について説明する。まず、初期目標領域情報取得部１０１が、外部の別装置により抽出された初期画像(第１フレーム画像)内の目標を含む最小矩形領域の、左上および右下の画像座標(画像座標とは、画像上の画素位置を意味する)を初期目標領域情報として取得する。
次に、追尾点決定部１０２が、初期目標領域情報取得部１０１が取得した初期目標領域情報より、目標領域の中心点の画像座標および強度値を、追尾点情報とし、追尾処理部２０１に対して出力する。 Next, a series of operations of the tracking device according to the first embodiment will be described. First, the initial target area information acquisition unit 101 has upper left and lower right image coordinates (image coordinates are the minimum rectangular area including the target in the initial image (first frame image) extracted by another external device. Means the pixel position on the image) as the initial target area information.
Next, the tracking point determination unit 102 uses the image coordinates and the intensity value of the center point of the target region as tracking point information based on the initial target region information acquired by the initial target region information acquisition unit 101, and performs the tracking processing unit 201. Output.

次に、追尾処理部２０１が、距離画像と観測センサの３次元直交座標上の位置情報と観測センサの画角情報から、距離画像各画素の３次元直交座標上の位置(以降、３次元位置と呼ぶ)を算出し、追尾点決定部１０２より入力される追尾点情報の画像座標も同様にして３次元位置に変換する。
さらに、追尾処理部２０１は、距離画像と観測センサの３次元位置情報と観測センサの画角情報とから算出した追尾点の３次元位置と強度画像より取得する追尾点の強度を観測ベクトルとして、例えばカルマンフィルタからなる追尾フィルタを用いて、次の観測時刻(すなわち、次フレーム画像取得時刻)における追尾点の、３次元位置と３次元速度と強度の予測を行う(状態予測値)。
なお、カルマンフィルタは一般的に用いられるフィルタであるので、ここでは処理の説明は省略する。
次に、遅延処理部２０２が、次フレーム画像が目標検出装置１００に入力されるのとほぼ同時に、追尾処理部２０１の処理結果である追尾目標の状態予測値が目標検出装置１００に入力されるよう、１フレーム分遅延処理を行う。 Next, the tracking processing unit 201 determines the position of each pixel of the distance image on the 3D orthogonal coordinate (hereinafter referred to as the 3D position) from the position information on the 3D orthogonal coordinate of the distance image and the observation sensor and the angle of view information of the observation sensor. The image coordinates of the tracking point information input from the tracking point determination unit 102 are similarly converted into a three-dimensional position.
Further, the tracking processing unit 201 uses the three-dimensional position of the tracking point calculated from the distance image, the three-dimensional position information of the observation sensor, and the angle-of-view information of the observation sensor and the intensity of the tracking point acquired from the intensity image as an observation vector. For example, a tracking filter composed of a Kalman filter is used to predict the three-dimensional position, three-dimensional velocity, and intensity of the tracking point at the next observation time (that is, the next frame image acquisition time) (state prediction value).
Since the Kalman filter is a commonly used filter, description of the processing is omitted here.
Next, the delay processing unit 202 inputs the tracking target state predicted value, which is the processing result of the tracking processing unit 201, to the target detection device 100 almost simultaneously with the next frame image being input to the target detection device 100. In this way, delay processing for one frame is performed.

次に、最新フレームの距離画像および強度画像と、追尾目標の状態予測値が残差算出部１０３に入力され、残差算出部１０３は、まず距離画像と観測センサの３次元直交座標上の位置情報と観測センサの画角情報から、距離画像各画素の３次元位置を算出する。
さらに、下記式(１)を用いて入力画像各画素に対する残差２次形式を算出する。なお、下記式(１)における残差共分散行列Ｓ_{ｋ｜ｋ−１}は、追尾処理部２０１内の処理(例えばカルマンフィルタ処理)において算出される。 Next, the distance image and the intensity image of the latest frame and the state prediction value of the tracking target are input to the residual calculation unit 103. The residual calculation unit 103 first positions the distance image and the observation sensor on the three-dimensional orthogonal coordinates. The three-dimensional position of each pixel in the distance image is calculated from the information and the angle of view information of the observation sensor.
Further, a residual quadratic form for each pixel of the input image is calculated using the following formula (1). Note that the residual covariance matrix S _{k | k−1} in the following equation (1) is calculated in the processing in the tracking processing unit 201 (for example, Kalman filter processing).

位置および速度の予測値に加えて、強度の予測値を用いて上記式(１)に示す残差２次形式を算出することにより、追尾する目標の形状が残差２次形式の分布に現れる。残差２次形式は、値が小さいほど予測値に近いことを示している。この発明では、この特徴を利用して追尾目標の画素抽出を行う。 By calculating the residual quadratic form shown in the above equation (1) using the predicted value of intensity in addition to the predicted value of position and velocity, the shape of the target to be tracked appears in the distribution of the residual quadratic form . The residual quadratic form indicates that the smaller the value, the closer to the predicted value. In the present invention, pixel extraction of the tracking target is performed using this feature.

次に、ゲート処理部１０４が、自由度が４(残差２次形式の算出は３次元位置と強度を合わせた４次元ベクトルを用いているため自由度を４とする)の場合の、信頼度(下側確率)が９５％となるχ２乗分布の値をゲートサイズｄとして、下記式(２)を満たす画素をゲート内と判定する(以降、ゲート内と判定した画素をゲート内画素と記述する)。なお、使用する信頼度の値は９５％に限定されるものではなく、適用先に応じて変更しても良い。
ゲート処理部１０４によるゲート内画素の抽出により以降の処理の対象をゲート内画素に限定し、処理負荷の軽減と目標画素の誤抽出の低減を図ることができる。 Next, the reliability in the case where the gate processing unit 104 has a degree of freedom of 4 (the degree of freedom is 4 because the calculation of the residual quadratic form uses a four-dimensional vector combining the three-dimensional position and intensity). A pixel satisfying the following formula (2) is determined to be in the gate with the value of the chi-square distribution having a degree (lower probability) of 95% as the gate size d. Describe). Note that the reliability value to be used is not limited to 95%, and may be changed according to the application destination.
By extracting the in-gate pixels by the gate processing unit 104, the target of subsequent processing is limited to the in-gate pixels, so that the processing load can be reduced and the erroneous extraction of the target pixel can be reduced.

次に、残差正規化部１０５が、ゲート処理部１０４において求めたゲート内画素を対象として、残差算出部１０３において算出した残差２次形式を、最小値が１、最大値が０となるように正規化を行う。正規化値の算出式を下記式(３)に示す。下記式(３)は、上記式(１)とは逆に、値が大きいほど追尾予測値との差が小さいことを示している。 Next, the residual normalization unit 105 sets the residual quadratic form calculated by the residual calculation unit 103 for the in-gate pixel obtained by the gate processing unit 104 as a minimum value of 1 and a maximum value of 0. Normalization is performed so that The calculation formula of the normalized value is shown in the following formula (3). The following equation (3) indicates that the larger the value is, the smaller the difference from the tracking prediction value is, contrary to the above equation (1).

残差正規化部１０５より得られる残差２次形式正規化値を直接用いて目標画素の抽出を行った場合、追尾目標周辺に値の高い画素が残りやすいため目標画素の誤抽出が発生する可能性が高い。そこで、距離画像と強度画像を用いて目標画素の強調処理を行い、目標画素の誤抽出の低減を図る。 When the target pixel is extracted directly using the residual quadratic normalized value obtained from the residual normalization unit 105, a target pixel is erroneously extracted because a pixel having a high value tends to remain around the tracking target. Probability is high. Thus, the target pixel is enhanced using the distance image and the intensity image to reduce erroneous extraction of the target pixel.

まず、距離差分算出部１０６が、最新フレームの距離画像と観測センサの３次元直交座標上の位置情報と観測センサの画角情報から距離画像各画素の３次元位置を算出し、センサ位置から追尾予測位置までの３次元直交空間上の距離と、センサ位置から各画素位置までの３次元直交空間上の距離との差分絶対値を求め(以降、センサ距離差分値と記載する)、さらに、追尾予測位置と各画素位置との３次元直交空間上の距離をセンサ距離差分値に加えた値を、各画素の距離差分として算出する。 First, the distance difference calculation unit 106 calculates the three-dimensional position of each pixel of the distance image from the distance image of the latest frame, the position information of the observation sensor on the three-dimensional orthogonal coordinates, and the angle of view information of the observation sensor, and tracks from the sensor position. Obtain the absolute difference between the distance in the three-dimensional orthogonal space to the predicted position and the distance in the three-dimensional orthogonal space from the sensor position to each pixel position (hereinafter referred to as the sensor distance difference value). A value obtained by adding the distance in the three-dimensional orthogonal space between the predicted position and each pixel position to the sensor distance difference value is calculated as the distance difference of each pixel.

単純にセンサ距離差分値のみを用いると、センサ位置を中心とした半径がセンサ距離差分値の球面付近であれば追尾予測位置と離れている画素であっても強調される可能性があるため、３次元直交空間上で追尾予測点に近い画素が強調されるように、センサ距離差分値に追尾予測位置と画素位置の距離を加えている。 If only the sensor distance difference value is used simply, even if the radius around the sensor position is near the spherical surface of the sensor distance difference value, even a pixel that is far from the tracking predicted position may be emphasized. The distance between the predicted tracking position and the pixel position is added to the sensor distance difference value so that pixels close to the predicted tracking point in the three-dimensional orthogonal space are emphasized.

下記式(４)に第ｋフレーム画素(ｉ，ｊ)における距離差分の算出式を示す。下記式(４)より得られる値が小さいほど、すなわち距離差分値が小さいほど、直交３次元空間上での追尾予測位置との距離が近いことを表している。 The following formula (4) shows a formula for calculating the distance difference at the k-th frame pixel (i, j). The smaller the value obtained from the following equation (4), that is, the smaller the distance difference value, the closer the distance to the predicted tracking position in the orthogonal three-dimensional space.

次に、距離差分正規化部１０７が、ゲート処理部１０４により抽出したゲート内画素を処理対象として、距離差分算出部１０６が算出した距離差分値を、最小値が１、最大値が０となるように正規化を行う。正規化値の算出式を下記式(５)に示す。下記式(５)は、上記式(４)とは逆に値が大きいほど追尾予測位置に近いことを示している。また、残差正規化部１０５と同様に処理対象をゲート内画素に限定することで処理負荷の軽減と目標画素の誤抽出の低減を図っている。 Next, the distance difference normalization unit 107 sets the in-gate pixel extracted by the gate processing unit 104 as a processing target, and the distance difference value calculated by the distance difference calculation unit 106 has a minimum value of 1 and a maximum value of 0. Normalization is performed as follows. The calculation formula of the normalized value is shown in the following formula (5). The following equation (5) indicates that the larger the value is, the closer to the tracking prediction position, as opposed to the above equation (4). Further, similarly to the residual normalization unit 105, the processing target is limited to the in-gate pixels, thereby reducing the processing load and reducing the erroneous extraction of the target pixel.

次に、強度差分算出部１０８が、最新フレームの強度画像における各画素強度値と、追尾予測強度値との差分を算出する。下記式(６)に第ｋフレーム画素(ｉ，ｊ)における強度差分の算出式を示す。下記式(６)より得られる値が小さいほど、すなわち強度差分値が小さいほど、追尾予測強度値との差が小さいことを表している。 Next, the intensity difference calculation unit 108 calculates the difference between each pixel intensity value in the intensity image of the latest frame and the predicted tracking intensity value. Equation (6) below shows a formula for calculating the intensity difference at the k-th frame pixel (i, j). The smaller the value obtained from the following equation (6), that is, the smaller the intensity difference value, the smaller the difference from the tracking predicted intensity value.

次に、強度差分正規化部１０９が、ゲート処理部１０４により抽出したゲート内画素を処理対象として、強度差分算出部１０８が算出した強度差分値を、最小値が１，最大値が０となるように正規化を行う。正規化値の算出式を下記式(７)に示す。下記式(７)は、上記式(６)とは逆に値が大きいほど追尾予測強度に近いことを示している。また、残差正規化部１０５と同様に処理対象をゲート内画素に限定することで処理負荷の軽減と目標画素の誤抽出の低減を図っている。 Next, the intensity difference normalization unit 109 sets the in-gate pixel extracted by the gate processing unit 104 as a processing target, and the intensity difference value calculated by the intensity difference calculation unit 108 has a minimum value of 1 and a maximum value of 0. Normalization is performed as follows. The calculation formula of the normalized value is shown in the following formula (7). The following equation (7) indicates that the larger the value is, the closer to the tracking prediction strength, contrary to the above equation (6). Further, similarly to the residual normalization unit 105, the processing target is limited to the in-gate pixels, thereby reducing the processing load and reducing the erroneous extraction of the target pixel.

次に、目標画素指標算出部１１０が、残差正規化部１０５の出力である残差２次形式正規化値と、距離差分正規化部１０７の出力である距離差分正規化値と、強度差分正規化部１０９の出力である強度差分正規化値とにより、目標画素の抽出指標となる目標画素指標を算出する。目標画素指標は、重み係数をαとし、残差２次形式正規化値を距離差分正規化値により強調した値と、残差２次形式正規化値を強度差分正規化値により強調した値の和として求める。目標画素指標の算出式を下記式(８)に示す。なお、重み係数αはパラメータであり、事前に与えられるものとする。下記式(８)より得られる値は、上記式(１)より得られる残差２次形式に対して追尾予測値に近い画素の値を強調したものであり、値が高いほど目標画素である可能性が高いと判断する。 Next, the target pixel index calculation unit 110 outputs a residual quadratic normalization value that is an output of the residual normalization unit 105, a distance difference normalization value that is an output of the distance difference normalization unit 107, and an intensity difference. A target pixel index that is an extraction index of the target pixel is calculated based on the intensity difference normalized value that is the output of the normalization unit 109. The target pixel index is a value in which the weighting factor is α, a value obtained by emphasizing the residual quadratic normalized value with the distance difference normalized value, and a value obtained by enhancing the residual quadratic normalized value with the intensity difference normalized value. Find as sum. The formula for calculating the target pixel index is shown in the following formula (8). The weight coefficient α is a parameter and is given in advance. The value obtained from the following equation (8) is obtained by emphasizing the pixel value close to the tracking prediction value with respect to the residual quadratic form obtained from the above equation (1), and the higher the value, the more the target pixel. Judge that the possibility is high.

次に、目標画素抽出部１１１が、目標画素指標算出部１１０が算出した目標画素指標の分布に対して閾値処理を行い目標画素の抽出を行う。目標画素抽出に用いる閾値は、目標画素指標の平均および目標画素指標の標準偏差を用いて、下記式(９)を満たす目標画素指標値をもつ画素を目標画素とする。なお、下記式(９)においてβは重み係数であり、事前に与えられるものとする。 Next, the target pixel extraction unit 111 extracts a target pixel by performing threshold processing on the distribution of the target pixel index calculated by the target pixel index calculation unit 110. As a threshold used for target pixel extraction, a pixel having a target pixel index value satisfying the following formula (9) is set as a target pixel using an average of the target pixel index and a standard deviation of the target pixel index. In the following equation (9), β is a weighting factor and is given in advance.

上記式(９)の判定式により抽出された画素には目標以外の画素が含まれている可能性がある。目標画素は密集して存在すると考えられるので、目標画素抽出部１１１は、抽出した目標画素より孤立点の除去を行う。さらに目標画素の縦横各方向における周辺画素分布を求め、目標領域の切り出しを行う。 There is a possibility that pixels other than the target are included in the pixels extracted by the determination formula of the above formula (9). Since the target pixels are considered to exist densely, the target pixel extraction unit 111 removes isolated points from the extracted target pixels. Further, the peripheral pixel distribution in the vertical and horizontal directions of the target pixel is obtained, and the target area is cut out.

次に、追尾点決定部１０２が、初期目標領域における追尾点の決定と同様に、目標領域の中心点の画像座標および強度値を、追尾点情報とし、追尾処理部２０１に対して出力する。 Next, the tracking point determination unit 102 outputs the image coordinates and the intensity value of the center point of the target region as tracking point information to the tracking processing unit 201 in the same manner as the tracking point determination in the initial target region.

上記のような実施の形態１による追尾装置によれば、追尾目標の位置および速度に加え、強度の予測を行うことにより、追尾予測値に基づく残差２次形式に目標の形状が現れる。この残差２次形式を、追尾予測値に基づく距離差分値と強度差分値とを用いて強調処理を行うことにより、追尾目標がより強調され、追尾目標を構成する目標画素の抽出および目標領域の抽出を容易に行うことが可能となる。
すなわち、位置と速度に加えて強度の予測を行い、追尾処理と目標検出処理を連携させ、追尾予測値に基づく目標画素および目標領域の抽出を行うことにより、テンプレートマッチング等に代表される画像処理を用いた従来の目標画素および目標領域抽出方式と比べ、背景の誤検出を低減し、追尾目標の大きさ、向きの変化などの影響を受けない目標画素および目標領域の抽出が可能となり、追尾精度の向上を図ることができる。 According to the tracking device according to the first embodiment as described above, the target shape appears in the residual quadratic form based on the predicted tracking value by predicting the strength in addition to the position and speed of the tracking target. By performing enhancement processing on the residual quadratic form using the distance difference value and the intensity difference value based on the tracking prediction value, the tracking target is further emphasized, and extraction of target pixels constituting the tracking target and the target region are performed. Can be easily extracted.
In other words, image processing typified by template matching is performed by predicting intensity in addition to position and speed, linking tracking processing and target detection processing, and extracting target pixels and target regions based on tracking prediction values Compared with conventional target pixel and target area extraction methods using, the detection of target pixels and target areas that are not affected by changes in the size and orientation of the tracking target is reduced, and tracking is enabled. The accuracy can be improved.

なお、実施の形態１では、同一画角の距離画像と強度画像が同時に取得できる場合を例に説明を行ったが、観測センサから画像中心部の３次元直交座標上の距離と、取得画像の画角情報が精度良く得られていれば、入力画像が強度画像のみであっても、距離画像に相当する情報を算出することは可能である。例えば、観測センサから画像中心部の３次元直交座標上の距離は、追尾目標と観測センサとの距離を検出する外部の測距センサ(図示省略)からの検出信号を入力し、取得画像の画角情報は記憶部に予め格納し、これらから距離画像に相当する情報を算出する処理手段(図示省略)を追尾装置に設けておけばよい。従って、この発明は、画像中心部のセンサ距離と画角情報が得られれば、入力画像が強度画像のみの場合であっても同様の効果を得ることができる。 In the first embodiment, the case where the distance image and the intensity image having the same angle of view can be acquired simultaneously has been described as an example. However, the distance on the three-dimensional orthogonal coordinate from the observation sensor to the center of the image, and the acquired image If the angle of view information is obtained with high accuracy, it is possible to calculate information corresponding to the distance image even if the input image is only the intensity image. For example, the distance from the observation sensor to the center of the image on the three-dimensional Cartesian coordinate system is obtained by inputting a detection signal from an external distance measuring sensor (not shown) that detects the distance between the tracking target and the observation sensor. The corner information may be stored in the storage unit in advance, and processing means (not shown) for calculating information corresponding to the distance image from these may be provided in the tracking device. Therefore, according to the present invention, if the sensor distance and the angle of view information at the center of the image can be obtained, the same effect can be obtained even when the input image is only the intensity image.

実施の形態２．
図２はこの発明の実施の形態２による追尾装置の機能ブロック図である。図２において目標検出装置３００は、目標画素指標算出部３１０と目標画素抽出部３１１が実施の形態１による追尾装置と異なっている。従って、実施の形態２による追尾装置の説明は、目標画素指標算出部３１０と目標画素抽出部３１１の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 2. FIG.
FIG. 2 is a functional block diagram of a tracking device according to Embodiment 2 of the present invention. In FIG. 2, the target detection apparatus 300 is different from the tracking apparatus according to the first embodiment in a target pixel index calculation unit 310 and a target pixel extraction unit 311. Therefore, the description of the tracking device according to the second embodiment is limited to the operations of the target pixel index calculation unit 310 and the target pixel extraction unit 311, and description of other functional blocks is omitted.

目標画素指標算出部３１０は、複数の重み係数αの値を保持し、それぞれのαの値に対して、実施の形態１における目標画素指標算出部１１０と同様に、上記式(８)を用いて目標画素指標を算出する。例えば、目標画素指標算出部３１０が、α＝{０．３，０．５，０．７}の３つの値を保持する場合、目標画素指標算出部３１０は、α＝０．３の場合の目標画素指標ＩＤＣ_ｋ ^{α＝０．３}と、α＝０．５の場合の目標画素指標ＩＤＣ_ｋ ^{α＝０．５}と、α＝０．７の場合の目標画素指標ＩＤＣ_ｋ ^{α＝０．７}とを算出する。 The target pixel index calculation unit 310 holds a plurality of weighting coefficient α values, and uses the above equation (8) for each α value, similar to the target pixel index calculation unit 110 in the first embodiment. To calculate a target pixel index. For example, when the target pixel index calculation unit 310 holds three values of α = {0.3, 0.5, 0.7}, the target pixel index calculation unit 310 Target pixel index IDC _k ^{α = 0.3} , target pixel index IDC _k ^{α = 0.5} when ^{α = 0.5} , target pixel index IDC _k ^{α = 0.7 when α = 0.7} And calculate.

次に、目標画素抽出部３１１が、目標画素指標算出部３１０が算出した複数の目標画素指標それぞれに対して、実施の形態１における目標画素抽出部１１１と同様に、上記式(９)を用いて目標画素の抽出および目標領域の抽出を行う。例えば、目標画素指標算出部３１０が、α＝{０．３，０．５，０．７}の３つの値を保持する場合、目標画素抽出部３１１は、ＩＤＣ_ｋ ^{α＝０．３}とＩＤＣ_ｋ ^{α＝０．５}と目標画素指標ＩＤＣ_ｋ ^{α＝０．７}の、３つの目標画素指標に対して、それぞれ目標画素および目標領域の抽出を行う。
さらに、目標画素抽出部３１１は、それぞれのαに対する抽出結果の中から、目標領域の面積と目標領域内の目標画素密度が１フレーム前の処理で求めた目標領域面積および目標画素密度に最も近い抽出結果を選択し、選択した抽出結果から目標画素および目標領域を決定する。 Next, the target pixel extraction unit 311 uses the above formula (9) for each of the plurality of target pixel indexes calculated by the target pixel index calculation unit 310, similarly to the target pixel extraction unit 111 in the first embodiment. The target pixel and the target area are extracted. For example, when the target pixel index calculation unit 310 holds three values of α = {0.3, 0.5, 0.7}, the target pixel extraction unit 311 determines that IDC _k ^{α = 0.3} and IDC. Extraction of the target pixel and the target area is performed for each of the three target pixel indexes of _k ^{α = 0.5} and target pixel index IDC _k ^{α = 0.7} .
Furthermore, the target pixel extraction unit 311 has an area of the target region and a target pixel density in the target region that are closest to the target region area and the target pixel density obtained by the processing one frame before from the extraction results for each α. An extraction result is selected, and a target pixel and a target area are determined from the selected extraction result.

上記のような実施の形態２による追尾装置によれば、目標画素指標の算出に用いる重み係数αの値を複数保持し、各αの値を用いて目標画素指標の算出と目標画素および目標領域の抽出を行い、１フレーム前の抽出結果に最も近いものを選択することにより適切な重み係数αを選択できる。その結果、目標画素および目標領域の抽出精度が向上し、追尾精度が向上する。 According to the tracking device according to the second embodiment as described above, a plurality of values of the weight coefficient α used for calculating the target pixel index are held, and the target pixel index is calculated using each α value, and the target pixel and target area are calculated. And an appropriate weighting factor α can be selected by selecting the one closest to the extraction result one frame before. As a result, target pixel and target region extraction accuracy is improved, and tracking accuracy is improved.

実施の形態３．
図３はこの発明の実施の形態３による追尾装置の機能ブロック図である。図３において目標検出装置４００は、目標画素指標算出部４１０が実施の形態１による追尾装置と異なっている。従って、実施の形態３による追尾装置の説明は、目標画素指標算出部４１０の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 3 FIG.
FIG. 3 is a functional block diagram of a tracking device according to Embodiment 3 of the present invention. In FIG. 3, the target detection device 400 is different from the tracking device according to the first embodiment in a target pixel index calculation unit 410. Therefore, the description of the tracking device according to the third embodiment is only the operation of the target pixel index calculation unit 410, and the description of the other functional blocks is omitted.

距離差分正規化部１０７が算出する距離差分正規化値は追尾目標予測位置に近いほど値が高くなる。一方、強度差分正規化部１０９が算出する強度差分正規化値は、追尾目標予測位置に関係なく、予測強度に近い程値が高くなる。すなわち、追尾目標以外であっても強度値が近ければ強度差分正規化値は高くなる。 The distance difference normalization value calculated by the distance difference normalization unit 107 increases as it is closer to the tracking target predicted position. On the other hand, the intensity difference normalized value calculated by the intensity difference normalizing unit 109 becomes higher as the predicted intensity is closer regardless of the tracking target predicted position. That is, the intensity difference normalized value is higher if the intensity value is close even if it is not a tracking target.

そこで、目標画素指標算出部４１０は、距離差分正規化値の重みを１とし、追尾目標の予測位置から離れる程、強度差分正規化値が小さくなるように、距離差分正規化値を強度差分正規化値の重みとし、目標画素指標を算出する。下記式(１０)に、目標画素指標算出部４１０が使用する目標画素指標算出式を示す。 Therefore, the target pixel index calculation unit 410 sets the weight of the distance difference normalized value to 1, and sets the distance difference normalized value to the intensity difference normalized value so that the intensity difference normalized value becomes smaller as the distance from the tracking target predicted position becomes larger. The target pixel index is calculated as the weight of the conversion value. The following formula (10) shows a target pixel index calculation formula used by the target pixel index calculation unit 410.

上記のような実施の形態３による追尾装置によれば、距離差分正規化値を強度差分正規化差分値の重み係数として目標画素指標を算出することにより、たとえ強度差分正規化値の値が高くても追尾予測位置から離れた位置の目標画素指標は低く抑えられるため、追跡目標がより強調された目標画素指標を得ることができる。その結果、目標画素および目標領域の抽出精度が向上し、追尾精度が向上する。 According to the tracking device according to the third embodiment as described above, the target pixel index is calculated using the distance difference normalized value as the weighting coefficient of the intensity difference normalized difference value, thereby increasing the value of the intensity difference normalized value. However, since the target pixel index at a position away from the predicted tracking position is kept low, it is possible to obtain a target pixel index in which the tracking target is more emphasized. As a result, target pixel and target region extraction accuracy is improved, and tracking accuracy is improved.

実施の形態４．
図４はこの発明の実施の形態４による追尾装置の機能ブロック図である。図４において目標検出装置５００は、目標画素指標算出部５１０が実施の形態２による追尾装置と異なっている。従って、実施の形態４による追尾装置の説明は、目標画素指標算出部５１０の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 4 FIG.
4 is a functional block diagram of a tracking device according to Embodiment 4 of the present invention. In FIG. 4, the target detection apparatus 500 is different from the tracking apparatus according to the second embodiment in a target pixel index calculation unit 510. Therefore, the description of the tracking device according to the fourth embodiment is limited to the operation of the target pixel index calculation unit 510, and the description of the other functional blocks is omitted.

目標画素指標算出部５１０は、複数の重み係数αの値を保持し、それぞれのαの値に対して、実施の形態１における目標画素指標算出部１１０と同様に、上記式(８)を用いて目標画素指標を算出する。さらに、実施の形態３における目標画素指標算出部４１０と同様に上記式(１０)を用いて目標画素指標を算出する。 The target pixel index calculation unit 510 holds a plurality of values of the weighting factor α, and uses the above equation (8) for each α value as in the target pixel index calculation unit 110 in the first embodiment. To calculate a target pixel index. Further, the target pixel index is calculated using the above equation (10) in the same manner as the target pixel index calculation unit 410 in the third embodiment.

上記のような実施の形態４による追尾装置によれば、異なる重み係数と、異なる算出式を用いて複数の目標画素指標を算出し、各目標画素指標に対して目標画素および目標領域の抽出を行い、１フレーム前の抽出結果に最も近いものを選択することにより、目標画素および目標領域の抽出精度が向上し、追尾精度が向上する。 According to the tracking device according to the fourth embodiment as described above, a plurality of target pixel indices are calculated using different weighting factors and different calculation formulas, and target pixels and target areas are extracted for each target pixel index. By performing the selection and selecting the one closest to the extraction result of the previous frame, the extraction accuracy of the target pixel and the target region is improved, and the tracking accuracy is improved.

実施の形態５．
図５はこの発明の実施の形態５による追尾装置の機能ブロック図である。図５において目標検出装置６００は、目標画素指標算出部６１０と閾値処理部６１２が実施の形態１による追尾装置と異なっている。従って、実施の形態５による追尾装置の説明は、目標画素指標算出部６１０および閾値処理部６１２のみとし、ほのか機能ブロックの説明は省略する。 Embodiment 5 FIG.
FIG. 5 is a functional block diagram of a tracking device according to Embodiment 5 of the present invention. In FIG. 5, a target detection apparatus 600 is different from the tracking apparatus according to the first embodiment in a target pixel index calculation unit 610 and a threshold processing unit 612. Therefore, the description of the tracking device according to the fifth embodiment is limited to the target pixel index calculation unit 610 and the threshold processing unit 612, and the explanation of the faint functional block is omitted.

閾値処理部６１２は、残差正規化部１０５が算出した残差２次形式正規化値と、距離差分正規化部１０７が算出した距離差分正規化値と、強度差分正規化部１０９が算出した強度差分正規化値それぞれに対して閾値処理を行い、閾値を超える画素のみを目標画素指標算出対象とする。
例えば、残差２次形式正規化値の平均と標準偏差を求め、下記式(１１)を満たす画素(ｉ，ｊ)を、目標画素指標算出対象とする。 The threshold processing unit 612 is calculated by the residual quadrature normalized value calculated by the residual normalizing unit 105, the distance difference normalized value calculated by the distance difference normalizing unit 107, and the intensity difference normalizing unit 109. Threshold processing is performed on each intensity difference normalized value, and only pixels exceeding the threshold are set as target pixel index calculation targets.
For example, the average and standard deviation of the residual quadratic normalized values are obtained, and the pixel (i, j) that satisfies the following formula (11) is set as the target pixel index calculation target.

距離差分正規化値、強度差分正規化値それぞれに対しても、残差２次形式正規化値の場合と同様にして目標画素指標算出対象を求める。 For each of the distance difference normalized value and the intensity difference normalized value, the target pixel index calculation target is obtained in the same manner as in the case of the residual quadratic normalized value.

次に、目標画素指標算出部６１０は、閾値処理部６１２が抽出した、残差２次形式正規化値における目標画素指標算出対象と距離差分正規化値における目標画素指標算出対象と強度差分正規化値における目標画素指標算出対象の、全てに含まれる画素のみを処理対象として、目標画素指標を算出する。なお、算出式は、実施の形態１における目標画素指標算出部１１０が用いる上記式(８)と同じである。 Next, the target pixel index calculation unit 610 extracts the target pixel index calculation target in the residual quadratic normalized value and the target pixel index calculation target in the distance difference normalized value and the intensity difference normalization extracted by the threshold processing unit 612. The target pixel index is calculated by setting only the pixels included in all of the target pixel index calculation targets in the values to be processed. The calculation formula is the same as the formula (8) used by the target pixel index calculation unit 110 in the first embodiment.

なお、上記説明では実施の形態１による追尾装置に閾値処理部６１２を追加した場合について示したが、実施の形態２〜４による追尾装置に対しても同様に閾値処理部６１２の追加が可能である。 In the above description, the threshold processing unit 612 is added to the tracking device according to the first embodiment. However, the threshold processing unit 612 can be similarly added to the tracking device according to the second to fourth embodiments. is there.

上記のような実施の形態５による追尾装置によれば、残差２次形式正規化値と距離差分正規化値と強度差分正規化値に対して閾値処理を施し、目標画素指標算出対象の絞り込みを行うことにより、背景部分が除去され、目標画素がより鮮明に強調される。その結果、目標画素および目標領域の抽出精度が向上し、追尾精度が向上する。 According to the tracking device according to the fifth embodiment as described above, threshold processing is performed on the residual quadratic form normalized value, the distance difference normalized value, and the intensity difference normalized value to narrow down the target pixel index calculation target. By performing the above, the background portion is removed, and the target pixel is more clearly emphasized. As a result, target pixel and target region extraction accuracy is improved, and tracking accuracy is improved.

実施の形態６．
図６はこの発明の実施の形態６による追尾装置の機能ブロック図である。図６において目標検出装置７００は、閾値処理部７１２と目標画素指標算出部７１０が実施の形態２による追尾装置と異なっている。従って、実施の形態６による追尾装置の説明は、閾値処理部７１２および目標画素指標算出部７１０の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 6 FIG.
FIG. 6 is a functional block diagram of a tracking device according to Embodiment 6 of the present invention. In FIG. 6, the target detection device 700 is different from the tracking device according to the second embodiment in a threshold processing unit 712 and a target pixel index calculation unit 710. Therefore, the description of the tracking device according to the sixth embodiment is limited to the operations of the threshold processing unit 712 and the target pixel index calculation unit 710, and description of other functional blocks is omitted.

閾値処理部７１２の動作は、基本的に実施の形態５における閾値処理部６１２と同じ閾値処理を行うが、複数の重み係数γを保持し、それぞれのγに対して閾値処理を実行する点が異なっている。 The operation of the threshold processing unit 712 is basically the same threshold processing as that of the threshold processing unit 612 in the fifth embodiment. However, the threshold processing unit 712 holds a plurality of weighting factors γ and executes threshold processing for each γ. Is different.

目標画素指標算出部７１０は、閾値処理部７１２が算出した複数の閾値処理結果それぞれに対して目標画素指標の算出を行う。
目標画素指標算出部７１０は、実施の形態１における目標画素指標算出部１１０と同様に、上記式(８)により目標画素指標の算出を行うが、式(８)における重み係数αの値は１つ以上保持するものとし、αの値を複数保持する場合は、閾値処理部７１２が算出した各閾値処理結果に対して、各αの値を用いて複数の目標画素指標を算出する。
また、目標画素指標算出部７１０は、式(８)の代わりに、実施の形態３における目標画素指標算出部４１０と同様に上記式(１０)を用いて目標画素指標を算出してもよいし、実施の形態４における目標画素指標算出部５１０と同様に、上記式(８)および式(１０)それぞれを用いて、閾値処理部７１２が算出した各閾値処理結果に対して複数の目標画素指標を算出しても良い。 The target pixel index calculation unit 710 calculates a target pixel index for each of a plurality of threshold processing results calculated by the threshold processing unit 712.
The target pixel index calculation unit 710 calculates the target pixel index by the above equation (8), similarly to the target pixel index calculation unit 110 in the first embodiment, but the value of the weighting factor α in the equation (8) is 1. When a plurality of values of α are held, a plurality of target pixel indices are calculated using each α value for each threshold processing result calculated by the threshold processing unit 712.
Further, the target pixel index calculation unit 710 may calculate the target pixel index using the above formula (10) in the same manner as the target pixel index calculation unit 410 in the third embodiment instead of the formula (8). Similarly to the target pixel index calculation unit 510 in the fourth embodiment, a plurality of target pixel indexes are calculated for each threshold processing result calculated by the threshold processing unit 712 using each of the above equations (8) and (10). May be calculated.

上記のような実施の形態６による追尾装置によれば、残差２次形式正規化値と距離差分正規化値と強度差分正規化値に対する閾値処理を、閾値処理の重み係数を変えて複数回実施し、各閾値処理結果に対して目標画素指標の算出を行い、さらに各目標画素指標に対して目標画素および目標領域の抽出を行い、１フレーム前の抽出結果に最も近いものを選択することにより、目標画素および目標領域の抽出精度が向上し、追尾精度が向上する。 According to the tracking device according to the sixth embodiment as described above, the threshold processing for the residual quadratic normalized value, the distance difference normalized value, and the intensity difference normalized value is performed a plurality of times by changing the weighting coefficient of the threshold processing. Implement the target pixel index for each threshold processing result, extract the target pixel and target area for each target pixel index, and select the one closest to the extraction result one frame ago As a result, the extraction accuracy of the target pixel and the target region is improved, and the tracking accuracy is improved.

実施の形態７．
図７はこの発明の実施の形態７による追尾装置の機能ブロック図である。図７において目標検出装置８００は、追尾点決定部８０２が実施の形態１による追尾装置と異なっている。従って、実施の形態７による追尾装置の説明は追尾点決定部８０２の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 7 FIG.
FIG. 7 is a functional block diagram of a tracking device according to Embodiment 7 of the present invention. In FIG. 7, the target detection device 800 is different from the tracking device according to the first embodiment in a tracking point determination unit 802. Therefore, the description of the tracking device according to the seventh embodiment is only the operation of the tracking point determination unit 802, and the description of the other functional blocks is omitted.

追尾点決定部８０２は、目標画素抽出部１１１により抽出された目標領域内の目標画素の中で、強度値が高い順にＮ個の目標画素を選択し、強度値を質量と見なして選択したＮ個の目標画素の重心を追尾点とする。
なお、重心点の算出には、強度値の代わりに目標画素指標を用いても良い。
また、上記説明では実施の形態１による追尾装置を例に示したが、実施の形態２〜６による追尾装置に対しても同様の処理が可能である。 The tracking point determination unit 802 selects N target pixels in descending order of the intensity value among the target pixels in the target area extracted by the target pixel extraction unit 111, and selects the N values that are determined by regarding the intensity value as mass. The center of gravity of each target pixel is set as a tracking point.
In calculating the barycentric point, a target pixel index may be used instead of the intensity value.
In the above description, the tracking device according to the first embodiment is shown as an example, but the same processing can be performed for the tracking device according to the second to sixth embodiments.

上記のような実施の形態７の追尾装置によれば、目標領域内の、強度値の高いまたは目標画素指標の高い複数の目標画素の重心を追尾点とすることにより、追尾フィルタ装置に入力する追尾点が安定し、追尾精度が向上する。 According to the tracking device of the seventh embodiment as described above, the center of gravity of a plurality of target pixels having a high intensity value or a high target pixel index in the target region is set as the tracking point, and input to the tracking filter device. The tracking point is stabilized and the tracking accuracy is improved.

実施の形態８．
図８はこの発明の実施の形態８による追尾装置の機能ブロック図である。図８において追尾フィルタ装置９００は、強度揺らぎ算出部９０３が実施の形態１による追尾装置と異なっている。従って、実施の形態８による追尾装置の説明は強度揺らぎ算出部９０３の動作のみとし、他の機能ブロックの説明は省略する。 Embodiment 8 FIG.
FIG. 8 is a functional block diagram of a tracking device according to Embodiment 8 of the present invention. In FIG. 8, the tracking filter device 900 is different from the tracking device according to the first embodiment in the intensity fluctuation calculation unit 903. Therefore, the description of the tracking device according to the eighth embodiment is only the operation of the intensity fluctuation calculation unit 903, and the description of the other functional blocks is omitted.

追尾処理部２０１は、入力される追尾点の状態に一定量(所定量)の揺らぎを駆動雑音パラメータとして考慮し、追尾予測を行っている。位置および速度の揺らぎについては、想定する運動モデルに基づいて設定可能であるが、強度の揺らぎについては想定が難しい。
そこで、強度揺らぎ算出部９０３では、追尾点強度の時系列変化に対する標準偏差を毎フレーム算出し、算出した強度の標準偏差を、強度の駆動雑音パラメータとして追尾処理部２０１に入力する。 The tracking processing unit 201 performs tracking prediction by considering a certain amount (predetermined amount) of fluctuation as a driving noise parameter in the state of the input tracking point. The position and velocity fluctuations can be set based on the assumed motion model, but the intensity fluctuations are difficult to assume.
Therefore, the intensity fluctuation calculation unit 903 calculates the standard deviation for the time-series change of the tracking point intensity every frame, and inputs the calculated standard deviation of the intensity to the tracking processing unit 201 as an intensity driving noise parameter.

上記のような実施の形態８の追尾装置によれば、観測される追尾点強度よりその標準偏差を求め、追尾処理に用いる強度の駆動雑音パラメータとして用いることにより、強度値の予測精度が向上し、残差２次形式において目標の形状が現れやすくなる。その結果、目標画素および目標領域の抽出精度が向上し、追尾精度の向上が図られる。 According to the tracking device of the eighth embodiment as described above, the standard deviation is obtained from the observed tracking point intensity, and is used as the driving noise parameter of the intensity used for the tracking process, thereby improving the accuracy prediction of the intensity value. The target shape is likely to appear in the residual quadratic form. As a result, target pixel and target area extraction accuracy is improved, and tracking accuracy is improved.

なお、残差算出部１０３，距離差分算出部１０６，強度差分算出部１０８は第１の処理手段を構成し、ゲート処理部１０４，残差正規化部１０５，距離差分正規化部１０７、強度差分正規化部１０９は第２の処理手段を構成し、目標画素指標算出部１１０，３１０，４１０，５１０，６１０，７１０、目標画素抽出部１１１，３１１、閾値処理部６１２，７１２は第３の処理手段を構成し、追尾点決定部１０２，８０２は第４の処理手段を構成する。 The residual calculation unit 103, the distance difference calculation unit 106, and the intensity difference calculation unit 108 constitute a first processing unit, and include a gate processing unit 104, a residual normalization unit 105, a distance difference normalization unit 107, and an intensity difference. The normalization unit 109 constitutes a second processing unit, and the target pixel index calculation units 110, 310, 410, 510, 610, 710, the target pixel extraction units 111, 311 and the threshold processing units 612, 712 are the third processing. The tracking point determination units 102 and 802 constitute a fourth processing unit.

またこの発明は、上記各実施の形態に限定されるものではなく、上記各実施の形態の可能な組み合わせを全て含むことは云うまでもない。 The present invention is not limited to the above-described embodiments, and it is needless to say that all possible combinations of the above-described embodiments are included.

１００，３００，４００，５００，６００，７００，８００目標検出装置、１０１初期目標領域情報取得部、１０２，８０２追尾点決定部、１０３残差算出部、１０４ゲート処理部、１０５残差正規化部、１０６距離差分算出部、１０７距離差分正規化部、１０８強度差分算出部、１０９強度差分正規化部、１１０，３１０，４１０，５１０，６１０，７１０目標画素指標算出部、１１１，３１１目標画素抽出部、２００，９００追尾フィルタ装置、２０１追尾処理部、２０２遅延処理部、６１２，７１２閾値処理部、９０３強度揺らぎ算出部。 100, 300, 400, 500, 600, 700, 800 Target detection apparatus, 101 Initial target area information acquisition unit, 102, 802 Tracking point determination unit, 103 Residual calculation unit, 104 Gate processing unit, 105 Residual normalization unit , 106 Distance difference calculation unit, 107 Distance difference normalization unit, 108 Intensity difference calculation unit, 109 Intensity difference normalization unit, 110, 310, 410, 510, 610, 710 Target pixel index calculation unit, 111, 311 Target pixel extraction Unit, 200, 900 tracking filter device, 201 tracking processing unit, 202 delay processing unit, 612, 712 threshold processing unit, 903 intensity fluctuation calculation unit.

Claims

A distance image and intensity image of the same angle of view simultaneously observed by the observation sensor are input in units of frames.
The tracking point information of the latest frame image is input, and the three-dimensional position coordinates obtained from the distance image of the tracking point and the intensity value obtained from the intensity image are used as observation vectors, and the state prediction value of the tracking target in the next frame is output by the tracking filter. A tracking filter device;
Pixels in which each pixel constitutes a target based on the difference between the second-order residual format in the latest frame image based on the state prediction value of the tracking target from the tracking filter device, the distance difference between the tracking prediction position, and the tracking prediction intensity Calculating a target pixel index representing the certainty, further extracting a target area constituting a target in the latest frame image from the target pixel index, and performing a new tracking from the pixels included in the target area to the tracking filter device A target detection device that calculates and outputs point information;
Bei to give a,
The tracking filter device uses the three-dimensional position coordinates obtained from the distance image of the tracking point of the tracking point information and the intensity value obtained from the intensity image as an observation vector, and uses the tracking filter to determine the three-dimensional position coordinates of the tracking point in the next frame. Obtain 7-dimensional state prediction values of dimensional velocity and intensity,
The target detection device includes:
A residual quadratic form value of each pixel obtained from the state prediction value output from the tracking filter device, a three-dimensional position of each pixel obtained from the distance image, and an intensity value of each pixel obtained from the intensity image, and the state prediction A first processing means for obtaining a distance difference value between the predicted tracking position of the value and the three-dimensional position of each pixel, and an intensity difference value between the tracking predicted intensity of the state predicted value and the intensity value of each pixel;
Second processing means for normalizing the residual quadratic form value, the distance difference value, and the intensity difference value;
A distance difference normalized value and an intensity difference normalized value are respectively multiplied by a weighting factor, and a target pixel index of each pixel is calculated by a product sum with the residual quadratic normalized value, and the average and standard of the target pixel index A third processing means for determining a target pixel detection threshold from the deviation, extracting a pixel whose target pixel index value exceeds the target pixel detection threshold as a target pixel, and further extracting a target region from the extracted distribution of the target pixels;
A fourth processing means for setting the center position of the extracted target area as a tracking point in the latest frame image and inputting the tracking point information to a tracking filter device;
including,
A tracking device characterized by that.

In the target detection device, the third processing unit holds a plurality of weights of the distance difference normalized value and the intensity difference normalized value, and calculates the target pixel index and extracts the target pixel and target area using each weight. 2. The tracking device according to claim 1 , wherein the tracking device selects a target region having a size of the target region and a density of the target pixels in the target region that are closest to an extraction result one frame before.

In the target detection apparatus, the third processing unit calculates a target pixel index by setting the weight of the distance difference normalized value to 1, and further using the distance difference normalized value as the weight of the intensity difference normalized value. The tracking device according to claim 1 .

In the target detection device, the third processing means holds a plurality of weights of the distance difference normalized value and the intensity difference normalized value, calculates a target pixel index using each weight, and further, the distance difference normalization The target pixel index is calculated by using a weight of the normalized value as 1, and using the distance difference normalized value as the weight of the intensity difference normalized value, and for each of the obtained target pixel indices, was extracted target area, according to claim 1, the density of the target pixel size and the target area of the target area and selects the nearest object area in one frame before the extraction result Tracking device.

In the target detection device, the third processing unit calculates a processing target detection threshold based on the average and standard deviation for the residual quadratic normalized value, the distance difference normalized value, and the intensity difference normalized value, A target pixel index is calculated for pixels whose residual quadratic form normalized value, distance difference normalized value, and intensity difference normalized value all exceed the respective processing target detection thresholds. The tracking device according to any one of claims 1 to 4 .

In the target detection apparatus, the third processing unit uses a plurality of coefficients for multiplying the standard deviation based on the average and the standard deviation for the residual quadratic form normalized value, the distance difference normalized value, and the intensity difference normalized value. A plurality of processing target detection thresholds, and using each of the plurality of processing target detection thresholds, the residual quadratic form normalized value, the distance difference normalized value, and the intensity difference normalized value are all A plurality of target pixel indices are calculated for pixels that exceed the processing target detection threshold, and target pixels and target areas are extracted for each of the plurality of target pixel indices, and the size of the target area and the target area 5. The tracking device according to claim 1 , wherein a target region in which a density of the target pixel is closest to an extraction result of one frame before is selected.

In the target detection device, the fourth processing unit selects N pixels in order of increasing intensity value from the target pixels in the target area, and regards the N target pixels selected by regarding the intensity value as mass. The tracking device according to any one of claims 1 to 6 , wherein the center of gravity is used as a tracking point.

In the target detection device, the fourth processing unit selects N pixels in order from the highest target pixel index among the target pixels in the target area, and selects the N number of pixels selected by regarding the target pixel index value as mass. The tracking device according to claim 1 , wherein the center of gravity of the target pixel is set as a tracking point.

The tracking filter apparatus, the standard deviation for the time series change of the tracking point strength of claims 1, which comprises carrying out the tracking process the tracking point intensity standard deviation as a drive noise parameters of the tracking point intensity until 8 The tracking device according to any one of the above.

A fifth processing means for calculating information corresponding to the distance image from the distance from the external observation sensor on the three-dimensional orthogonal coordinates of the image center and the angle-of-view information of the acquired image stored in advance in the storage unit; The tracking device according to claim 1 , wherein a distance image is not required as an input image.