JP6983049B2 - Object tracking device and its program - Google Patents

Description

The present invention relates to an object tracking device and a program thereof for tracking an object using an infrared image and a visible image.

In recent years, with the progress of video analysis technology, various applications using cameras have been proposed. The development of this technology is particularly remarkable in the video analysis of sports scenes. For example, the tennis Hawkeye system, which is also used in Wimbledon, tracks a tennis ball three-dimensionally using images from a plurality of fixed cameras and determines IN / OUT. In addition, at the 2014 FIFA World Cup, called goal line technology, images from several fixed cameras are analyzed to automate goal determination. In addition, a TRACAB system that installs a large number of stereo cameras in a soccer stadium and tracks all players in the field in real time is also known.

These video analysis techniques are often premised on using video taken with a camera having a time resolution of 30 frames / sec (fps). For example, when an object that moves at a high speed that is difficult to see, such as a fencing sword tip or a badminton shuttle, is photographed, extreme motion blur occurs in the object on the image (reference numeral α in FIG. 8). For this reason, it is extremely difficult to accurately measure the object position only from the video. When shooting an object that moves at high speed, motion blur can be reduced by using a high-speed camera that exceeds 30 fps or by increasing the shutter speed. On the other hand, high-speed cameras are expensive, and there is a problem that the brightness of the image decreases when the shutter speed is increased.

Therefore, a technique has been proposed in which a sensor is attached to an object and its movement is analyzed. For example, the inventions described in Patent Documents 1 and 2 identify the play content by machine learning from an acceleration sensor attached to a player's wear or wristband. According to this conventional technique, it is possible to automatically determine the content of play such as serve or volley in tennis. However, in the inventions described in Patent Documents 1 and 2, the sensor attached to the athlete often interferes with the competition, and it is not realistic to use the sensor in an actual game.

Therefore, a technique of not attaching a sensor to an object has also been proposed. For example, the invention described in Patent Document 3 accurately (robustly) by attaching a reflective tape to the tip of a fencing sword, irradiating the reflective tape with infrared light, and detecting the reflected light from an infrared image. It tracks objects.

Japanese Unexamined Patent Publication No. 2014-187481 Japanese Unexamined Patent Publication No. 2016-97228 Japanese Unexamined Patent Publication No. 2017-156796

However, in the invention described in Patent Document 3, when the object moves at high speed or the reflective tape does not face the camera, the reflected light from the reflective tape is weak, so that it is difficult to detect the object in the infrared image. Therefore, the detection position is missing. Even if the reflected light has a sufficient amount of light and the tracking can be resumed, if the missing section of the detection position is long, the trajectory becomes unnatural, and it is difficult to express the accurate trajectory of the object.

Therefore, it is an object of the present invention to provide an object tracking device and a program thereof that can accurately track an object even if a detection position is missing.

In view of the above-mentioned problems, the object tracking device according to the present invention has an infrared image of one or more moving objects with an infrared light marker taken with infrared light and a visible image of the object taken with visible light. An object tracking device that tracks the object using an image, such as an infrared light detection unit, a position data generation unit, a missing section identification unit, a feature point detection unit, a position data interpolation unit, and an object. It is configured to include a tracking unit.

According to such a configuration, the object tracking device detects the region of the infrared light marker from the infrared image as an infrared light region by the infrared light detection unit.
Further, in the object tracking device, when the position data generation unit can detect the infrared light region, the detection position of the infrared light region is detected, or when the infrared light region cannot be detected, the detection position is missing. Generates position data including time information of the infrared image to be detected in the infrared light region.

Further, the object tracking device identifies the missing section of the position data by the missing section specifying unit, and determines whether or not the time length from the start time to the end time of the specified missing section is equal to or longer than a preset threshold value. judge.

Here, in the infrared images immediately before the start time and immediately after the end time, the infrared light marker attached to the object is successfully detected. Therefore, in the visible images of the start time and the end time, there is a high possibility that an object exists around the detection position in those infrared images.

Therefore, when the time length of the missing section is equal to or longer than the threshold value, the object tracking device acquires the detection position from the position data immediately before the start time and immediately after the end time by the feature point detection unit, and the acquired detection position. The image area is set to the visible image of the start time and the end time based on the position, and the feature point is detected from the set image area.

Further, the object tracking device interpolates the position data of the start time and the end time at the detection position of the feature point by the position data interpolation unit, and outputs the interpolated position data to the missing section identification unit.
Further, the object tracking device tracks the object based on the position data when the time length of the missing section is less than the threshold value by the object tracking unit.
Then, the object tracking device identifies the missing section of the position data input from the position data interpolation unit by the missing section specifying unit.

In this way, the object tracking device does not detect feature points in chronological order, but detects feature points from visible images at the start time and end time when the object is likely to exist, and at the detection position of the feature points. Interpolate the position data. Then, the object tracking device repeats the process so as to narrow the missing section of the position data from both ends.

According to the present invention, the following excellent effects are obtained.
The object tracking device according to the present invention detects a feature point from a visible image of a start time and an end time at which an object is likely to exist, and interpolates position data at the detection position of the feature point. This allows the object tracking device to accurately track the object even if the detection position is missing.

It is a schematic block diagram of the object tracking system which concerns on embodiment of this invention. It is an enlarged view of the tip of the sword. It is a block diagram which shows the structure of the object tracking apparatus of FIG. (A) and (b) are explanatory views explaining position data. (A) and (b) are explanatory views explaining the FAST feature amount. It is explanatory drawing explaining an example of a locus composite image. It is a flowchart which shows the operation of the object tracking apparatus of FIG. It is explanatory drawing explaining the motion blur in the image of fencing.

[Overview of object tracking system]
An outline of the object tracking system 1 according to the embodiment of the present invention will be described with reference to FIG.
In the following embodiments, in fencing, the sword tip (object) possessed by the player will be described as a tracking target. During fencing, the tips of both players often move at high speeds.

The object tracking system 1 tracks the positions of two sword tips moving at high speed by using a visible / infrared optical axis camera 20 capable of photographing visible light and infrared light on the same optical axis, and its trajectory C. (C ₁ , C ₂ ) is drawn. As shown in FIG. 1, the object tracking system 1 includes an infrared light projector 10, a visible / infrared optical axis camera 20, and an object tracking device 30.

The infrared light projector 10 is a general floodlight that projects infrared light.
As shown in FIG. 2, the infrared light emitted by the infrared light projector 10 is reflected by the reflective tape (infrared light marker) 91 attached to the sword tips 90 of both athletes, and is the same as visible and infrared rays described later. The image is taken by the optical axis camera 20.

The reflective tape 91 reflects infrared rays from the infrared light projector 10. One or more reflective tapes 91 may be attached to the sword tip 90, and the size and number of the reflective tapes 91 are not particularly limited. In the example of FIG. 2, the sword tip 90 has one rectangular reflective tape 91 attached to its side surface. Here, the sword tip 90 may have one reflective tape 91 added to the opposite side surface, or a strip-shaped reflective tape 91 may be wrapped around the side surface (not shown).

The visible / infrared same optical axis camera 20 captures visible light and infrared light on the same optical axis to generate a visible image I and an infrared image H having the same number of pixels. In the present embodiment, the visible / infrared optical axis camera 20 generates a visible image I in which a fencing competition is photographed and an infrared image H in which the reflective tape 91 of the sword tip 90 is photographed. Here, since the image coordinates of the sword tip 90 of the visible image I and the reflective tape 91 of the infrared image H correspond to each other, the locus C can be drawn without performing viewpoint conversion in the three-dimensional space.

The object tracking device 30 tracks the sword tips 90 of both athletes by using the infrared image H and the visible image I input from the visible / infrared optical axis camera 20. _{Then, the object tracking device 30 draws the trajectories C 1} and C ₂ of the sword tips 90 of both the tracked athletes in different colors, and combines the drawn trajectories C ₁ and C ₂ with the visible image I to create a locus composite image. Generate F.
_{In FIG. 1, the locus C 1 of the} sword tip 90 held by the player on the left side is shown by a broken line, and the locus C _{2 of the} sword tip 90 held by the player on the right side is shown by a dashed line.

[Object Tracker Configuration]
The configuration of the object tracking device 30 will be described with reference to FIG.
As shown in FIG. 3, the object tracking device 30 includes an infrared light detecting means 31, a visible image analysis means 33, and a time code interlocking CG synthesis means 35. Hereinafter, the time code interlocking CG synthesizing means 35 is abbreviated as TC interlocking CG synthesizing means 35.

Here, in the object tracking device 30, the infrared image H and the visible image I of the frames 1, ..., T-1, t, ... Continuous in the time direction are input, and the input infrared image H and the visible image I are input. Will be processed sequentially. Thereafter, the current frame (present frame) and t, the infrared image H and the visible image I of the current frame t and infrared image H _t and the visible image I _t. The subscript t is time information (for example, representing a time code) of each image or position information.

The infrared light detecting means 31 detects the _{sword tip 90 from the infrared image H t} of the current frame t and generates the position data of the sword tip 90. In the present embodiment, infrared light detecting means 31 detects the two hook point 90 included in the infrared image H _t, to generate position data for each point of a sword 90. The infrared light detection means 31 includes an infrared light detection unit 311 and a position data generation unit 313.

Infrared detection unit 311 detects a region of the infrared light marker as infrared light region from the input from the visible and infrared same optical axis camera 20 infrared image H _t. The infrared light detection unit 311 can detect an infrared light region by any method. Hereinafter, an example of a detection method for an infrared light region in the infrared light detection unit 311 will be described.

In this embodiment, it extracts an area of the reflective tape 91 as a candidate blob from the infrared image H _t. Then, the infrared light detection unit 311 detects the candidate blobs within the range of the preset area and the shape feature amount as the detection blobs in descending order of the area of the candidate blobs by the preset upper limit number of objects. In this embodiment, the upper limit of the number of objects is set to '2', which is the same number as the sword tip 90.

First, the infrared light detecting unit 311, using Equation (1) below, generates a binary infrared difference image and the infrared image H _t, of the previous frame and the infrared image H _t-1 by, to extract only the area Q _t of the moving object. In other words, the infrared light detecting section 311, an infrared image _{H t} of pixel _(x t, _{y t)} and the luminance value ^{H xy} _t of infrared image _{H t-1} of the pixel _{(x t-1, y} t- The difference from the luminance value H ^xy _{t-1 of 1) is obtained.} Then, the infrared photodetection unit 311 extracts the region Q ^xy _{t of the moving object whose difference obtained exceeds the preset threshold value R bri2} as a candidate blob.

Here, x and y represent horizontal and vertical image coordinates. Further, the threshold value R _bri2 is set in advance with an arbitrary value. Further, "0" and "255" in the equation (1) represent the luminance value of each pixel.
The infrared light detection unit 311 generated a binary infrared difference image Q ^xy _t in order to suppress the generation of a stationary noise blob, but extracted a region having high brightness in the infrared image H _{t as a candidate blob.} You may.

Next, the infrared photodetection unit 311 performs morphology treatment (opening morphology treatment) on the extracted candidate blobs to eliminate noise blobs in a small region. This morphology processing is a processing for erasing a noise blob in a small area by performing an image-to-image calculation between an image group in which an image is shifted in several directions on a pixel-by-pixel basis and the original image.

Next, the infrared photodetector 311 applies a labeling treatment to the candidate blobs remaining in the morphology treatment. This labeling process is a process of assigning a label (number) to a candidate blob.

Next, the infrared photodetection unit 311 obtains the position, area, and shape feature amount of the candidate blob that has been subjected to the labeling treatment. Here, the position of the candidate blob is the center position or the center of gravity position of the candidate blob. The shape feature amount of the candidate blob is the circularity or the aspect ratio of the circumscribed rectangle.

Then, the infrared light detection unit 311 erases the candidate blobs that are not within the range from the preset minimum area to the maximum area. Further, the infrared light detection unit 311 erases candidate blobs whose shape feature amount is not within the preset range. Further, when the number of candidate blobs exceeds the upper limit number of objects, the infrared light detection unit 311 leaves two candidate blobs having a large area as detection blobs and erases the other candidate blobs.

If you can detect the infrared light region (if detected blob remained), infrared light detecting unit 311 in association with the time information of the infrared image H _t, the position of the detection blob detection of infrared region It is output to the position data generation unit 313 as a position.
On the other hand, (when the detected blob is not left) If you can not detect the infrared light region, the infrared light detecting section 311, together with the time information of the infrared image H _t, the position can not be detected infrared light region data generating unit Notify 313 (notification of detection failure).

Position data generating unit 313, a detection position or detection failure notification infrared region input from the infrared light detecting section 311, based on the time information of the infrared image H _t, generates position data P _t It is a thing. Then, the position data generation unit 313 outputs the generated position data P to the missing section specifying unit 331.

<Position data>
Hereinafter, the position data will be described in detail.
The position data P _t is the information that associates the position information of the infrared region, and the time information of the infrared image H _t became detected in the infrared light region (time code).
In the present embodiment, the position information in the infrared light region represents the detection position of the sword tip 90. Here, the position information of the infrared region, when the detected position of the infrared region from the infrared light detecting section 311 is input, the image coordinates (x _{t, y t)} of the infrared light region becomes. On the other hand, the position information in the infrared light region becomes image coordinates (-1, -1) indicating the lack of the detection position when the detection failure notification is input from the infrared light detection unit 311.

As shown in FIG. 4 (a), each row of position data P is represents a time code of the infrared image H _t, and the positional information of the infrared region included in the infrared image H _t. In the position data P of FIG. 4A, the 1st to 4th lines and the 10th to 11th lines represent the detection position in the infrared light region, and the 5th to 9th lines indicate the missing detection position. show.

For example, in the first line of the position data P, the time code of the infrared image H is "10: 25: 30.25", and the image coordinates of the infrared light region are (1250, 350). Further, in the fifth line of the position data P, the time code of the infrared image H is "10: 25: 30.29" and the image coordinates of the infrared light region are (-1, -1), so that the detection position is detected. Represents the lack of.

The position data P is generated for each object. Since there are two sword tips 90 in this embodiment, the position data P in FIG. 4 corresponds to the sword tip 90 of the left player, and the position data P corresponding to the sword tip 90 of the right player is also generated separately.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
The visible image analysis means 33 identifies a missing section included in the position data P, detects feature points from the visible images at both ends of the missing section, and interpolates the position data P. The visible image analysis means 33 includes a missing section specifying unit 331, a feature point detecting unit 333, and a position data interpolation unit 335.

The missing section specifying unit 331 identifies the missing section of the position data P input from the position data generation unit 313 or the position data interpolation unit 335, and determines whether or not the time length d of the specified missing section is equal to or greater than the threshold value m. It is a judgment.

First, the missing section specifying unit 331 refers to each line of the position data P, and identifies the missing section in which the position information is continuously missing. In the present embodiment, the missing section specifying unit 331 specifies a row including image coordinates (-1, -1) in the position data P.

Then, the missing section identifying unit 331, image coordinates (-1, -1) of the missing section to be continuous, most previous time was the start time T _ST missing section, the end time of the missing section of the latest time _Obtained as T ED. Then, the missing section specifying unit 331 obtains the time length d _{from the start time T ST} to the end time T _{ED of the specified missing section.}

Next, the missing section specifying unit 331 determines whether or not the time length d of the obtained missing section is equal to or greater than the threshold value m. This threshold value m is preset with an arbitrary value (for example, '3' frame).
When the time length d is the threshold value m or more, the missing section specifying unit 331 outputs the obtained start time T _ST and end time T _ED of the missing section to the feature point detecting unit 333 together with the position data P.
On the other hand, when the time length d is less than the threshold value m, the missing section specifying unit 331 writes the position data P in the position data storage unit 351.

When the time length d of the missing section is equal to or greater than the threshold m, the feature point detection unit 333 acquires the detection positions immediately before the start time and immediately after the end time from the position data P, and based on the acquired detection positions, the start time and the end time. It sets the image area to the time of the visual image I _t, and detects a feature point from an image region set. The feature point detection unit 333, feature point detection unit 333, the position data P from the missing section identifying unit 331 is input, a visible image I _t is inputted from the visible and infrared same optical axis camera 20.

First, the feature point detection unit 333 obtains the _{time T ST-1} immediately before the start from _{the start time T ST of the missing section.} The time immediately before the start T _ST-1 represents a time (time code) immediately before the start time T _ST. Further, the feature point detection unit 333 obtains the _{time T ED + 1} immediately after the end from _{the end time T ED of the missing section.} The time T _{ED + 1} immediately after the end represents the time (time code) immediately after the end time T _ED.

Here, in the infrared image H at the time immediately before the start T _ST-1 and the time immediately after the end T _{ED + 1} , the sword tip 90 is successfully detected. Therefore, in the visible image _{I ST} start time _{T ST,} there is a high possibility that the loop-taker point 90 is present around the detection position immediately before the start time _{T ST-1 (x ST-} 1, y ST-1). Similarly, also in the visible image _{I ED} end time _{T ED,} it is likely that there is a point of a sword 90 around the detection position of the end immediately after time _{T ED + 1 (x ED +} 1, y ED + 1).

Therefore, the feature point detection unit 333 acquires the detection position (x _ST-1 , y _ST-1 ) from the position data P _{ST-1 at the time immediately before the start T ST-1} , and the detection position (x _ST-1 ,. The image area (search area) based on _{y ST-1} ) is set in the _{visible image IST.} This image area can be set in any shape and size centered on the detection position, and is, for example, a rectangular area of 160 pixels in length × 160 pixels in width.
The feature point detection unit 333 obtains the detected position from the position data _{P ED + 1} of immediately after the end time _{T ED + 1 (x ED +} 1, y ED + 1), an image area the detected position _{(x ED + 1, y ED} + 1) as a reference Set to visible image I _ED.

Then, feature point detection unit 333 in the image area, using the equation (2) below, a visible image I _t, of the previous frame difference image S _t of a visible image I _t-1 Generate. That is, the feature point detection unit 333, the luminance value ^{I xy} _t of a pixel of the visible image _{I t (x t, y t} ), the visible image _{I t-1} pixels _{(x t-1, y t} -1) A difference image S ^xy _{t in} which the difference from the luminance value I ^xy _t-1 exceeds the preset threshold value R _bri is extracted.

When the equation (2) is _{applied to the visible image IST} , t = ST, and when the equation (2) is _{applied to the visible image EED} , t = ED. Further, the threshold value R _bri is set in advance with an arbitrary value.
The feature point detection unit 333, similarly to the infrared light detecting section 311, subjected to opening morphological processing to the differential image S _t, may be removed Noizuburobu subregion.

By detecting the feature point from the difference image S _t, the case of a fixed camera image, it is possible to suppress the occurrence of Noizuburobu stationary. For moving the camera image, since the difference due to the movement of the camera occurs, the effect is reduced to utilize the difference image S _t. In that case, the feature point detection unit 333 may create an edge image and detect the feature point from an image region having a high edge value. Further, the feature point detection unit 333 may detect the feature point by using the difference value, the edge value or the luminance value as it is without performing binarization.

Next, the feature point detection unit 333 detects the feature points from the difference image ^Sxy _{ST set in the visible image IST} , and from the detected feature points, the infrared light region in the _{time T ST-1 immediately before the start} The feature point closest to the detection position (x _ST-1 , y _ST-1 ) is acquired as the feature point at the start time T _ST.
Further, the feature point detection unit 333 detects the feature points from the difference image ^Sxy _{ED set in the visible image I ED} , and among the detected feature points, the detection position of the infrared light region at the _{time T ED + 1 immediately after the end (} The _{feature point closest to x ED + 1} , y _{ED + 1} ) is acquired as the feature point of the end time T _ED.
After that, the feature point detection unit 333 outputs the positions of the acquired feature points of the start time T _ST and the end time T _ED to the position data interpolation unit 335.

SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and the like are typical examples of the feature point detection method described above, but what kind of feature point detection method is used by the feature point detection unit 333. May be. In this embodiment, the FAST (Features from Accelerated Segment Test) features described in the following references will be used.

References: Edward Rosten and Tom Drummond, "Machine learning for high-speed corner detection," In Proc. Of European Conference on Computer Vision (ECCV2006), pp.430-443, 2006.

<FAST feature amount>
This FAST feature amount targets only the corners as the feature points, and detects the feature points quickly and efficiently by the decision tree. Here, it will be described as detecting the FAST feature amount from the visible image I of FIG. 5 (a). In the FAST feature amount, as shown in FIG. 5B, 16 pixels around the pixel of interest p are observed. Then, in the FAST feature amount, when n or more pixel values of 16 observation pixels are continuously brightened or darkened by a threshold value t or more as compared with the pixel value of the pixel of interest p, the pixel of interest p Is detected as a corner (however, n is a natural number of 1 or more).
In FIG. 5B, the pixel of interest of the visible image I is shown by a numerical value of p, and the observed pixel is shown by a numerical value of 1 to 16.

In the FAST feature quantity, a decision tree is used to detect feature points with high reproducibility at high speed and efficiently. As shown in the following equation (3), the 16 observation pixels are classified into three values of bright (brighter), similar (similar), and dark (darker).
In the equation (3), I _p represents the luminance value of the pixel of interest p, x represents the position of the observed pixel, I _{p → x} represents the luminance value of the observed pixel, and t represents the threshold value.

The observation pixels classified into three values are used as a feature vector, and when the pixel values of the observation pixels are continuously n or more and the conditions of bright (brighter) or dark (darker) are satisfied, they are detected as corners. On the other hand, when this condition is not satisfied, it is treated as a non-corner. The decision tree is constructed in this way, and the process of recursively selecting the observation pixels on the circle that can optimally classify the corners and non-corners as branch nodes is performed to obtain the entire tree structure.

In many feature point detection methods, only the effective pixels among the adjacent feature points are detected by extracting the local maximum value of the response value representing the feature point likeness. However, since the response value is not extracted in the FAST feature amount, the response value V is calculated by the following equation (4). By setting the pixel having the highest response value V among the adjacently detected corner points as the corner point, the corner points separated from each other can be detected.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
The position data interpolation unit 335 interpolates _{the position data P of the start time T ST} and the end time T _ED at the detection position of the feature point input from the feature point detection unit 333. That is, the position data P in which the detection position is missing is updated at the position of the feature point detected by the feature point detection unit 333.

Subsequently, the position data interpolation unit 335 outputs the interpolated position data P to the missing section specifying unit 331.
Then, the missing section specifying unit 331 again specifies the missing section from the position data P input from the position data interpolation unit 335. After that, the visible image analysis means 33 repeats the above-mentioned process.

<Interpolation of position data>
With reference to FIG. 4, interpolation of position data by the visible image analysis means 33 will be described as a specific example (see FIG. 3 as appropriate).
Here, it is assumed that the position data P of FIG. 4A is input and the threshold value m is the '3' frame.

First, the first interpolation in the position data P of FIG. 4A will be described.
The missing section specifying unit 331 refers to each line of the position data P, and specifies the 5th to 9th lines in which the image coordinates (-1, -1) are continuous as the missing section. Then, the missing section identification unit 331, out of line 5 to 9 line of the missing section, the most ahead of time "10: 25: 30.29" and the start time _{T ST,} the slowest time "10:25: 31.03 "seek as the end time _{T ED.}

Next, the missing section specifying unit 331 sets _{the time length d from the start time T ST} “10: 25: 30.29” _{of the missing section to the end time T ED} “10: 25: 31.03” to '5' frames. Is calculated, and the threshold value is determined between the time length d of the missing section and the threshold value m. Here, since the time length d = '5' is equal to or greater than the threshold value m = '3' in the missing section specifying unit 331, the position data P, the start time T _ST “10: 25: 30.29”, and the end time T _ED "10:25: 31.03" is output to the feature point detection unit 333.

Feature point detector 333, the start time _{T ST} starts immediately before time "10: 25: 30.29" is one frame before _{T ST-1} "10: 25: 30.28" is calculated. The feature point detection unit 333, end time _{T ED} "10: 25: 31.03" is after 1 frame immediately after the end time _{T ED + 1} "10: 25: 31.04" is calculated. Then, the feature point detection unit 333 _{acquires the image coordinates (1255, 344) from the position data P at the time T ST-1} immediately before the start, and obtains the image coordinates (1265, 331) from the position data P at the time T _{ED + 1 immediately after the end.} get.

Then, feature point detection unit 333 sets the image region around the image coordinates (1255,344) into a visible image _{I ST} start time _{T ST,} detects feature points from the image area. Further, the feature point detection unit 333 sets an image area centered on the image coordinates (1265, 331) in the visible image I _{ED at the end time T ED, and detects the feature points from this image area.}

Position data interpolation section 335, the position data P of FIG. 4 (a), the start time _T image coordinates (-1, -1) of the _ST is updated with image coordinates of the feature point (1256,343). The position data interpolation unit 335 updates the image coordinates of the end time _{T ED} (-1, -1) image coordinates of feature points (1264,331).

Subsequently, the second interpolation in the position data P of FIG. 4B will be described.
The missing section specifying unit 331 refers to each line of the position data P, identifies the 6th to 8th lines in which the image coordinates (-1, -1) are continuous as the missing section, and specifies the start time _TST ". 10:25:31.00 ”and the end time T _ED “ 10:25: 31.02 ”are obtained.

Next, in the missing section specifying unit 331, since the time length d = '3' of the missing section is equal to or greater than the threshold value m = '3', the position data P, the start time T _ST "10: 25: 31.00", and the end The time T _ED "10:25: 31.02" is output to the feature point detection unit 333.

The feature point detection unit 333 calculates the time immediately before the start T _ST-1 “10:25: 30.29” and the time immediately after the end T _{ED + 1} “10:25: 31.03”. Then, the feature point detection unit 333 _{acquires the image coordinates (1256, 343) from the position data P at the time T ST-1} immediately before the start, and obtains the image coordinates (1264, 331) from the position data P at the time T _{ED + 1 immediately after the end.} get.

Then, feature point detection unit 333 sets the image region around the image coordinates (1256,343) into a visible image _{I ST} start time _{T ST,} detects feature points from the image area. Further, the feature point detection unit 333 sets an image area centered on the image coordinates (1264, 331) in the visible image I _{ED at the end time T ED, and detects the feature points from this image area.}

Position data interpolation section 335, the position data P of FIG. 4 (b), the start time _{T ST} and end time _{T ED} image coordinates (-1, -1), and the feature point by the feature point detection unit 333 detects Update with image coordinates.

Thus, a visible image analysis means 33, the visible image I _ED visual image I _ST and end time T _ED detectable starting time T _ST objects with high probability to detect an object, gradually the position data P By narrowing the missing section, accurate position data P can be generated.

Although it has been described that the position data P in FIG. 4 has one missing section, the position data P may include a plurality of missing sections. In this case, the visible image analysis means 33 may apply the above-mentioned processing to each missing section.
Needless to say, the position data is not limited to the example of FIG.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
TC linked CG synthesizing unit 35 as a trigger time code T _t of the visible image I _t input from the visible and infrared same optical axis camera 20, from the stored position data P to obtain the position coordinates of the object trajectory draw a, it is to synthesize drawing trajectory into a visible image I _t.

Normally, in a playback slow VTR, a live video or a CG video generated in real time is recorded on the VTR, rewound at the timing of use, and then played. However, when the CG image cannot be generated in real time, it is necessary to interpolate the position data P and then reproduce the visible image I recorded on the VTR to draw a locus. In this case, a certain amount of working time is required and efficiency is impaired. Therefore, it is preferable to draw a locus at a timing synchronized with the reproduction of the video recorded on the VTR and synthesize the CG on the spot.

As shown in FIG. 3, the TC-linked CG synthesizing means 35 includes a position data storage unit 351 and a locus drawing unit (object tracking unit) 353.
The position data storage unit 351 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or memory for storing position data P. This position data P is written by the missing section specifying unit 331 and referred to by the locus drawing unit 353.

Locus drawing unit 353 reads the position data P _t corresponding to the time information of the visual image I _t input from the visible and infrared same optical axis camera 20 from the position data storage unit 351. Then, the locus drawing section 353, pixels on the read position data _{P t (x t, y t} ) to draw a trajectory of the hook point 90 to an external drawing trajectory and visible images _{I t} the combined locus-synthesized image _{F t} It is output to.

For example, the locus drawing unit 353, a locus _C 1, _{C 2} of the loop-taker point 90 of the two players drawn in different colors, a locus _C 1, _{C 2} drawn by synthesizing the visual image _{I t,} locus-synthesized image F Generate _t. Since the position data P even if failure to detect the point of a sword 90 of the left player is interpolated trajectory drawing unit 353, as shown in FIG. 6, draw a locus C ₁ of the loop-taker point 90 of the left player can.
In FIG. 6, the rectangle β represents the image area set by the feature point detection unit 333, and the circle γ represents the sword tip 90 detected by the infrared light detection unit 311.

[Operation of object tracking device]
The operation of the object tracking device 30 will be described with reference to FIG. 7 (see FIG. 3 as appropriate).
As shown in FIG. 7, the infrared photodetector 311 generates a binary infrared difference image and performs morphology processing on the extracted candidate blobs (step S1).

The infrared photodetection unit 311 performs labeling treatment on the candidate blobs remaining in the morphology treatment, and obtains the position, area, and shape feature amount of the candidate blobs. Then, the infrared light detecting means 31 filters based on the area and the shape feature amount, and detects the detection blob (infrared light region) (step S2).
The position data generation unit 313 generates position data in which the detection position in the infrared light region and the time information of the infrared image are associated with each other (step S3).

_{The missing section specifying unit 331 acquires the start time T ST} and the end time T _ED of the missing section from the position data, and obtains the time length d from the start time T _ST to the end time T _ED (step S4).
The missing section specifying unit 331 determines whether or not the time length d of the missing section is equal to or greater than the threshold value m (step S5).

When the time length d is the threshold value m or more (Yes in step S5), the feature point detection unit 333 acquires the detection positions of the _{time T ST-1} immediately before the start and the time T _{ED + 1 immediately after the end from the position data (step S6).} ..
The feature point detection unit 333 sets an image area in the visible image based on the detection positions of the _{time T ST-1} immediately before the start and the time T _{ED + 1 immediately after the end (step S7).}

The feature point detection unit 333 detects feature points in the image area set in the visible image (step S8).
The position data interpolation unit 335 interpolates the position data at the feature point closest to the past detected position (step S9), and returns to the process of step S4.

When the time length d is less than the threshold value m (No in step S5), the missing section specifying unit 331 writes the position data in the position data storage unit 351.
The locus drawing unit 353 reads out the position data corresponding to the time information of the visible image input from the visible / infrared optical axis camera 20 from the position data storage unit 351 (step S10).
The locus drawing unit 353 draws the locus of the object according to the detection position of the read position data, and synthesizes the drawn locus into the visible image (step S11).

[Action / Effect]
The object tracking device 30 according to the embodiment of the present invention _{detects a feature point from the visible images of the start time T ST} and the end time T _ED where the sword tip 90 is likely to exist, and position data at the detection position of the feature point. Interpolate. As a result, the object tracking device 30 can accurately draw the trajectory of the sword tip 90 even if the detection position is missing.

Further, the object tracking device 30 can reliably track the sword tip 90 moving at high speed and draw an accurate trajectory without attaching a sensor to the sword tip 90. That is, the object tracking device 30 can visualize the movement of a high-speed moving object that is difficult to see with the naked eye or on a normal camera image. The object tracking device 30 can visually express the movement of the fencing sword tip 90, which moves at a high speed that cannot be confirmed by the naked eye. As a result, the object tracking device 30 can draw a trajectory of the sword tip 90 in fencing, and can improve the quality of the trajectory drawing.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like within a range that does not deviate from the gist of the present invention.
The method of using the position data interpolated by the object tracking device is not particularly limited, and the position data can be used for purposes other than drawing a trajectory.

In the above-described embodiment, the object tracking device has been described as inputting a visible image and an infrared image from a visible / infrared optical axis camera, but the object tracking device is not limited thereto. That is, the object tracking device may be arranged after the other image processing device, and a visible image or an infrared image that has failed to be tracked by the other image processing device may be input.

In the above-described embodiment, fencing has been described as an example, but the application target of the object tracking device is not limited to this. That is, the object tracking device can also be applied to sports such as tennis, badminton, and volleyball.

In addition, the object tracking device can be applied to sports where the positions of athletes are switched if the trajectory is not drawn in different colors. For example, an object tracking device can track a badminton shuttle and draw its trajectory. In addition, the object tracking device can draw the trajectory of a baton in an orchestra and the trajectory of a sword or the like in a drama or a movie.

The object tracking device is not particularly limited in the moving speed of the object to be tracked, and in particular, can reliably track the moving speed at which motion blur occurs. This moving speed largely depends on the distance between the visible / near-infrared optical axis camera and the object, the shooting angle of view and sensitivity of the visible / near-infrared optical axis camera, and the amount of noise. For example, when the time resolution is 30 frames / sec, the motion blur increases when the moving speed exceeds 100 km / h.

In the above-described embodiment, the object tracking device has been described as independent hardware, but the present invention is not limited thereto. For example, the object tracking device can also be realized by an object tracking program that cooperates with the hardware resources such as the CPU, memory, and hard disk of the computer as the above-mentioned means. This program may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory.

The present invention can be used for video analysis of sports scenes such as fencing. The present invention can also be used in the production of movies, games and dramas. Furthermore, the present invention can also be used in industry and security systems.

1 Object tracking system 10 Infrared light floodlight 20 Visible / infrared same light axis camera 30 Object tracking device 31 Infrared light detection means 311 Infrared light detection unit 313 Position data generation unit 33 Visible image analysis means 331 Missing section identification unit 333 Feature point detection unit 335 Position data interpolation unit 35 Time code interlocking CG synthesis means (TC interlocking CG synthesis means)
351 Position data storage unit 353 Trajectory drawing unit (object tracking unit)
90 Sword tip 91 Reflective tape (infrared light marker)

Claims (4)

An object tracking device that tracks an object using an infrared image of one or more moving objects with an infrared light marker taken with infrared light and a visible image of the object taken with visible light. hand,
An infrared light detection unit that detects the region of the infrared light marker as an infrared light region from the infrared image,
When the infrared light region can be detected, the detection position of the infrared light region or when the infrared light region cannot be detected, the detection position is missing and the infrared region to be detected is detected. A position data generator that generates position data including the time information of the image,
A missing section specifying unit that identifies a missing section of the position data and determines whether or not the time length from the start time to the end time of the specified missing section is equal to or longer than a preset threshold value.
When the time length of the missing section is equal to or longer than the threshold value, the detection position is acquired from the position data immediately before the start time and immediately after the end time, and the image area is set to the start time and the end based on the acquired detection position. A feature point detection unit that sets a visible image of the time and detects feature points from the set image area,
A position data interpolation unit that interpolates the position data of the start time and the end time at the detection position of the feature point and outputs the interpolated position data to the missing section identification unit.
When the time length of the missing section is less than the threshold value, an object tracking unit that tracks the object based on the position data is provided.
The missing section specifying unit is an object tracking device characterized by specifying a missing section of position data input from the position data interpolation unit. A position data storage unit for accumulating the position data is further provided.
When the time length of the missing section is less than the threshold value, the missing section specifying unit writes the position data to the position data storage unit.
As the object tracking unit, position data corresponding to the time information of the visible image is acquired from the position data storage unit, the locus of the object is drawn based on the detected position of the acquired position data, and the drawn locus is drawn. The object tracking device according to claim 1, further comprising a locus drawing unit for synthesizing the visible image. The object tracking device according to claim 1 or 2, wherein the feature point detecting unit detects a FAST feature amount from the image area. An object tracking program for operating a computer as the object tracking device according to any one of claims 1 to 3.

Images