JP6772593B2 - Mobile identification device, mobile identification system, mobile identification method and program - Google Patents

Description

The present invention relates to a moving body identification device, a moving body identification system, a moving body identification method and a program.

In order to track the position of a plurality of moving bodies and analyze the motion, a technique is known in which a plurality of moving bodies are imaged and the plurality of moving bodies are identified from the captured moving images. Using a moving image has an advantage that the positions of a plurality of moving objects and the surrounding conditions can be easily grasped. On the other hand, there is a demerit that it may be difficult to identify each moving body, such as when the appearances of a plurality of moving bodies are similar, or when the moving bodies overlap each other or the moving bodies are hidden by obstacles. ..

In order to make up for such a demerit, a method is known in which a measuring device is attached to each of a plurality of moving bodies, and measurement by the measuring device and imaging of a moving image are combined to identify the plurality of moving bodies. By using the wearable measuring device, the corresponding moving body is specified, and quantitative movement information of the moving body can be acquired.

For example, Patent Document 1 discloses a system for detecting the position of a moving infant in a kindergarten or a nursery school. The position detection system disclosed in Patent Document 1 detects the position of each infant by photographing with a camera, and detects the amount of movement of each infant by an acceleration sensor attached to each infant. Then, this position detection system collates the movement amount of each infant detected by the acceleration sensor with the position of each infant detected by the image taken by the camera, and identifies the position of each infant individually while identifying each infant individually. To detect.

Further, Patent Document 2 discloses a moving object identification system that tracks the position of an infant, which is a moving object, while recognizing who the infant is. The moving body identification system disclosed in Patent Document 2 detects the vertical movement and the horizontal movement of the moving body by two cameras, and the sensor attached to the moving body detects the vertical movement of the moving body in the vertical direction of the moving body. And horizontal acceleration is detected. Then, this moving body identification system collates the acceleration obtained by the sensor with the movement of the moving body obtained by the camera in each of the vertical direction and the horizontal direction, and selects the moving body having the highest degree of matching. To do.

Japanese Unexamined Patent Publication No. 2004-96501 Japanese Unexamined Patent Publication No. 2004-274101

In the technique for identifying a plurality of moving bodies by associating the plurality of moving bodies in the moving image with the plurality of measuring devices attached to the plurality of moving bodies as described above, the correspondence may be erroneous. Therefore, it is desired to reduce the error of association.

The present invention is for solving the above problems, and when associating at least one moving body in a moving body with at least one measuring device attached to at least one moving body, an error in association is made. It is an object of the present invention to provide a mobile body identification device, a mobile body identification system, a mobile body identification method and a program capable of reducing the number of mobile bodies.

In order to achieve the above object, the uniform state of the moving body identification device according to the present invention is
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. a second time series feature relating to movement of at least one mobile, a specifying unit which specifies a relationship between said at least one mobile and at least one measuring device in the moving image based on,
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the movement information measured by the at least one measuring device, and
With
The particular means, in said at least one measuring device the movement information time interval satisfies a predetermined condition which is determined by either measuring apparatus of was measured by the one of the measuring device The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
It is characterized by that.

In addition, in order to achieve the above object, one aspect of the mobile identification system according to the present invention is
The moving body identification device described above, the imaging device, and at least one measuring device are provided.
It is characterized by that.

Further, in order to achieve the above object, the uniform state of the moving body identification method according to the present invention is
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on a second time-series feature relating to the movement of at least one mobile body, a specific step of identifying the correspondence between the at least one mobile body and the at least one measuring device in the moving image.
A position estimation step for estimating the position of the at least one moving object in the vertical direction from the movement information measured by the at least one measuring device, and
Including
Wherein in the specifying step, in the at least one measuring device the movement information time interval satisfies a predetermined condition which is determined by either measuring apparatus of was measured by the one of the measuring device The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
It is characterized by that.

In addition, in order to achieve the above object, the uniform state of the program according to the present invention is
The computer of the mobile identification device,
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. A specific means for identifying a correspondence between the at least one mobile body and the at least one measuring device in the moving image based on a second time-series feature relating to the movement of the at least one mobile body.
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the movement information measured by the at least one measuring device.
To function as
The particular means, in said at least one measuring device the movement information time interval satisfies a predetermined condition which is determined by either measuring apparatus of was measured by the one of the measuring device The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
It is characterized by that.

According to the present invention, when associating at least one moving body in a moving body with at least one measuring device attached to at least one moving body, a moving body identification device, which can reduce an error in matching, moves. A body identification system, a mobile identification method and a program can be provided.

It is a figure which shows the outline of the mobile body identification system which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the scene where a mobile body identification system is applied. It is a block diagram which shows the hardware composition of a plurality of measuring apparatus, image pickup apparatus, and moving body identification apparatus. It is a figure which shows the example of the moving image of a plurality of moving bodies imaged by an image pickup apparatus. It is a block diagram which shows the functional structure of the moving body identification apparatus. (A) is a figure which shows an example of the time change of the speed obtained from the moving image. (B) and (c) are diagrams showing an example of the time change of the dispersion of acceleration obtained from the measuring device. (D) is a figure which shows an example of the 1st correlation value. (A) is a figure which shows an example of the time change of the acceleration obtained from the moving image. (B) and (c) are diagrams showing an example of the time change of the acceleration in the vertical direction obtained from the measuring device. (D) is a figure which shows an example of the 2nd correlation value. (A) is a figure which shows an example of time change of the direction of movement obtained from the moving image. (B) and (c) are diagrams showing an example of the time change of the direction of movement obtained from the measuring device. (D) is a figure which shows an example of the 3rd correlation value. It is a figure which shows the example of the time change of the acceleration in the vertical direction when a moving body jumps. It is a figure which shows the display example of the moving image which was displayed by combining the identification information of the measuring device corresponding to each of a plurality of moving bodies. It is a flowchart which shows the flow of the moving body identification processing executed by the moving body identification apparatus. It is a flowchart which shows the flow of the specific processing executed in the moving body identification processing.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals.

(Embodiment 1)
FIG. 1 shows an outline of the mobile identification system 1 according to the first embodiment. The mobile body identification system 1 is a system that identifies a plurality of mobile bodies 10a, 10b, 10c ... In a moving image of a plurality of mobile bodies 10a, 10b, 10c ... Imaged by the imaging device 200. Each of the plurality of moving bodies 10a, 10b, 10c ... Is a movable object in the space of the real world, and is a human being in the example of FIG.

As shown in FIG. 1, the moving body identification system 1 includes a plurality of measuring devices 100a, 100b, 100c ..., an imaging device 200, and a moving body identification device 300. The plurality of measuring devices 100a, 100b, 100c ... Are wearable type measuring devices mounted on the plurality of moving bodies 10a, 10b, 10c ..., respectively, and the movement and movement of the plurality of moving bodies 10a, 10b, 10c ... It is equipped with a motion sensor that measures.

In the following, when the plurality of moving bodies 10a, 10b, 10c ... Are referred to without distinction, they are collectively referred to as the moving body 10, and the plurality of measuring devices 100a, 100b, 100c ... Are not distinguished from each other. In the case of, it is generically referred to as a measuring device 100.

FIG. 2 shows an example of a scene in which the mobile identification system 1 is applied. In the example of FIG. 2, six athletes, each of which is a mobile body 10a to 10f, are playing an athletic competition on the court. Athletics are sports played indoors, such as basketball, handball, volleyball or futsal. Each of these six athletes (moving bodies 10a to 10f) is equipped with one measuring device 100a to 100f. Each of the measuring devices 100a to 100f measures the movement of the athlete (moving body 10) equipped with the measuring device 100 during the competition. Each of the measuring devices 100a to 100f is attached to the trunk of each athlete (each moving body 10a to 10f) so that the movement and movement of the athlete can be accurately measured.

The image pickup apparatus 200 images a plurality of moving bodies 10a to 10f and generates moving images of the plurality of moving bodies 10a to 10f. The image pickup device 200 is, for example, a video camera, a camera built in a mobile terminal such as a smartphone, or the like. The moving image generated by the image pickup apparatus 200 includes a plurality of images captured in frame units. The image pickup apparatus 200 is installed at a position diagonally above the plurality of moving bodies 10a to 10f (audience seats in the example of FIG. 2) so that all of the plurality of moving bodies 10a to 10f are in the field of view. The image pickup apparatus 200 can capture both the movement of the plurality of moving bodies 10a to 10f in the horizontal direction and the movement in the vertical direction by taking a bird's-eye view from diagonally above.

As shown in FIG. 2, the world coordinate system is set so that the xy plane corresponds to the horizontal plane and the z direction corresponds to the vertical direction in the space where the plurality of moving bodies 10a to 10f move. There is.

Next, the configuration of the mobile identification system 1 will be described with reference to FIG. Each of the plurality of measuring devices 100 includes a control unit 101, a storage unit 102, a measuring unit 103, and a communication unit 104.

The control unit 101 includes a CPU (Central Processing Unit) and is a central processing unit that executes various processes and operations. The control unit 101 is connected to each unit of the measuring device 100 via a system bus which is a transmission path for transferring commands and data, and controls the entire measuring device 100.

The storage unit 102 includes a RAM (Random Access Memory) that functions as a work memory of the control unit 101, and a non-volatile memory such as a ROM (Read Only Memory) or a flash memory. The storage unit 102 stores various programs and data used by the control unit 101 to perform various processes. In addition, the storage unit 102 stores various data generated or acquired by the control unit 101 performing various processes.

The measuring unit 103 measures the movement information of the moving body 10 to which the measuring device 100 is mounted, which is the measurement target. Here, the movement information is information on the movement of the moving body 10, such as the speed, acceleration, or angular velocity of the moving body 10. Specifically, the measuring unit 103 includes an acceleration sensor (inertia sensor) and measures the movement and vibration of the moving body 10 by detecting the acceleration applied to the measuring device 100. Further, the acceleration sensor measures the inclination of the measuring device 100 by detecting gravity and specifying its direction (vertically downward). The measuring unit 103 further includes a geomagnetic sensor that detects the direction, and an angular velocity sensor (gyro sensor) that detects the rotation (angular velocity) of the measuring device 100.

The communication unit 104 is provided with an interface for wired or wireless communication with the mobile identification device 300 under the control of the control unit 101. The communication unit 104 communicates with the mobile identification device 300 via a wireless LAN (Local Area Network) such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), USB (Universal Serial Bus), or the like. Specifically, the communication unit 104 transmits the movement information of the moving body 10 to which the measuring device 100 is mounted, which is obtained by the measurement unit 103, to the moving body identification device 300.

Next, the description of the configuration of the image pickup apparatus 200 will be given. As shown in FIG. 3, the image pickup apparatus 200 includes a control unit 201, a storage unit 202, an image pickup unit 203, an image processing unit 204, and a communication unit 205.

The control unit 201 is a central processing unit that includes a CPU and executes various processes and operations. The control unit 201 is connected to each unit of the image pickup apparatus 200 via a system bus which is a transmission path for transferring instructions and data, and controls the entire image pickup apparatus 200.

The storage unit 202 includes a RAM that functions as a work memory of the control unit 201, and a non-volatile memory such as a ROM or a flash memory. The storage unit 202 stores various programs and data used by the control unit 201 to perform various processes. In addition, the storage unit 202 stores various data generated or acquired by the control unit 201 performing various processes.

The imaging unit 203 has a function of acquiring a still image and a moving image by imaging. The image pickup unit 203 is arranged at a lens that collects the light flux emitted from the subject (object) and a light collection position by the lens, and acquires an optical image of the subject as an electric signal by photoelectric conversion (image sensor). Etc. The image pickup device is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

The image processing unit 204 includes an A / D (Analog / Digital) converter that converts data indicating a captured image of a subject sent as an electric signal from the image sensor into digital data. The image processing unit 204 generates digital data of the moving image captured by the image capturing unit 203 under the control of the control unit 201. Further, the image processing unit 204 includes a frame memory such as VRAM (Video RAM), a processor (image processing engine) for image processing such as a GPU (Graphics Processing Unit), and the like. The image processing unit 204 has a function of executing processing processing on the moving image captured by the imaging unit 203 under the control of the control unit 201, and image recognition such as object recognition processing and face recognition processing on the moving image. It has a function to execute processing.

The communication unit 205 includes an interface for wired or wireless communication with the mobile identification device 300 under the control of the control unit 201. The communication unit 205 communicates with the mobile identification device 300 via a wireless LAN such as Wi-Fi, Bluetooth (registered trademark), USB, or the like. Specifically, the communication unit 205 transmits the moving image data obtained by imaging the plurality of moving objects (players) 10 by the imaging unit 203 to the moving object identification device 300.

FIG. 4 shows an example of the moving image 5 captured by the imaging device 200. Specifically, FIG. 4 shows an image of one frame included in the moving image 5 captured by the imaging device 200. The image pickup apparatus 200 captures the entire movement of the plurality of moving bodies 10a to 10f as shown in FIG. 4 by imaging the plurality of moving bodies 10a to 10f playing an athletic competition on the court from diagonally above. Generate the captured moving image 5.

Next, the description of the configuration of the moving body identification device 300 will be given. As shown in FIG. 3, the moving body identification device 300 includes a control unit 301, a storage unit 302, an image processing unit 303, an operation unit 304, a display unit 305, and a communication unit 306.

The control unit 301 is a central processing unit that includes a CPU and executes various processes and operations. The control unit 301 is connected to each unit of the mobile body identification device 300 via a system bus which is a transmission path for transferring instructions and data, and controls the entire mobile body identification device 300.

The storage unit 302 includes a RAM that functions as a work memory of the control unit 301, and a non-volatile memory such as a ROM or a flash memory. The storage unit 302 stores various programs and data used by the control unit 301 to perform various processes. In addition, the storage unit 302 stores various data generated or acquired by the control unit 301 performing various processes.

The image processing unit 303 includes a frame memory such as VRAM, a processor (image processing engine) for image processing such as GPU, and the like. Under the control of the control unit 301, the image processing unit 303 executes image processing such as background subtraction method or feature point tracking on the moving image 5 captured by the image pickup device 200.

The operation unit 304 includes an input device such as an input key, a button, a switch, a touch pad, or a touch panel. The operation unit 304 receives various operation instructions input from the user and transmits the received operation instructions to the control unit 301.

The display unit 305 includes a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 305 acquires various information such as moving image data received from the image pickup apparatus 200 from the control unit 301, and displays an image showing the acquired information. The display unit 305 may be arranged so as to overlap with the operation unit 304, and the display unit 305 and the operation unit 304 may form a so-called touch panel (touch screen). The display unit 305 functions as a display means for displaying the moving image 5 captured by the image pickup device 200.

The communication unit 306 includes an interface for wired or wireless communication with each of the plurality of measuring devices 100 and the imaging device 200 under the control of the control unit 301. The communication unit 306 communicates with each of the plurality of measuring devices 100 and the imaging device 200 via a wireless LAN such as Wi-Fi or Bluetooth (registered trademark). Specifically, the communication unit 306 receives the movement information of the plurality of moving bodies (players) 10 from the plurality of measuring devices 100, and receives the moving image data of the plurality of moving bodies (players) 10 from the imaging device 200. ..

Next, the functional configuration of the moving body identification device 300 will be described with reference to FIG. As shown in FIG. 5, the moving body identification device 300 functionally includes a receiving unit 311, a first feature acquisition unit 312, a second feature acquisition unit 313, a calculation unit 314, and a position estimation unit 315. A specific unit 316 and a display control unit 317 are provided. The control unit 301 functions as each of these units by reading the program stored in the ROM into the RAM, executing the program, and controlling the program.

The receiving unit 311 receives the movement information from each of the plurality of measuring devices 100, and also receives the moving image 5 from the imaging device 200. The movement information received by the receiving unit 311 is the movement information of the plurality of moving bodies 10a to 10f measured by the plurality of measuring devices 100a to 100f mounted on the plurality of moving bodies 10a to 10f, respectively. The moving image 5 received by the receiving unit 311 is a moving image 5 obtained by the image pickup apparatus 200 imaging a plurality of moving mobile bodies 10a to 10f.

Each of the plurality of measuring devices 100 transmits the movement information measured by the measuring device 100 to the moving body identification device 300 according to the instruction received from the user or in response to the request from the moving body identification device 300. The receiving unit 311 receives the movement information transmitted from each of the plurality of measuring devices 100 in this way via the communication unit 306. Further, the image pickup device 200 transmits the moving image 5 captured by the image pickup device 200 to the moving body identification device 300 according to the instruction received from the user or in response to the request from the moving body identification device 300. The receiving unit 311 receives the moving image data indicating the moving image 5 transmitted from the imaging device 200 in this way via the communication unit 306. The receiving unit 311 is realized by the control unit 301 cooperating with the communication unit 306. The receiving unit 311 functions as a receiving means.

The first feature acquisition unit 312 obtains the first time-series feature relating to the movement of the plurality of mobile bodies 10a to 10f from the moving image 5 in which the plurality of mobile bodies 10a to 10f are captured received by the receiving unit 311. get. The first time-series feature is an index showing how the positions, velocities, accelerations, directions, and the like of the plurality of moving bodies 10a to 10f in the moving image 5 change with time. The first feature acquisition unit 312 is realized by the control unit 301 collaborating with the image processing unit 303. The first feature acquisition unit 312 functions as the first feature acquisition means.

In order to acquire the first time-series features, the first feature acquisition unit 312 detects the positions of the plurality of moving objects 10a to 10f in the moving image 5 by using the background subtraction method. Specifically, the first feature acquisition unit 312 takes a difference between each of the plurality of images included in the moving image 5 and the background image prepared in advance, so that the plurality of moving objects 10a to 10a in each image are taken. A foreground object region that is a candidate for the region of 10f is extracted. The background image is an image captured by the image pickup apparatus 200 in a state where none of the plurality of moving bodies 10a to 10f is in the field of view.

The first feature acquisition unit 312 organizes the foreground object region into several regions by executing expansion / contraction processing and labeling processing on the extracted foreground object region. The first feature acquisition unit 312 detects a plurality of regions collected in this way as mobile regions. The moving body region is a region where the moving body 10 is estimated to be located in each image. The first feature acquisition unit 312 executes the process of detecting such a moving body region for the image of each frame included in the moving image 5.

When the positions of the plurality of moving bodies 10a to 10f in each frame are detected by detecting the moving body region, the first feature acquisition unit 312 associates the moving bodies 10 between consecutive frames. Specifically, the first feature acquisition unit 312 is the image of the nth frame of the plurality of moving body regions detected in the image of the (n-1) frame (n is a natural number of 2 or more). The mobile region located closest to the position of the mobile region detected in is specified. Then, the first feature acquisition unit 312 sets the moving body region in the specified (n-1) frame image as the moving body region corresponding to the moving body region in the nth frame image, and these two movements. Connect body regions.

The first feature acquisition unit 312 performs such a connection process for each of the plurality of moving body regions detected in the image of the nth frame, and from the first frame to the last frame included in the moving image 5. Execute for each image. As a result, the first feature acquisition unit 312 acquires a locus indicating a change in the position in the moving image 5 for each of the plurality of moving bodies 10a to 10f.

Since the imaging device 200 images a plurality of moving bodies 10a to 10f from diagonally above, the coordinate system in the moving image 5 is tilted with respect to the world coordinate system. Therefore, the first feature acquisition unit 312 projects the position information of the acquired locus in the moving image 5 onto the horizontal plane (xy plane in FIG. 2) by the calibration information of the imaging device 200 prepared in advance. This calibration information is information for converting the coordinate system in the moving image 5 into the world coordinate system, which is determined by the position where the image pickup apparatus 200 is installed and the direction of the line of sight.

When the locus of the moving body region in the moving object 5 is acquired in this way, the first feature acquisition unit 312 acquires the following features (1) to (3) as the first time-series features.
(1) Speeds of a plurality of moving bodies 10a to 10f in the moving image 5.
(2) Acceleration of a plurality of moving bodies 10a to 10f in the moving image 5 in a direction other than the vertical direction.
(3) Direction of movement of the plurality of moving bodies 10a to 10f in the moving image 5.

First, the first feature acquisition unit 312 acquires the velocities of the plurality of moving bodies 10a to 10f in the moving body 5 by differentiating the positions of the plurality of moving body regions detected from the moving body 5. Specifically, the first feature acquisition unit 312 describes the difference between the position of the moving body region in the image of the nth frame and the position of the corresponding moving body region in the image of the (n-1) frame (n-1). Distance) is calculated, and the calculated difference is divided by the time for one frame. As a result, the first feature acquisition unit 312 acquires the velocity of the moving body region in the image of the nth frame. The first feature acquisition unit 312 performs such processing for each of the plurality of moving body regions detected in the image of the nth frame, and for each of the first frame to the last frame included in the moving image 5. Execute for the image. As a result, the first feature acquisition unit 312 acquires the magnitude of the velocity in the moving image 5 for each of the plurality of moving bodies 10a to 10f.

Secondly, the first feature acquisition unit 312 acquires the accelerations of the plurality of moving bodies 10a to 10f in the moving body 5 by differentiating the positions of the plurality of moving body regions detected from the moving body 5 twice. Therefore, the first feature acquisition unit 312 further differentiates the velocities of the plurality of moving body regions acquired as described above. Specifically, the first feature acquisition unit 312 describes the difference between the speed of the moving body region in the image of the nth frame and the speed of the corresponding moving body region in the image of the (n-1) frame (n-1). (Speed difference) is divided by the time for one frame. As a result, the first feature acquisition unit 312 acquires the acceleration of the moving body region in the image of the nth frame. The first feature acquisition unit 312 performs such processing for each of the plurality of moving body regions detected in the image of the nth frame, and for each of the first frame to the last frame included in the moving image 5. Execute for the image. As a result, the first feature acquisition unit 312 acquires the magnitude of the acceleration in the moving image 5 for each of the plurality of moving bodies 10a to 10f.

The image pickup apparatus 200 images the plurality of moving bodies 10a to 10f from diagonally above, not from the horizontal direction (right beside). Therefore, the acceleration of the plurality of moving bodies 10a to 10f taken from the moving image 5 as described above is not an acceleration in the vertical direction, but a direction other than the vertical direction (specifically, between the vertical direction and the horizontal direction). It is the acceleration in the diagonal direction).

Thirdly, the first feature acquisition unit 312 acquires the moving directions of the plurality of moving bodies 10a to 10f in the moving body 5 by acquiring the speed directions of the plurality of moving body regions detected from the moving body 5. To do. Specifically, the first feature acquisition unit 312 calculates the direction from the position of the moving body region in the image of the (n-1) frame to the position of the corresponding moving body region in the image of the nth frame. .. As a result, the first feature acquisition unit 312 acquires the direction of movement of the moving body region in the image of the nth frame. The first feature acquisition unit 312 performs such processing for each of the plurality of moving body regions detected in the image of the nth frame, and for each of the first frame to the last frame included in the moving image 5. Execute for the image. As a result, the first feature acquisition unit 312 acquires the direction of movement in the moving image 5 for each of the plurality of moving bodies 10a to 10f.

6 (a), 7 (a) and 8 (a) show the velocity, acceleration and movement direction of the player A, which is one of the plurality of moving bodies 10a to 10f, obtained from the moving image 5. Shows the time change of. From FIG. 6A, it can be seen that the speed increases in the time section in which the player A is running as compared with the time section in which the player A is walking. Further, from FIG. 7A, it can be seen that the acceleration changes significantly in the time section in which the athlete A is running, and from FIG. 8A, it can be seen that the athlete A changes its direction in various ways.

As shown in FIGS. 6A and 8A, acquisition of the locus of the moving body 10 from the moving image 5 may fail depending on the time interval. This is because there is a time interval in which the position of the moving body 10 in the moving object 10 cannot be estimated due to a plurality of players overlapping in the moving image 5 or hiding in an obstacle. Further, if the locus of the moving body 10 is fragmented in this way, the association of the moving body 10 between the frames may be erroneous. For example, when two players are separated after overlapping, it may not be possible to correctly determine whether the two players have been replaced or not.

Returning to FIG. 5, the second feature acquisition unit 313 uses the movement information of the plurality of moving bodies 10a to 10f measured by the plurality of measuring devices 100a to 100f received by the receiving unit 311 to obtain the plurality of moving bodies. Acquire a second time-series feature for the movement of 10a-10f. The second time-series feature is an index showing how the positions, velocities, accelerations, their directions, and the like of the plurality of moving bodies 10a to 10f obtained from the movement information change with time. The second feature acquisition unit 313 is realized by the control unit 301 collaborating with the image processing unit 303. The second feature acquisition unit 313 functions as a second feature acquisition means.

The second feature acquisition unit 313 converts the movement information received from each of the plurality of measuring devices 100a to 100f into the movement information in the world coordinate system before acquiring the second time-series feature. More specifically, the second feature acquisition unit 313 determines the vertical direction (z direction in FIG. 2) from the direction of gravity detected by the acceleration sensor. Further, the second feature acquisition unit 313 determines the x-direction and the y-direction in the horizontal plane (xy plane in FIG. 2) from the directions detected by the geomagnetic sensor. The second feature acquisition unit 313 determines the world coordinate system based on such gravitational and geomagnetic information. Then, the second feature acquisition unit 313 calibrates the direction of each speed or acceleration of the plurality of moving bodies 10a to 10f that move in various directions to the direction in the world coordinate system.

The second feature acquisition unit 313 acquires the respective accelerations of the plurality of moving bodies 10a to 10f measured by the acceleration sensor from the movement information received by the receiving unit 311. Then, the second feature acquisition unit 313 acquires the following features (1) to (3) as the second time-series features.
(1) Dispersion of accelerations of a plurality of moving bodies 10a to 10f in movement information.
(2) Acceleration of a plurality of moving bodies 10a to 10f in the moving information in the vertical direction.
(3) Direction of movement of a plurality of moving bodies 10a to 10f in the movement information.

First, the second feature acquisition unit 313 calculates the variance of acceleration at a plurality of time points while the imaging device 200 is capturing the moving image 5 for each of the plurality of moving bodies 10a to 10f. The variance of acceleration is an index showing how much the magnitude of acceleration varies with time. The second feature acquisition unit 313 calculates the variance of the acceleration in a time interval having a width of 1 second centered on each of the plurality of time points, for example. The plurality of time points for calculating the variance of the acceleration are set at the same timing as, for example, the timing at which the first feature acquisition unit 312 acquires the velocity from the moving image 5.

Secondly, the second feature acquisition unit 313 calculates the vertical acceleration at a plurality of time points while the image pickup apparatus 200 is capturing the moving image 5 for each of the plurality of moving bodies 10a to 10f. Specifically, the second feature acquisition unit 313 extracts a component of the magnitude of acceleration in the vertical direction from the acceleration information included in the movement information. The vertical direction can be determined by detecting gravity with an acceleration sensor. The plurality of time points for calculating the variance of the acceleration are set at the same timing as, for example, the timing at which the first feature acquisition unit 312 acquires the acceleration from the moving image 5.

Thirdly, the second feature acquisition unit 313 integrates the accelerations of each of the plurality of moving bodies 10a to 10f at a plurality of time points while the imaging device 200 is capturing the moving image 5, thereby performing a plurality. Calculate the speed at the time of. Then, the second feature acquisition unit 313 acquires the speed directions at a plurality of time points as the movement directions of the plurality of moving bodies 10a to 10f. The plurality of time points for calculating the variance of the acceleration are set at the same timing as, for example, the timing at which the first feature acquisition unit 312 acquires the direction of movement from the moving image 5.

6 (b), 7 (b) and 8 (b) show the dispersion of acceleration obtained from the measuring device 100 of the athlete A, which is one of the plurality of moving bodies 10a to 10f, in the vertical direction. It shows the time change of the acceleration and the direction of movement of. Further, in FIGS. 6 (c), 7 (c) and 8 (c), the dispersion of the acceleration obtained from the measuring device 100 of the athlete B, which is one of the plurality of moving bodies 10a to 10f, is shown. The time change of acceleration in the vertical direction and the direction of movement is shown. As shown in these figures, the second time-series feature obtained from the measurement of the measuring device 100 is not fragmented like the first time-series feature obtained from the moving image 5. Further, since the measuring device 100 is mounted on the moving body 10, it is necessary to accurately specify which of the moving bodies 10a to 10f each of the plurality of measuring devices 100a to 100f is measuring. Can be done.

Returning to FIG. 5, the calculation unit 314 includes the first time-series feature acquired by the first feature acquisition unit 312 and the second time-series feature acquired by the second feature acquisition unit 313. Calculate an index showing the correlation between. The calculation unit 314 is realized by the control unit 301. The calculation unit 314 functions as a calculation means.

More specifically, the calculation unit 314 calculates the following three correlation values.
(1) A first correlation value showing a correlation between the speed of the moving body 10 in the moving body 5 and the variance of the acceleration measured by the measuring device 100.
(2) A second correlation value showing a correlation between the acceleration of the moving body 10 in the moving body 10 in a direction other than the vertical direction in the moving body 5 and the acceleration in the vertical direction measured by the measuring device 100.
(3) A third correlation value indicating the correlation between the moving direction of the moving body 10 in the moving body 5 and the moving direction measured by the measuring device 100.

First, the calculation unit 314 calculates a first correlation value (correlation coefficient) indicating a correlation between the speed of the moving body 10 in the moving object 5 and the variance of the acceleration measured by the measuring device 100. When the dispersion of acceleration is large, the moving speed tends to be large, and when the dispersion of acceleration is small, the moving speed tends to be small. The first correlation value is an index that reflects such a tendency.

Specifically, as shown in FIG. 6 (d), the correlation value between the speed of player A obtained from the moving image 5 and the variance of acceleration obtained from the measuring device 100 of player A becomes a relatively large value. Become. On the other hand, the correlation value between the speed of the player A obtained from the moving image 5 and the variance of the acceleration obtained from the measuring device 100 of the player B is close to 0. The first correlation value is an effective index when only the single moving body 10 is moving among the plurality of moving bodies 10a to 10f, for example, when only a single player is running.

Secondly, the calculation unit 314 has a second correlation value (phase relationship) showing a correlation between the acceleration of the moving body 10 in the moving body 10 in the moving object 10 in a direction other than the vertical direction and the acceleration in the vertical direction measured by the measuring device 100. Number) is calculated. When the moving body 10 walks or runs, it often involves vertical movement. Therefore, the acceleration of the moving body 10 in the vertical direction tends to correlate with the acceleration of the moving body 10 in a direction other than the vertical direction (for example, the horizontal direction or the diagonal direction between the horizontal direction and the vertical direction). The second correlation value is an index that reflects such a tendency.

Specifically, as shown in FIG. 7D, the correlation value between the acceleration of player A obtained from the moving image 5 and the acceleration in the vertical direction obtained from the measuring device 100 of player A is a relatively large value. become. On the other hand, the correlation value between the acceleration of the player A obtained from the moving image 5 and the acceleration in the vertical direction obtained from the measuring device 100 of the player B is close to 0. The second correlation value is an effective index when the speed rhythm of the moving moving body 10 is remarkable, for example, when a player is running.

Thirdly, the calculation unit 314 calculates a third correlation value indicating the correlation between the movement direction of the moving body 10 in the moving body 5 and the movement direction measured by the measuring device 100. Specifically, the calculation unit 314, as a third correlation value, is the difference (specifically, the angle) between the direction of movement of the moving body 10 in the moving body 5 and the direction of movement measured by the measuring device 100. The difference) is calculated by averaging over time.

Specifically, as shown in FIG. 8 (d), the average value of the difference between the movement direction of the player A obtained from the moving image 5 and the movement direction obtained from the measuring device 100 of the player A is relative. Becomes a small value. On the other hand, the average value of the difference between the movement direction of the player A obtained from the moving image 5 and the movement direction obtained from the measuring device 100 of the player B is a relatively large value. The third correlation value is an effective index when a plurality of moving bodies 10 are moving in different directions, such as when a plurality of athletes are running in different directions.

The calculation unit 314 sets the first time-series feature and the second time-series feature based on the first correlation value, the second correlation value, and the third correlation value calculated in this way. Calculate the index showing the correlation. Specifically, the calculation unit 314 calculates the weighted sum of the first correlation value, the second correlation value, and the third correlation value as such an index. The weights of the three correlation values can be freely determined. For example, the weights of the three correlation values may be constant values or values that change depending on the situation. The index calculated in this way is used as the final index for the specific processing in the specific unit 316.

The position estimation unit 315 estimates the positions of the plurality of moving bodies 10a to 10f in the vertical direction. Specifically, the position estimation unit 315 is based on the movement information received from each of the plurality of measuring devices 100a to 100f, and the position estimating unit 315 is in the vertical direction at a plurality of time points while the imaging device 200 is capturing the moving image 5. The acceleration of is calculated for each of the plurality of moving bodies 10a to 10f. Then, the position estimation unit 315 double-integrates the calculated acceleration in the vertical direction, and estimates the obtained value as the position in the vertical direction of each of the plurality of moving bodies 10a to 10f at a plurality of time points. .. The position estimation unit 315 is realized by the control unit 301. The position estimation unit 315 functions as a position estimation means.

The position of the moving body 10 in the vertical direction changes as the moving body 10 jumps or bends. Specifically, FIG. 9 shows the time change of the acceleration in the vertical direction when the moving body 10 jumps. As shown in FIG. 9, when the moving body 10 is running, the acceleration measured by the measuring device 100 mounted on the moving body 10 repeats increasing and decreasing at a constant cycle. On the other hand, when the moving body 10 jumps, the acceleration increases over a relatively long time, so that the position of the moving body 10 increases vertically upward. The position of the moving body 10 in the vertical direction is relatively limited because the range of change is limited, and the long-term average value is stable to the value when the moving body 10 is in the basic posture. It can be estimated with high accuracy.

The calculation unit 314 uses the above-mentioned index from the movement information in the time interval in which the movement information measured by the measurement device 100 of the plurality of measurement devices 100a to 100f satisfies a predetermined condition. The index is calculated from this movement information in a time interval in which the movement information does not satisfy a predetermined condition without calculation. When the predetermined conditions are satisfied, there is a probability that the correlation between the measuring device 100 and the moving body 10, which should originally correspond to each other, becomes poor, and the association between the measuring device 100 and the moving body 10 is erroneous. This is the case when it is expected to be high. Therefore, the calculation unit 314 excludes the time interval in which such an association is likely to be mistaken from the above-mentioned three correlation values and the time interval for calculating the final index determined from them.

Specifically, the predetermined condition is satisfied when the absolute value of the difference between the vertical position of the moving body 10 and the average position of the moving body 10 in the vertical direction exceeds the permissible value. .. When the absolute value of the difference between the vertical position of the moving body 10 and its average position exceeds the permissible value, for example, the position of the moving body 10 in the vertical direction becomes large due to the athlete jumping or bending. If it changes. When the position of the moving body 10 in the vertical direction changes in this way, from the moving image 5 obtained by capturing the moving body 10 from diagonally above, whether the moving body 10 has moved to the back side or the front side in the horizontal direction, or in the vertical direction. It is not possible to determine whether it has moved. Therefore, there is a high probability that the first time-series feature and the second time-series feature are erroneously associated with each other.

In order to avoid such a problem, in the position estimation unit 315, the absolute value of the difference between the position of any of the plurality of moving bodies 10a to 10f in the vertical direction and the average position thereof is an allowable value. If it exceeds, it is estimated that the moving body 10 has jumped or bent. Then, in the time interval in which the moving body 10 jumps or bends by the position estimation unit 315, the second time-series feature obtained from the moving information of the moving body 10 is the first time-series feature. Do not calculate the index showing the correlation of. In other words, the calculation unit 314 uses the movement information only for a time period in which the absolute value of the difference between the vertical position of the moving body 10 estimated from the movement information and its average position is within the permissible value. Using the obtained second time-series feature, an index showing the correlation with the first time-series feature is calculated.

The average position of the moving body 10 in the vertical direction is the average of the positions of the moving body 10 in the vertical direction estimated by the position estimation unit 315 in, for example, the entire time interval in which the moving image 5 is imaged. The permissible value is set in advance to a value having a size capable of detecting that the moving body 10 has jumped or bent, and is stored in the ROM, the storage unit 302, or the like.

Returning to FIG. 5, the specific unit 316 includes the plurality of moving objects 10a to 10f in the moving image 5 captured by the image pickup device 200 and the plurality of measurement devices 100a to 100f based on the index calculated by the calculation unit 314. Identify the correspondence of ,. To specify this correspondence is to specify which of the plurality of measuring devices 100a to 100f the measuring device 100 is attached to each of the plurality of moving bodies 10a to 10f captured in the moving image 5. .. In other words, in the specific unit 316, which of the plurality of measuring devices 100a to 100f is the measuring device 100 attached to each of the plurality of moving bodies 10a to 10f imaged in the moving image 5. To identify. The specific unit 316 is realized by the control unit 301. The specific unit 316 functions as a specific means.

Specifically, the specific unit 316 refers to the measuring device 100 having the largest final index calculated with one moving body 10 among the plurality of measuring devices 100a to 100f. It is specified that it is the measuring device 100 corresponding to. The specific unit 316 repeatedly executes such a process for each of the plurality of mobile bodies 10a to 10f detected as the mobile body region in the moving image 5 and for each of the fragmented loci. As a result, the specific unit 316 associates the plurality of moving bodies 10a to 10f imaged in the moving image 5 with the plurality of measuring devices 100a to 100f, respectively.

The time interval in which the movement information measured by any of the plurality of measuring devices 100a to 100f satisfies the predetermined condition is excluded from the time interval in which the calculation unit 314 calculates the index. ing. Therefore, in this time interval, the specifying unit 316 specifies the correspondence relationship between the plurality of moving bodies 10a to 10f in the moving image 5 and the plurality of measuring devices 100a to 100f by using the movement information measured by the measuring device 100. Not used for

The display control unit 317 displays the moving image 5 received from the image pickup apparatus 200 by the reception unit 311 on the display unit 305. At this time, the display control unit 317 refers to the correspondence relationship specified by the specific unit 316, and refers to the measuring device 100 corresponding to each of the plurality of moving bodies 10a to 10f among the plurality of measuring devices 100a to 100f. The identification information to be identified is displayed on the display unit 305. The display control unit 317 is realized by the control unit 301 collaborating with the image processing unit 303 and the display unit 305. The display control unit 317 functions as a display control means.

Specifically, as shown in FIG. 10, the display control unit 317 displays the numbers of the corresponding measuring devices 100 at the respective positions of the plurality of moving bodies 10a to 10f in the moving image 5. The display control unit 317 displays the identification information of the measuring device 100 by following the moving body 10 that moves to various positions with time. As a result, the user who watches the moving image 5 can easily determine which of the plurality of measuring devices 100a to 100f each of the plurality of moving bodies 10a to 10f shown in the moving body 5 is equipped with the measuring device 100. Can be recognized. In particular, even when a plurality of moving bodies 10a to 10f are densely packed or intersected, or when the moving body 10 is hidden by an obstacle, the user makes a mistake in the correspondence relationship between the moving body 10 and the measuring device 100. It is possible to accurately identify each of the plurality of moving bodies 10a to 10f.

The flow of the moving body identification process executed in the moving body identification device 300 as described above will be described with reference to the flowcharts shown in FIGS. 11 and 12.

The moving body identification process shown in the flowchart of FIG. 11 starts when the moving body identification device 300 receives an instruction to execute the moving body identification process from the user via the operation unit 304. At this time, the movement information of the plurality of moving bodies 10a to 10f to be identified and the moving image 5 in which the plurality of moving bodies 10a to 10f are imaged are received from the plurality of measuring devices 100 and the imaging device 200, respectively. Already received by 311.

When the moving body identification process is started, the control unit 301 of the moving body identification device 300 detects the positions of the plurality of moving bodies 10a to 10f in the image of each frame included in the moving image 5 (step S1). Specifically, the control unit 301 extracts the foreground object region by subtracting the image of each frame included in the moving image 5 from the background image by the background subtraction method. Then, the control unit 301 detects the moving body region by executing the expansion / contraction processing and the labeling processing on the extracted foreground object region. The control unit 301 estimates the position of the moving body region detected in this way as the position of the plurality of moving bodies 10a to 10f in the moving image 5.

Upon detecting the positions of the plurality of moving bodies 10a to 10f in the moving image 5, the control unit 301 associates the moving bodies 10 between the frames (step S2). Specifically, the control unit 301 shows changes in the positions of the plurality of moving bodies 10a to 10f from the beginning to the end of the moving image 5 by connecting the moving body regions having a short distance between adjacent frames. Get the trajectory.

When the loci of the plurality of moving bodies 10a to 10f are acquired, the control unit 301 aligns the coordinate system between the moving image 5 and the measuring device 100 (step S3). Specifically, the control unit 301 converts the position information in the moving image 5 obtained by imaging the plurality of moving bodies 10a to 10f from diagonally above onto the horizontal plane to convert the position information into the position information in the world coordinate system. To do. Further, the control unit 301 determines the direction of the acceleration included in the movement information received from each of the plurality of measuring devices 100a to 100f based on the information of gravity and geomagnetism, and converts the direction into the direction in the world coordinate system. To do. As a result, the control unit 301 aligns the coordinate system between the moving image 5 and the measuring device 100.

When the coordinate system is matched between the moving image 5 and the measuring device 100, the control unit 301 shifts to the process of specifying the correspondence between the moving body 10 and the measuring device 100. First, the control unit 301 designates the moving body 10 to be identified and the identification time interval (step S4). More specifically, the control unit 301 designates any one of the plurality of moving bodies 10a to 10f in the moving image 5 as the moving body 10 to be identified. Further, the identification time interval is a time interval from the first frame to the last frame included in the moving image, and is a time interval for specifying the corresponding measuring device 100 for the designated moving body 10. As the identification time interval, the control unit 301 specifies, for example, a continuous time interval in which the locus of the designated moving body 10 in the moving body 10 is not fragmented (that is, a time interval in which a continuous locus is obtained).

When the moving body 10 to be identified and the identification time interval are specified, the control unit 301 identifies the measuring device 100 corresponding to the designated moving body 10 in the designated identification time interval (step S5). The details of the specific process in step S5 will be described with reference to the flowchart shown in FIG.

When the specific process shown in FIG. 12 starts, the control unit 301 designates the measuring device 100 for calculating the index from the plurality of measuring devices 100a to 100f (step S51). Then, the control unit 301 excludes the time interval in which the movement information measured by the designated measuring device 100 satisfies the predetermined condition from the time interval for calculating the index (step S52).

Specifically, the control unit 301 functions as a position estimation unit 315, and double-integrates the acceleration in the vertical direction included in the movement information to obtain the moving body 10 equipped with the designated measuring device 100. Estimate the position in the vertical direction. Then, the control unit 301 determines whether or not the designated identification time interval includes a time interval in which the absolute value of the difference between the estimated position and the average position exceeds the permissible value. As a result of the determination, when such a time interval is included in the specified identification time interval, the control unit 301 calculates an index for the time interval excluding the time interval from the specified identification time interval. Set as the time interval to be performed. On the other hand, when such a time interval is not included in the designated identification time interval, the control unit 301 sets the entire designated identification time interval as the time interval for calculating the index.

When the time interval for calculating the index is set in this way, the control unit 301 sets the first correlation value, the second correlation value, which indicates the correlation between the designated moving body 10 and the measuring device 100 in the designated identification time interval. And the third correlation value are calculated (steps S53 to S55).

Specifically, the control unit 301 functions as the first feature acquisition unit 312, and from the moving images of the plurality of moving bodies 10a to 10f imaged by the imaging device 200, the designated identification time of the designated moving body 10 Acquire the first time series feature in the interval. In addition, the control unit 301 functions as the second feature acquisition unit 313, and acquires the second time-series feature in the designated identification time interval from the movement information measured by the designated measuring device 100. Then, the control unit 301 functions as a calculation unit 314, and as described above, (1) a first unit showing the correlation between the speed of the moving body 10 in the moving object 5 and the variance of the acceleration measured by the measuring device 100. (2) A second correlation value showing a correlation between the acceleration of the moving body 10 in a direction other than the vertical direction in the moving body 10 and the acceleration in the vertical direction measured by the measuring device 100, and (3). ) A third correlation value indicating a correlation between the moving direction of the moving body 10 in the moving body 5 and the moving direction measured by the measuring device 100 is calculated.

When these three correlation values are calculated, the control unit 301 functions as the specific unit 316 and calculates the final index showing the correlation between the designated moving body 10 and the measuring device 100 in the specified identification time interval. (Step S56). Specifically, the control unit 301 calculates the final index by taking the weighted sum of the three correlation values.

When the final index is calculated, the control unit 301 determines whether or not all the measuring devices 100 have been designated (step S57). If all the measuring devices 100 have not been designated (step S57; NO), the control unit 301 returns the process to step S51. Then, the control unit 301 newly designates an undesignated measuring device 100 from the plurality of measuring devices 100a to 100f, and executes the processes of steps S51 to S57 for the newly designated measuring device 100. In this way, the control unit 301 repeats the processes of steps S51 to S57 until all the measuring devices 100 have been designated, and correlates all the measuring devices 100 with the designated moving body 10 in the designated identification time interval. Calculate the indicated index.

Finally, when all the measuring devices 100 have been designated (step S57; YES), the control unit 301 assigns the measuring device 100 having the largest index among the measuring devices 100 for which the index has been calculated to the designated identification time interval. It is specified as the measuring device 100 corresponding to the moving body 10 specified in (step S58). When the process of step S58 is completed, the control unit 301 ends the specific process shown in FIG.

When the measuring device 100 corresponding to the designated moving body 10 is specified in the designated identification time interval, the control unit 301 determines whether or not all the identification time sections of all the moving bodies 10 have been specified (step). S6). When all the identification time intervals of all the moving bodies 10 have not been specified (step S6; NO), the control unit 301 returns the process to step S4. Then, the control unit 301 newly designates an undesignated combination of the combination of the plurality of moving bodies 10a to 10f captured in the moving image and the identification time interval, and identifies the moving body 10 in the newly designated combination. The processing of steps S4 to S6 is executed for the time interval. In this way, the control unit 301 repeats the processes of steps S4 to S6 until all the identification time intervals of all the moving bodies 10 are specified, and the corresponding measurements are made for all the identification time sections of all the moving bodies 10. Identify the device 100.

Finally, when all the identification time intervals of all the moving bodies 10 are specified (step S6; YES), the control unit 301 functions as the display control unit 317 and displays the specific result (step S7). Specifically, as shown in FIG. 10, for example, the control unit 301 provides the display unit 305 with the identification information for identifying the measuring device 100 corresponding to each of the plurality of moving bodies 10a to 10f together with the moving image 5. indicate. As a result, the mobile body identification process shown in FIG. 11 is completed.

As described above, the mobile body identification system 1 and the mobile body identification device 300 according to the first embodiment have the first time-series features from the moving images 5 of the plurality of moving bodies 10a to 10f imaged by the image pickup device 200. The second time-series feature is acquired from the movement information measured by the plurality of measuring devices 100a to 100f mounted on the plurality of moving bodies 10a to 10f, respectively. Then, the mobile body identification system 1 and the mobile body identification device 300 calculate an index showing the correlation between the first time-series feature and the second time-series feature, and based on the calculated index, in the moving image 5. The correspondence between the plurality of moving bodies 10a to 10f and the plurality of measuring devices 100a to 100f is specified.

At that time, in the moving body identification system 1 and the moving body identification device 300, the time interval in which the movement information measured by the measuring device 100 of the plurality of measuring devices 100a to 100f satisfies a predetermined condition. In, the above-mentioned index is not calculated from this movement information. In other words, the mobile body identification system 1 and the mobile body identification device 300 exclude the time interval in which the probability of misalignment between the measuring device 100 and the mobile body 10 is high from the time interval for calculating the index. .. As a result, it is possible to reduce the error of the association, and it is possible to accurately identify the plurality of moving bodies 10a to 10f imaged in the moving image 5.

Further, the moving body identification system 1 and the moving body identification device 300 according to the first embodiment use (1) the velocity of the moving body 10 in the moving image 5 and the dispersion of the acceleration measured by the measuring device 100 as the above-mentioned indexes. The first correlation value showing the correlation of (2) the acceleration in the direction other than the vertical direction of the moving body 10 in the moving body 10 and the second correlation showing the correlation in the vertical direction measured by the measuring device 100. A third correlation value indicating a correlation between the correlation value and (3) the direction of movement of the moving body 10 in the moving body 5 and the direction of movement measured by the measuring device 100 is calculated, and these three correlations are obtained. Calculate using the weighted sum of values as an index. As described above, the moving body identification system 1 and the moving body identification device 300 calculate the correlation value by combining the first time-series feature and the feature that tends to have a high correlation as the second time-series feature. As a result, the plurality of moving bodies 10a to 10f in the moving image 5 and the plurality of measuring devices 100a to 100f can be accurately associated with each other.

In particular, when a plurality of mobile bodies 10a to 10f are moving indoors, for example, when playing an indoor competition, it is difficult for radio waves from the outside to reach the mobile body 10 with GPS (Global Positioning System) or the like. It is difficult to accurately detect the position of. Even in such a case, the mobile body identification system 1 and the mobile body identification device 300 according to the first embodiment can accurately identify a plurality of mobile bodies 10a to 10f imaged in the moving image 5.

(Modification example)
Although the embodiment of the present invention has been described above, the embodiment is an example, and the scope of application of the present invention is not limited to this. That is, the embodiments of the present invention can be applied in various ways, and all the embodiments are included in the scope of the present invention.

For example, in the above-described embodiment, an example in which the plurality of moving bodies 10a to 10f in the moving body 5 and the plurality of measuring devices 100a to 100f mounted on the plurality of moving bodies 10a to 10f are associated with each other is shown, but at least in the moving body 5 One mobile body 10 may be associated with at least one measuring device 100 mounted on the at least one mobile body 10. In this case, the correspondence between the one moving body 10 and the only one measuring device 100 mounted on the only moving body 10 in the moving body 10 is obvious and is excluded. That is, the present invention is not limited to associating the plurality of moving bodies 10a to 10f in the moving body 5 with the plurality of measuring devices 100a to 100f mounted on the plurality of moving bodies 10a to 10f, respectively, and for example, in the moving body 5 One moving body 10 is associated with a plurality of measuring devices 100a to 10f mounted on a plurality of moving bodies 10a to 10f, and among the plurality of moving bodies 10a to 10f and the plurality of moving bodies 10a to 10f in the moving image 5. It also includes associating with one measuring device 100 mounted on one moving body 10.

Further, the specific unit 316 is measured by the measuring device 100 in a time interval in which the movement information measured by the measuring device 100 among the plurality of measuring devices 100a to 100f satisfies a predetermined condition. The movement information was not used to identify the above-mentioned correspondence. However, the specific unit 316 has described above the information regarding the movement of at least one moving body 10 obtained from the moving image 5 captured by the imaging device 200 in the time interval in which the movement information satisfies the predetermined condition. It does not have to be used to specify the correspondence. The information regarding the movement of at least one mobile body 10 is, for example, a first time-series feature. In other words, the specific unit 316 provides information on the movement obtained from the moving image 5 for all of at least one moving body 10 in the moving body 5 in a time interval in which the moving information satisfies a predetermined condition. 316 may not be used to identify the correspondence.

Further, for example, in the above embodiment, the calculation unit 314 calculates the first correlation value, the second correlation value, and the third correlation value, and the weighted sum of these three correlation values is calculated as the first correlation value. It was calculated as an index showing the correlation between the time-series features and the second time-series features. However, in the present invention, the calculation unit 314 does not calculate all of these three correlation values, and only one of these three correlation values is used as the first time series feature and the second time series. It may be calculated as an index showing the correlation with the target characteristics. Alternatively, the calculation unit 314 may use any two of these three correlation values to calculate an index showing the correlation between the first time-series feature and the second time-series feature. .. Further, the calculation unit 314 may use a calculation method other than the weighted sum.

In the above embodiment, the first feature acquisition unit 312 detects the positions of a plurality of moving bodies 10a to 10f in the moving image 5 using the background subtraction method, and tracks the positions. However, in the present invention, the first feature acquisition unit 312 is not limited to the background subtraction method, and other methods may be used. For example, the first feature acquisition unit 312 can detect and track the feature points in the moving image 5 by using the technique of feature point tracking (optical flow).

In the above embodiment, the specific unit 316 is attached to the moving body 10 when the absolute value of the difference between the vertical position of the moving body 10 and the average position of the moving body 10 in the vertical direction exceeds an allowable value. The measuring device 100 was excluded from the association. However, in the present invention, the predetermined conditions for excluding from the association may be satisfied in cases other than such cases. For example, the predetermined conditions may be satisfied when it is determined that the position of the moving body 10 in the horizontal direction is out of the field of view of the image pickup apparatus 200. In this case, the position of the moving body 10 in the horizontal direction can be estimated from the moving information measured by the measuring device 100 mounted on the moving body 10. As described above, the specific unit 316 satisfies the plurality of moving bodies 10a to 10f in the moving body 5 when various conditions that are assumed to have a high probability of erroneously associating the measuring device 100 with the moving body 10 are satisfied. It can be determined that the moving body 10 corresponding to the measuring device 100 is not specified.

In the above embodiment, the display control unit 317 displays the number of the corresponding measuring device 100 at each position of the plurality of moving bodies 10a to 10f in the moving image 5 as the identification information for identifying the measuring device 100. However, the display control unit 317 may display characters, symbols, or figures instead of numbers as identification information. Further, the display control unit 317 may display the identification information in the vicinity of the moving body 10 or may superimpose the identification information on the moving body 10 for display.

Alternatively, the display control unit 317 may display the loci of the plurality of moving bodies 10a to 10f obtained from the moving image 5 by a method such as background subtraction method or feature point tracking on the display unit 305 as identification information. In this case, the display control unit 317 displays each of the trajectories of the plurality of moving bodies 10a to 10f in different display modes so that the corresponding measuring device 100 can be identified. For example, the display control unit 317 changes the color of the locus according to the corresponding measuring device 100, changes the thickness of the locus, changes the drawing method of the locus depending on the solid line and the dotted line, and the like, thereby changing the corresponding measuring device 100. Display the trajectory so that you can identify.

In the above embodiment, the moving body identification device 300 includes a first feature acquisition unit 312 and a second feature acquisition unit 313. However, in the present invention, the moving object identification device 300 does not have a function of acquiring the first time-series feature and the second time-series feature, and the image pickup device 200 obtains the first feature acquisition unit 312. Each of the plurality of measuring devices 100 may include a second feature acquisition unit 313. In this case, the image pickup apparatus 200 analyzes the moving image 5 obtained by imaging the plurality of moving bodies 10 to acquire the first time-series feature, and obtains the information indicating the acquired first time-series feature. It is transmitted to the mobile body identification device 300. Further, each of the plurality of measuring devices 100 analyzes the measured movement information to acquire the second time-series feature, and transfers the acquired information indicating the second time-series feature to the moving body identification device 300. Send. Upon receiving the information indicating the first time-series feature and the information indicating the second time-series feature, the mobile body identification device 300 receives the information indicating the first time-series feature and the second time-series feature. An index showing the correlation is calculated to specify the correspondence between the plurality of moving bodies 10 and the plurality of measuring devices 100.

In the above embodiment, a human being has been described as an example of the moving body 10. However, in the present invention, the moving body 10 may be an animal other than a human being, or an artificial object such as an automobile or a robot. Further, in the above-described embodiment, a sport played indoors has been described as an example of a scene in which the mobile identification system 1 is applied. However, in the present invention, the mobile identification system 1 can also be applied to sports played outdoors. Further, the present invention is not limited to sports, and can be applied to any scene as long as a plurality of moving objects move. For example, the mobile identification system 1 can be applied to identification of infants in a nursery school or kindergarten, identification of students in a school, identification of visitors in a movie theater or a stadium, and the like.

Further, in the above-described embodiment, the plurality of measuring devices 100, the imaging device 200, and the moving body identification device 300 each include a CPU. However, in the present invention, the plurality of measuring devices 100, the imaging device 200, and the moving object identification device 300 may include a dedicated control circuit such as an ASIC (Application Specific Integrated Circuit) instead of the CPU. For example, in the above embodiment, the CPU is used as each of the receiving unit 311, the first feature acquisition unit 312, the second feature acquisition unit 313, the calculation unit 314, the position estimation unit 315, the specific unit 316, and the display control unit 317. Although functional, each of these parts may be implemented by a separate control circuit.

It should be noted that the present invention can be provided as a moving body identification device, a measuring device, and an imaging device having a configuration for realizing the function according to the present invention, and by applying a program, an existing information processing device or the like can be provided in the present invention. It can also function as each of the moving body identification device, the measuring device, and the imaging device. That is, a CPU or the like that controls an existing information processing device or the like can execute a program for realizing each functional configuration by the plurality of measuring devices 100, the image pickup device 200, and the moving body identification device 300 illustrated in the above embodiment. By applying to, it can function as a moving body identification device, a measuring device and an imaging device according to the present invention. Further, the display method according to the present invention can be carried out by using a moving body identification device, a measuring device and an imaging device.

Moreover, the method of applying such a program is arbitrary. The program can be stored and applied in a computer-readable storage medium such as a flexible disk, a CD (Compact Disc) -ROM, a DVD (Digital Versatile Disc) -ROM, or a memory card. Further, the program can be superimposed on the carrier wave and applied via a communication medium such as the Internet. For example, the program may be posted and distributed on a bulletin board system (BBS: Bulletin Board System) on a communication network. Then, by starting this program and executing it in the same manner as other application programs under the control of the OS (Operating System), the above processing may be executed.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the specific embodiment, and the present invention includes the invention described in the claims and the equivalent range thereof. included. The inventions described in the claims of the original application of the present application are described below.

(Appendix 1)
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on the second time-series feature relating to the movement of at least one moving body, the correspondence between the at least one moving body and the at least one measuring device in the moving image is determined by only one in the moving body. A specific means for specifying the moving body except for the correspondence between the moving body and only one measuring device attached to the one moving body.
With
The specific means is measured by any of the measuring devices in a time period in which the movement information measured by the measuring device of at least one of the measuring devices satisfies a predetermined condition. Do not use movement information to identify correspondence,
A moving body identification device characterized in that.

(Appendix 2)
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the moving information measured by the at least one measuring device.
With more
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
The moving body identification device according to Appendix 1, wherein the moving body identification device is described.

(Appendix 3)
A receiving means for receiving the movement information from each of the at least one measuring device and receiving the moving image from the imaging device, and
A first feature acquisition means for acquiring the first time-series feature from the moving image received by the receiving means, and a first feature acquisition means.
A second feature acquisition means for acquiring the second time-series feature from the movement information received by the receiving means, and
Further prepare
The moving body identification device according to Appendix 1 or 2, characterized in that.

(Appendix 4)
The first feature acquisition means acquires the velocity of the at least one moving body in the moving image as the first time-series feature.
As the second time-series feature, the second feature acquisition means acquires the variance of the acceleration of at least one moving body from the movement information.
The moving body identification device according to Appendix 3, wherein the moving body identification device is characterized.

(Appendix 5)
As the first time-series feature, the first feature acquisition means acquires acceleration in a direction other than the vertical direction of the at least one moving body in the moving image.
As the second time-series feature, the second feature acquisition means acquires the acceleration of at least one moving body in the vertical direction from the movement information.
The moving body identification device according to Appendix 3 or 4, characterized in that.

(Appendix 6)
The first feature acquisition means acquires the direction of movement of the at least one moving body in the moving image as the first time-series feature.
As the second time-series feature, the second feature acquisition means acquires the movement direction of the at least one moving body from the movement information.
The moving body identification device according to any one of Supplementary note 3 to 5, wherein the moving body identification device is described.

(Appendix 7)
Further provided with a calculation means for calculating an index showing the correlation between the first time-series feature and the second time-series feature.
The first feature acquisition means, as the first time-series feature, includes the velocity of the at least one moving body in the moving image and the acceleration of the at least one moving body in the moving object in a direction other than the vertical direction. , And the direction of movement of the at least one moving body in the moving image.
The second feature acquisition means, as the second time-series feature, disperses the acceleration of the at least one moving body and the acceleration of the at least one moving body in the vertical direction from the movement information. The direction of movement of at least one moving body and the direction of movement are obtained.
The calculation means has a first correlation value indicating a correlation between the speed acquired by the first feature acquisition means and the variance acquired by the second feature acquisition means, and the first feature acquisition. A second correlation value indicating the correlation between the acceleration acquired by the means and the acceleration acquired by the second feature acquisition means, the direction of movement acquired by the first feature acquisition means, and the movement. The index is calculated based on the third correlation value indicating the correlation with the movement direction acquired by the second feature acquisition means.
The moving body identification device according to any one of Supplementary note 3 to 6, wherein the moving body identification device is characterized.

(Appendix 8)
The calculation means calculates, as the index, the weighted sum of the first correlation value, the second correlation value, and the third correlation value.
The moving body identification device according to Appendix 7, wherein the moving body identification device is described.

(Appendix 9)
A display control means for displaying identification information identifying a measuring device corresponding to each of the at least one moving body of the at least one measuring device on the display means together with the moving image.
Further prepare
The moving body identification device according to any one of Supplementary Provisions 1 to 8, wherein the moving body identification device is characterized.

(Appendix 10)
The imaging device generates the moving image by imaging the at least one moving body that is moving indoors.
The moving body identification device according to any one of Supplementary Provisions 1 to 9, wherein the moving body identification device is characterized.

(Appendix 11)
The imaging device generates the moving image by imaging the at least one moving body from diagonally above.
The moving body identification device according to any one of Supplementary Provisions 1 to 10, wherein the moving body identification device is described.

(Appendix 12)
The at least one measuring device includes an acceleration sensor.
The moving body identification device according to any one of Supplementary note 1 to 11, wherein the moving body identification device is characterized.

(Appendix 13)
If the absolute value of the difference exceeds the permissible value, the position estimating means presumes that any of the moving objects has jumped or bent.
The moving body identification device according to Appendix 2, wherein the moving body identification device is described.

(Appendix 14)
The specific means does not use the information regarding the movement of at least one moving body obtained from the moving image captured by the imaging device in the time interval to identify the correspondence.
The moving body identification device according to any one of Supplementary note 1 to 13, wherein the moving body identification device is described.

(Appendix 15)
The moving body identification device according to any one of Supplementary note 1 to 14, the imaging device, and at least one measuring device.
A mobile identification system characterized by this.

(Appendix 16)
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on the second time-series feature relating to the movement of at least one moving body, the correspondence between the at least one moving body and the at least one measuring device in the moving image is determined by only one in the moving body. A specific step to specify except for the correspondence between the moving body and only one measuring device attached to the only one moving body.
Including
In the specific step, the movement information measured by any of the at least one measuring device is measured by the measuring device in the time interval in which the predetermined conditions are satisfied. Do not use movement information to identify correspondence,
A method for identifying a moving object.

(Appendix 17)
The computer of the mobile identification device,
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on the second time-series feature relating to the movement of at least one moving body, the correspondence between the at least one moving body and the at least one measuring device in the moving image is determined by only one in the moving body. A specific means for specifying the moving body except for the correspondence between the moving body and only one measuring device attached to the one moving body.
To function as
The specific means is measured by any of the measuring devices in a time period in which the movement information measured by the measuring device of at least one of the measuring devices satisfies a predetermined condition. Do not use movement information to identify correspondence,
A program characterized by that.

1 ... Mobile identification system, 5 ... Video, 10,10a, 10b, 10c, 10d, 10e, 10f ... Mobile, 100, 100a, 100b, 100c, 100d, 100e, 100f ... Measuring device, 101 ... Control unit, 102 ... storage unit, 103 ... measurement unit, 104 ... communication unit, 200 ... imaging device, 201 ... control unit, 202 ... storage unit, 203 ... imaging unit, 204 ... image processing unit, 205 ... communication unit, 300 ... mobile body Identification device, 301 ... control unit, 302 ... storage unit, 303 ... image processing unit, 304 ... operation unit, 305 ... display unit, 306 ... communication unit, 311 ... reception unit, 312 ... first feature acquisition unit, 313 ... Second feature acquisition unit, 314 ... calculation unit, 315 ... position estimation unit, 316 ... specific unit, 317 ... display control unit

Claims (12)

A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on a second time-series feature relating to the movement of at least one mobile body, a specific means for identifying the correspondence between the at least one mobile body and the at least one measuring device in the moving image.
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the movement information measured by the at least one measuring device, and
With
The specific means was measured by any of the measuring devices in a time interval in which the movement information measured by the measuring device of at least one of the measuring devices satisfied a predetermined condition. The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
A moving body identification device characterized in that. A receiving means for receiving the movement information from each of the at least one measuring device and receiving the moving image from the imaging device, and
A first feature acquisition means for acquiring the first time-series feature from the moving image received by the receiving means, and a first feature acquisition means.
A second feature acquisition means for acquiring the second time-series feature from the movement information received by the receiving means, and
Further prepare
The moving body identification device according to claim 1. A display control means for displaying identification information identifying a measuring device corresponding to each of the at least one moving body of the at least one measuring device on the display means together with the moving image.
Further prepare
The moving body identification device according to claim 1 or 2 . The imaging device generates the moving image by imaging the at least one moving body that is moving indoors.
The moving body identification device according to any one of claims 1 to 3, wherein the moving body identification device is characterized. The imaging device generates the moving image by imaging the at least one moving body from diagonally above.
The moving body identification device according to any one of claims 1 to 4, wherein the moving body identification device is characterized. The at least one measuring device includes an acceleration sensor.
The moving body identification device according to any one of claims 1 to 5, wherein the moving body identification device is characterized. A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on a second time-series feature relating to the movement of at least one mobile body, a specific means for identifying the correspondence between the at least one mobile body and the at least one measuring device in the moving image.
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the movement information measured by the at least one measuring device, and
With
The specific means was measured by any of the measuring devices in a time interval in which the movement information measured by the measuring device of at least one of the measuring devices satisfied a predetermined condition. The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Satisfies when is exceeded
If the absolute value of the difference exceeds the permissible value, the position estimating means presumes that any of the moving objects has jumped or bent.
A moving body identification device characterized in that . The specific means does not use the information regarding the movement of at least one moving body obtained from the moving image captured by the imaging device in the time interval to identify the correspondence.
The moving body identification device according to any one of claims 1 to 7, wherein the moving body identification device is characterized. A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. A specific means for identifying a correspondence between the at least one mobile body and the at least one measuring device in the moving image based on a second time-series feature relating to the movement of the at least one mobile body.
With
The specific means is measured by any of the measuring devices in a time period in which the movement information measured by the measuring device of at least one of the measuring devices satisfies a predetermined condition. The movement information is not used to identify the correspondence,
The correspondence between the at least one moving body and the at least one measuring device in the moving image is as follows: only one moving body in the moving body and only one measuring device attached to the one moving body. Specify except for the correspondence of
A moving body identification device characterized in that. The moving body identification device according to any one of claims 1 to 9, the imaging device, and at least one measuring device.
A mobile identification system characterized by this. A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by the imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. Based on a second time-series feature relating to the movement of at least one mobile body, a specific step of identifying the correspondence between the at least one mobile body and the at least one measuring device in the moving image.
A position estimation step for estimating the position of the at least one moving object in the vertical direction from the movement information measured by the at least one measuring device, and
Including
In the specific step, the movement information measured by the measuring device of at least one of the measuring devices was measured by the measuring device in the time interval in which the predetermined condition was satisfied. The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
A method for identifying a moving object. The computer of the mobile identification device,
A first time-series feature relating to the movement of at least one mobile body, obtained from a moving image captured by an imaging device, and the said, obtained from at least one measuring device mounted on the at least one mobile body. A specific means for identifying a correspondence between the at least one mobile body and the at least one measuring device in the moving image based on a second time-series feature relating to the movement of the at least one mobile body.
A position estimating means for estimating the position of the at least one moving body in the vertical direction from the movement information measured by the at least one measuring device.
To function as
The specific means was measured by any of the measuring devices in a time period in which the movement information measured by the measuring device of at least one of the measuring devices satisfied a predetermined condition. The movement information is not used to identify the correspondence,
The predetermined condition is an absolute value of the difference between the vertical position of any moving body of at least one moving body and the average position of any moving body in the vertical direction. Will be satisfied if is exceeded the permissible value,
A program characterized by that.

Images