JP7708296B2 - Movement trajectory information processing device - Google Patents

Description

The present invention relates to a movement trajectory information processing device, a movement trajectory information processing method, and a recording medium.

An example of a conventional device that calculates movement trajectory information is described in Patent Document 1. The conventional device uses a camera with fixed pan, tilt, and zoom (PTZ) values to continuously capture images of a moving person or other object. Furthermore, the conventional device extracts the position of the object from the image captured by the camera during each capture, and converts the coordinate values of the object in the image into coordinate values in real space using camera calibration information (internal and external parameters). The conventional device regards the time series of the object's real-space position thus obtained as movement trajectory information.

WO2014/010174

In a camera with a fixed PTZ value, the field of view of the camera is limited. Therefore, in order to monitor a wider range, it is important to change the PTZ value and calculate the movement trajectory information of the object from multiple images taken continuously by the camera while capturing the moving object within the imaging range. However, when the PTZ value of the camera changes, the camera calibration information of the camera changes. In addition, in order to obtain the camera calibration information corresponding to the changed PTZ value, the operator needs to perform calibration work. Therefore, in a conventional device that directly converts the coordinate values on the image of the object to coordinate values in real space using the camera calibration information from each of the multiple images taken continuously, it was difficult to calculate the movement trajectory information from multiple images taken continuously of the object with a camera that captures the moving object within the imaging range.

The present invention aims to provide a movement trajectory information processing device that solves the above-mentioned problems.

A movement trajectory information processing device according to one aspect of the present invention includes:
a calculation means for calculating movement trajectory information, which is a time series of position information of an object on an image, from a plurality of images continuously taken of the object by a camera capturing the object within a shooting range;
a conversion means for calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
The device is configured to include:

A movement trajectory information processing method according to another aspect of the present invention includes:
Calculating movement trajectory information, which is a time series of position information on an image of a moving object, from a plurality of images continuously captured by a camera capturing the moving object within a shooting range of the object;
calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
It is structured as follows.

A computer-readable recording medium according to another aspect of the present invention includes:
On the computer,
A process of calculating movement trajectory information, which is a time series of position information on an image of a moving object, from a plurality of images continuously captured by a camera capturing the object within a shooting range of the object;
a process of calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
The recording medium is configured to record a program for causing the recording medium to perform the above steps.

By having the above-mentioned configuration, the present invention can calculate movement trajectory information, which is a time series of the position of an object in real space, from multiple images captured continuously by a camera that captures a moving object within its shooting range.

1 is a schematic diagram illustrating a configuration example of a tracking system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a control device in a tracking system according to an embodiment of the present invention. 1 is a diagram showing an example of tracked object movement trajectory information used by a control device in a tracking system according to an embodiment of the present invention; 1 is a diagram showing an example of matched movement trajectory information used by a control device in a tracking system according to an embodiment of the present invention; FIG. 10 is a schematic diagram for explaining an example of a method for converting position information on an image from an image coordinate system to a world coordinate system in a control device in a tracking system according to an embodiment of the present invention. FIG. 5 is a flowchart showing an example of a tracking process performed by a control device using one of the cameras in the tracking system according to the embodiment of the present invention. 10 is a flowchart showing an example of a tracking process performed by a control device using the other camera in the tracking system according to one embodiment of the present invention. 1 is a block diagram of an embodiment of a movement trajectory information processing device according to the present invention;

Next, the embodiment of the present invention will be described in detail with reference to the drawings. In the description of the specification, the terms "tracking" and "tracking" are used together, but the terms are used for convenience only to explain the configuration of the part that is mainly related to a person, and the configuration of the part that is mainly related to a device.
[First embodiment]
First, in order to facilitate understanding of the first embodiment of the present invention, a problem assumed by the first embodiment of the present invention will be described.

Detecting and tracking people from surveillance camera images is useful for crime prevention, marketing, and the like. The field of view of a camera with a fixed PTZ value is limited. The field of view of a camera can be expanded by changing the PTZ value of the camera. However, the field of view of a single camera is still limited. Therefore, a system is considered in which multiple PTZ cameras are installed in a surveillance area with a common part of the camera field of view, and people are tracked over a wider area in space and time. In such a tracking system, it is preferable that the system automatically recognizes that a person being tracked by one camera has entered the field of view of another camera, and that the other camera can start tracking the person. In order to determine the person to take over tracking from among the many people present in the field of view of the camera, the movement trajectory of the person can be used. That is, the tracking target is determined by comparing the information on the movement trajectory of the person being tracked by one camera (tracking target movement trajectory information) with the information on the movement trajectory of the person in the field of view of the other camera (matched movement trajectory information). In order to perform such matching, the tracked object trajectory information and the matched trajectory information need to be expressed in the same coordinate system.

Generally, coordinate systems that represent the position of a person captured by a camera include an image coordinate system and a camera coordinate system that are unique to each camera, and a world coordinate system that is common to multiple cameras. The image coordinate system is a two-dimensional coordinate system on the image sensor, and the positions of points on an image are usually represented in this image coordinate system. The camera coordinate system is a coordinate system determined by the image coordinate system and the internal parameters of the camera. Therefore, the image coordinate system of a camera can be converted into the camera coordinate system of that camera using the internal parameters of that camera. The world coordinate system is a coordinate system determined by the image coordinate system and camera calibration information, which is composed of the internal and external parameters of the camera. Therefore, the image coordinate system of a camera can be converted into the world coordinate system of that camera using the camera calibration information of that camera.

Conventional devices extract the position of a person in an image coordinate system from a captured image, and then use camera calibration information to convert the coordinate values of the extracted image coordinate system of the person into coordinate values in a world coordinate system. However, when the PTZ value of a camera is changed to track a person, the camera calibration information changes. In addition, it is difficult to obtain camera calibration information corresponding to the changed PTZ value in real time. Therefore, with conventional devices, it is difficult to calculate movement trajectory information while tracking an object using a camera with a tracking function that captures a moving object within the shooting range. As a result, it is difficult to compare the movement trajectories of a person between multiple PTZ cameras in real time.

The present embodiment solves the above problem. Below, a tracking system 10 according to one embodiment of the present invention is described in detail with reference to the drawings.

Figure 1 is a schematic diagram showing an example of the configuration of a tracking system 10 according to one embodiment of the present invention. Referring to Figure 1, the tracking system 10 is a system that detects and tracks people within a monitoring area, and is configured to include two PTZ cameras 11 and 12 and a control device 20.

The PTZ cameras (hereinafter, simply referred to as cameras) 11 and 12 include a solid-state imaging device such as a CMOS sensor or a CCD sensor and a pan head. The cameras 11 and 12 generate multiple images by, for example, continuously photographing the shooting range at a fixed cycle according to instructions from the control device 20. It is preferable that the shooting cycle and shooting timing of the cameras 11 and 12 are the same or similar, but they may be different. The cameras 11 and 12 can change the shooting range by changing the pan, tilt, and zoom according to instructions from the control device 20. Here, pan means to swing the camera direction left and right, tilt means to swing the camera direction up and down, and zoom means to change the angle of view to telephoto or wide angle, respectively. The cameras 11 and 12 are installed in a distributed manner in the monitoring area so that part of their shooting ranges are common. In the example shown in FIG. 1, the camera 11 monitors monitoring areas 13A, 13B, and 13C set on a passage extending in the left-right direction of the paper. Furthermore, camera 12 monitors monitoring area 13C and monitoring area 13D set on a passageway extending in the vertical direction of the paper. Monitoring area 13C corresponds to the corner of the passageway. In other words, the area of the corner of the passageway is monitoring area 13C shared by two cameras 11 and 12. It is assumed that a person enters the passageway from the left side of the paper, passes the corner of the passageway, and exits from the bottom of the paper. Moreover, monitoring areas 13A to 13D are flat surfaces.

The control device 20 is connected to the cameras 11 and 12 by wire or wirelessly. The control device 20 calculates the camera calibration information (internal parameters and external parameters) of each camera when the cameras 11 and 12 are installed. The calculation of this camera calibration information is performed by setting the PTZ values of the cameras 11 and 12 to the reference PTZ of each camera. For example, the reference PTZ of the camera 11 is set to be a camera field of view that can capture a person in the monitoring area 13A. The reference PTZ of the camera 12 is set to be a camera field of view that can capture a person in the monitoring area 13C. When the system starts to operate, the control device 20 displays images continuously captured by the camera 11 fixed to the reference PTZ on the screen display unit. When the operator specifies a person to be tracked on the image displayed on the monitor, the control device 20 tracks the specified person while changing the PTZ value of the camera 11. The control device 20 then calculates information on the movement trajectory of the person being tracked (tracking target movement trajectory information) in real time in the world coordinate system.

On the other hand, when the system starts operating, the control device 20 calculates in real time in the world coordinate system the movement trajectory information of all people in the monitoring area 13C (matched movement trajectory information) based on images continuously captured by the camera 12 fixed to the reference PTZ. The control device 20 then matches the tracked target movement trajectory information of the person being tracked by the camera 11 with the matched movement trajectory information of all people captured by the camera 12 in real time. In this way, the control device 20 determines which person captured by the camera 12 is the person being tracked by the camera 11. Next, the control device 20 continues to track the determined person with the camera 12 while capturing the person within the capture range of the camera 12, and displays the image on the monitor of the screen display unit.

The above is an overview of the tracking system 10. Next, the control device 20 will be described in detail.

Figure 2 is a block diagram showing an example of the control device 20. Referring to Figure 2, the control device 20 includes a communication I/F (interface) unit 21, an operation input unit 22, a screen display unit 23, a memory unit 24, and a calculation processing unit 25.

The communication I/F unit 21 is composed of a data communication circuit and is configured to perform data communication with the cameras 11, 12 and other external devices (not shown) by wire or wirelessly. The operation input unit 22 is composed of operation input devices such as a keyboard and a mouse, and is configured to detect the operation of the operator and output it to the calculation processing unit 25. The screen display unit 23 is composed of a display device such as an LCD (Liquid Crystal Display), and is configured to display images captured by the cameras 11, 12, etc.

The storage unit 24 is composed of one or more storage devices of one or more types, such as a hard disk or memory, and is configured to store processing information and a program 241 required for various processes in the calculation processing unit 25. The program 241 is a program that is read into the calculation processing unit 25 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input/output function such as the communication I/F unit 21 and stored in the storage unit 24. The main processing information stored in the storage unit 24 includes camera calibration information 242-1, 242-2, reference images 243-1, 243-2, image DBs (databases) 244-1, 244-2, tracked object movement trajectory information 245, and matched movement trajectory information 246.

Camera calibration information 242-1 is the camera calibration information of camera 11 (internal parameters and external parameters of camera 11). Camera calibration information 242-2 is the camera calibration information of camera 12 (internal parameters and external parameters of camera 12).

The reference image 243-1 is an image of the monitoring area 13A taken by the camera 11 set to the pan value, tilt value, and zoom value when the camera calibration information 242-1 was obtained. In addition to the image, the reference image 243-1 may include the pan value, tilt value, and zoom value (reference PTZ value of the camera 11) of the camera 11 at the time of taking the reference image, and the camera position (position of the camera 11 in the world coordinate system). The reference image 243-2 is an image of the monitoring area 13C taken by the camera 12 set to the pan value, tilt value, and zoom value when the camera calibration information 242-2 was obtained. In addition to the image, the reference image 243-2 may include the pan value, tilt value, and zoom value (reference PTZ value of the camera 12) and the camera position (position of the camera 12 in the world coordinate system) at the time of taking the reference image.

Image DB 244-1 stores a time series of images taken by camera 11. Each image stored in image DB 244-1 is assigned the camera ID of camera 11, the time of shooting, and the PTZ value at the time of shooting. Image DB 244-2 stores a time series of images taken by camera 12. Each image stored in image DB 244-2 is assigned the camera ID of camera 12, the time of shooting, and the PTZ value at the time of shooting.

The tracking target movement trajectory information 245 is information on the movement trajectory of the tracking target person calculated based on the image captured by the camera 11. FIG. 3 is a diagram showing an example of the configuration of the tracking target movement trajectory information 245. In this example, the tracking target movement trajectory information 245 is composed of an entry 2451 composed of a tracking target ID that uniquely identifies the tracking target, the camera ID of the camera 11, and association information, and a plurality of entries 2452 that correspond one-to-one to a plurality of images captured consecutively by the camera 11. The multiple entries 2452 are connected in a row in the order of the shooting time by the association information in the entry 2451 and the association information in the entry 2452.

Each entry 2452 is composed of a shooting time, a person area, position information (image coordinate system), image features, position information (world coordinate system), movement vector, movement speed, acceleration, and association information. The shooting time indicates the shooting time of the corresponding image. The person area indicates, for example, a circumscribing rectangle of the person area in the corresponding image. The position information (image coordinate system) indicates the coordinate value in the image coordinate system of one point representing the person area in the same entry. The one point representing the person area may be, for example, the center of gravity of the person area, but is not limited thereto, and may be the center of gravity of the face or the feet. The image features indicate the image features extracted from the person area in the corresponding image. The image features may be, but are not limited to, face features, clothing features, and the size of the person. The position information (world coordinate system) is set to a coordinate value obtained by converting the position information (image coordinate system) in the same entry from the image coordinate system to the world coordinate system. The movement vector indicates the amount of movement and the direction of movement of the tracked person between the corresponding image and the adjacent image. The movement speed indicates the movement speed of the tracked person. The acceleration indicates the acceleration of the person to be tracked. The movement vector, the movement speed, and the acceleration are calculated based on, for example, the position information (world coordinate system) in the entry 2452 of the corresponding image and the adjacent image. The types of data included in the entry 2452 are merely examples and are not limited to those described above.

The matched movement trajectory information 246 is information on the movement trajectory of a person extracted from an image captured by the camera 12. The matched movement trajectory information 246 exists for each person extracted from an image captured by the camera 12. FIG. 4 is a diagram showing an example of the configuration of the matched movement trajectory information 246. The matched movement trajectory information 246 in this example is composed of an entry 2461 composed of a person ID that uniquely identifies a person, a camera ID of the camera 12, and association information, and a plurality of entries 2462 that correspond one-to-one to a plurality of images captured consecutively by the camera 12. The plurality of entries 2462 are connected in a row in the order of the shooting time by the association information in the entry 2461 and the association information in the entry 2462. Each entry 2462 is composed of data similar to the entry 2452 of the tracked movement trajectory information 245 described with reference to FIG. 3. The type of data included in the entry 2462 is an example and is not limited to the above.

The calculation processing unit 25 has one or more microprocessors such as an MPU and their peripheral circuits, and is configured to read and execute the program 241 from the storage unit 24, thereby realizing various processing units by having the above hardware and the program 241 cooperate with each other. The main processing units realized by the calculation processing unit 25 include a camera calibration unit 251 and two monitoring units 252 and 253.

The camera calibration unit 251 is configured to acquire camera calibration information for the cameras 11 and 12 through interactive processing with an operator via the operation input unit 22 and the screen display unit 23. The camera calibration method used is not particularly limited.

The monitoring unit 252 is configured to detect and track people moving within the monitoring areas 13A to 13C using the camera 11. The monitoring unit 252 includes a detection unit 2521, a tracking unit 2522, and a coordinate conversion unit 2523.

The detection unit 2521 is configured to detect a specific person as a tracking target from among all people present within the monitoring area 13A based on images continuously captured of the monitoring area 13A by the camera 11 set to the reference PTZ value.

The tracking unit 2522 is configured to track the tracking target detected by the detection unit 2521 with the camera 11 so as to capture the tracking target within the shooting range, while calculating information on the movement trajectory of the tracking target in the image coordinate system of the camera 11, and store the information in the memory unit 24 as tracked target movement trajectory information 245. Since the tracking unit 2522 tracks the tracking target by changing the PTZ value of the camera 11, the tracking target can be captured within the shooting range even if the tracking target moves from the monitoring area 13A to the monitoring area 13B or the monitoring area 13C.

The coordinate conversion unit 2523 is configured to convert the tracked object movement trajectory information 245 calculated by the tracking unit 2522 from the image coordinate system of the camera 11 to the world coordinate system. That is, the coordinate conversion unit 2523 converts the position information on the image from the image coordinate system to the world coordinate system. Here, the coordinate conversion unit 2523 may use a plane (common map) where z = 0 (i.e., height 0) as the world coordinate system. Compared to a configuration in which points on the image coordinate system are converted to a three-dimensional world coordinate system, the conversion to a two-dimensional world coordinate system reduces the amount of calculation and error. However, the coordinate conversion unit 2523 may be configured to convert points in the image coordinate system to a three-dimensional world coordinate system.

Fig. 5 is a schematic diagram for explaining an example of a method for converting position information on an image from an image coordinate system to a world coordinate system. In Fig. 5, image _I0 is an image taken by camera 11 set to the same PTZ value as when camera calibration information 242-1 was calculated. Images _I1 , _I2 , and _I3 are multiple images taken by camera 11 successively while changing the PTZ value after image _I0 was taken.

The coordinate conversion unit 2523 converts the position information on the image _I0 from the image coordinate system to the world coordinate system using the camera calibration information 242-1.

In addition, in the conversion of the position information on the image _I1 to the world coordinate system, the coordinate conversion unit 2523 first converts the position information on the image _I1 to the corresponding position information on the image _I0 using a mapping function _f1 . Next, the coordinate conversion unit 2523 converts the position information on the image _I0 obtained by the above conversion from the image coordinate system to the world coordinate system using the camera calibration information 242-1. In addition, the coordinate conversion unit 2523 calculates the above mapping function _f1 as a planar projection transformation matrix _H1 from the image _I1 to the image _I0 . That is, _f1 = _H1 . In addition, in the calculation of the planar projection transformation matrix _H1 , the coordinate conversion unit 2523 obtains a group of corresponding points between the image _I1 and the image _I0 using a known feature point extraction method, and calculates the planar projection transformation matrix _H1 from the obtained group of corresponding points. The corresponding points between the images _I1 and _I0 are, in other words, pairs of corresponding feature points between the images _I1 and _I0 .

In addition, in the conversion of the position information on the image _I2 to the world coordinate system, the coordinate conversion unit 2523 first converts the position information on the image _I2 to the corresponding position information on the image _I0 using the mapping function _f2 , and then converts the position information on the image _I0 obtained by this conversion from the image coordinate system to the world coordinate system using the camera calibration information 242-1. In addition, the coordinate conversion unit 2523 obtains the above mapping function _f2 from the mapping function _f1 and a planar projection transformation matrix _H2 from the image _I2 to the image _I1 . That is, _f2 = _f1 * _H2 . In addition, the coordinate conversion unit 2523 obtains the above planar projection transformation matrix _H2 by obtaining a corresponding point group between the image _I2 and the image _I1 using a known feature point extraction method, and calculates the planar projection transformation matrix _H2 from the obtained corresponding point group.

In addition, in the conversion of the position information on the image _I3 to the world coordinate system, the coordinate conversion unit 2523 first converts the position information on the image _I3 to the corresponding position information on the image _I0 using the mapping function _f3 , and then converts the position information on the image _I0 obtained by this conversion from the image coordinate system to the world coordinate system using the camera calibration information 242-1. In addition, the coordinate conversion unit 2523 obtains the above mapping function _f3 from the mapping function _f2 and a planar projection transformation matrix _H3 from the image _I3 to the image _I2 . That is, _f3 = _f2 * _H3 . In addition, the coordinate conversion unit 2523 obtains the above planar projection transformation matrix _H3 by obtaining a corresponding point group between the image _I3 and the image _I2 using a known feature point extraction method, and calculates the planar projection transformation matrix _H3 from the obtained corresponding point group.

The method of calculating the mapping function is not limited to the above. For example, the coordinate conversion unit 2523 may calculate the mapping function f ₃ as f ₃ =f ₁ *H ₃₁ using the mapping function f ₁ and the planar projection transformation matrix H ₃₁ from image I ₃ to image I _1. Alternatively, the coordinate conversion unit 2523 may calculate the mapping function f ₃ as f ₃ =H ₃₀ using the planar projection transformation matrix H ₃₀ from image I ₃ to image I _0. However, the common area between adjacent images captured continuously by the camera 11 while tracking a moving person so as to capture it in the shooting range is relatively large, but the common area between the images tends to become smaller as the time distance increases. It is difficult to calculate the corresponding point group between both images between images with a small common area. In addition, if the difference between the two images is large, it may not be possible to accurately calculate the corresponding point group even if there is a common area. Therefore, it is desirable that the shooting time interval of the two images for which the planar projection transformation matrix is calculated is determined taking into account the above circumstances.

2, the monitoring unit 253 is configured to use the camera 12 to detect a person being tracked by the camera 11 from among people present in the monitoring area 13C, and to track the detected person in the monitoring areas 13C and 13D. The monitoring unit 253 includes a matching unit 2531, a tracking unit 2532, and a coordinate conversion unit 2533.

The matching unit 2531 is configured to calculate information on the movement trajectory of each person moving within the monitoring area 13C (matched movement trajectory information 246) in the image coordinate system of the camera 12 from multiple images of the monitoring area 13C continuously captured by the camera 12 set to the reference PTZ value.

The coordinate conversion unit 2533 is configured to convert the matched movement trajectory information 246 calculated by the matching unit 2531 from the image coordinate system of the camera 12 to the world coordinate system using the camera calibration information 242-2. Here, like the coordinate conversion unit 2523, the coordinate conversion unit 2533 uses a plane (common map) where z = 0 (i.e., height 0) as the world coordinate system.

The matching unit 2531 is also configured to match the matched movement trajectory information 246 converted into the world coordinate system with the tracked target movement trajectory information 245 of the person being tracked by the monitoring unit 252. The matching unit 2531 is also configured to determine, based on the result of the matching, which person is the person being tracked by the camera 11, among the people moving within the monitoring area 13C photographed by the camera 12.

An example of a method for matching the tracking target movement trajectory information 245 with the matched movement trajectory information 246 in the matching unit 2531 is described below.

First, the matching unit 2531 matches the latest N frames of each trajectory. Here, N is a predetermined value of 2 or more that is determined in advance. Therefore, the matching unit 2531 extracts N entries 2452 in order from the last entry 2452 of the tracking target movement trajectory information 245, and sets the extracted N entries 2452 as the movement trajectory information of the tracking target. If the tracking target movement trajectory information 245 does not contain N entries 2452, the matching unit 2531 does not perform matching at that time. On the other hand, the matching unit 2531 extracts N entries 2462 in order from the last entry 2462 from each of one or more pieces of matched movement trajectory information 246, and sets the extracted N entries 2462 as the movement trajectory information to be matched. The matching unit 2531 does not match the matched movement trajectory information 246 in which N entries 2462 do not exist at that time. If there is no matched trajectory information 246 having N entries 2462, the matching unit 2531 does not perform matching at that time.

Next, the matching unit 2531 calculates the matching between the movement trajectory information of the tracked object and the movement trajectory information of each matched object based on the degree of match of shape and the degree of match of orientation.

In calculating the degree of shape matching, the matching unit 2531 first normalizes each trajectory by the center of gravity to eliminate the influence of absolute position. Here, the trajectory of the tracked object is obtained by connecting the position information (world coordinate system) of the N entries 2452 constituting the trajectory information of the tracked object with a line segment in the order of the shooting time. The trajectory of the matched object is obtained by connecting the position information (world coordinate system) of the N entries 2462 constituting the trajectory information of the matched object with a line segment in the order of the shooting time. Next, the matching unit 2531 obtains the error between the positions of the trajectories of the tracked object and the matched object at the same time using the least squares method or the like, and calculates a value that increases as the error decreases as the degree of shape matching.

In addition, when calculating the degree of orientation matching, the matching unit 2531 calculates a value as the degree of orientation matching that is higher the smaller the sum of the errors between the movement vectors of the movement trajectories of the tracked object and the matched object at the same time.

Next, the matching unit 2531 calculates a matching result from the degree of match of the shape and the degree of match of the orientation. For example, the matching unit 2531 may calculate the matching result by multiplying the degree of match of the shape and the degree of match of the orientation. Alternatively, the matching unit 2531 may calculate the matching result by adding the degree of match of the shape and the degree of match of the orientation.

The above is an example of a method for matching the tracked target movement trajectory information 245 and the matched movement trajectory information 246 in the matching unit 2531. However, the matching method is not limited to the above. For example, the matching unit 2531 may match both pieces of movement trajectory information taking into account the image features, movement speed, and acceleration contained in entry 2452 of the tracked target movement trajectory information 245 and entry 2462 of the matched movement trajectory information 246.

The matching unit 2531 is configured to determine which person among the people moving within the monitoring area 13C photographed by the camera 12 is the person being tracked by the camera 11 based on the result of matching the tracking target movement trajectory information 245 with one or more matched movement trajectory information 246 as described above. For example, if the highest matching result is equal to or greater than a certain threshold, the matching unit 2531 determines that the person related to the matched movement trajectory information 246 with the highest matching result is the person being tracked by the camera 11. On the other hand, if the highest matching result is not equal to or greater than a certain threshold, the matching unit 2531 determines that the person being tracked by the camera 11 is not present among the people photographed by the camera 12.

The tracking unit 2532 is configured to track the tracking target determined by the collation unit 2531 (the person being tracked by the camera 11 and captured in the camera 12) with the camera 12 so as to capture the tracking target within the shooting range. The tracking unit 2532 tracks the tracking target by changing the PTZ value of the camera 12, so that the tracking target can be captured within the shooting range even if the tracking target moves from the monitoring area 14C to the monitoring area 13D.

Next, we will explain the operation of the control device 20. First, we will explain the camera calibration that is performed before the system is put into operation.

The camera calibration unit 251 of the control device 20 calibrates the cameras 11 and 12 at any time before the system is put into operation, such as when the cameras 11 and 12 are installed.

In calibrating the camera 11, the camera calibration unit 251 first adjusts the PTZ value of the camera 11 so that the entire monitoring area 13A can be captured. Next, the camera calibration unit 251 calculates the camera calibration information of the camera 11 using a predetermined camera calibration method by interactive processing with the operator through the operation input unit 22 and the screen display unit 23. The camera calibration unit 251 stores the calculated camera calibration information as camera calibration information 242-1 in the memory unit 24. The camera calibration unit 251 also acquires the PTZ value of the camera 11 at the time of calibration as a reference PTZ value. Furthermore, the camera calibration unit 251 acquires an image of the area 13A captured by the camera 11 set to the reference PTZ value as a reference image. The camera calibration unit 251 adds the reference PTZ value to the acquired reference image and stores it in the memory unit 24 as reference image 243-1.

In calibrating the camera 12, the camera calibration unit 251 first adjusts the PTZ value of the camera 12 so that the entire monitoring area 13C (common area with the camera 11) can be captured. Next, the camera calibration unit 251 calculates the camera calibration information of the camera 12 using a predetermined camera calibration method by interactive processing with the operator through the operation input unit 22 and the screen display unit 23. The camera calibration unit 251 stores the calculated camera calibration information as camera calibration information 242-2 in the memory unit 24. The camera calibration unit 251 also acquires the PTZ value of the camera 12 at the time of calibration as a reference PTZ value. Furthermore, the camera calibration unit 251 acquires an image of the area 15 captured by the camera 12 set to the reference PTZ value as a reference image. The camera calibration unit 251 adds the reference PTZ value to the acquired reference image and stores it in the memory unit 24 as reference image 243-2.

Next, the operation of the control device 20 during system operation will be described. Figure 6 is a flowchart showing an example of a tracking process performed using the camera 11. Below, the tracking process performed by the control device 20 using the camera 11 will be described with reference to Figure 6.

First, the detection unit 2521 in the monitoring unit 252 of the control device 20 performs initialization (step S11). In this initialization, the detection unit 2521 sets the camera 11 to a reference PTZ value. In the initialization, the detection unit 2521 further clears all entries of the tracking target movement trajectory information 245. Next, the detection unit 2521 acquires an image of the monitoring area 13A taken by the camera 11 set to the reference PTZ value, stores it in the image DB 244-1, and displays it on the monitor of the screen display unit 23 (step S12). Next, the detection unit 2521 detects all people from the image stored this time using various techniques such as pattern recognition and machine learning (step S13). Next, the detection unit 2521 displays the detection result on the screen display unit 23 (step S14). For example, the detection unit 2521 generates an image in which the circumscribing rectangle for each person detected from the image is superimposed on the image and displays it on the screen display unit 23. Next, the detection unit 2521 judges whether or not a person to be tracked has been designated from the operation input unit 22 (step S15). The operator can designate a person to be tracked by, for example, performing an operation such as clicking a rectangle of a person on the image of the camera 11 displayed on the screen display unit 23. If a person to be tracked has not been designated, the detection unit 2521 returns to step S12 and repeats the same process as described above. On the other hand, if a person to be tracked has been designated, the detection unit 2521 updates the tracked object movement trajectory information 245 (step S16).

In updating the tracking target movement trajectory information 245 in step S16, the detection unit 2521 sets the ID assigned to the tracking target person specified by the operator and the camera ID of the camera 11 in the entry 2451. The detection unit 2521 also secures one free entry 2452 and associates the secured entry 2452 with the entry 2451 by setting association information in the secured entry 2452 and entry 2451. Next, the detection unit 2521 focuses on the secured entry 2452 and sets the shooting time, person area, position information (image coordinate system), and image features in the focused entry 2452, and sets the position information (world coordinate system), movement vector, movement speed, and acceleration to NULL values.

Next, the coordinate conversion unit 2523 of the monitoring unit 252 performs coordinate conversion on the entry of interest 2452 of the tracked object movement trajectory information 245 (step S17). In this step S17, the coordinate conversion unit 2523 first uses the camera calibration information 242-1 to convert the position information (image coordinate system) set in the entry of interest 2452 from the image coordinate system to the world coordinate system. Next, the coordinate conversion unit 2523 sets the position information (world coordinate system) obtained by the above conversion to the entry of interest 25452.

Next, the tracking unit 2522 of the monitoring unit 252 performs tracking control (step S18). For example, the tracking unit 2522 generates a command to control the PTZ value of the camera 11 according to the position information (image coordinate system) set in the entry under focus 2452, and transmits the command to the camera 11 through the communication I/F unit 21. At this time, for example, the tracking unit 2522 may adjust the pan and tilt so that the center of gravity of the circumscribed rectangle of the tracked person represented by the position information (image coordinate system) of the entry under focus 2452 is displayed at the center of the image, and adjust the zoom so that the entire circumscribed rectangle fits within a predetermined angle of view. The camera 11 changes the imaging range by changing the pan, tilt, and zoom in response to the command.

Next, the tracking unit 2522 acquires an image captured by the camera 11 set to the changed PTZ value, stores it in the image DB 244-1, and displays it on the screen display unit 23 (step S19). Next, the tracking unit 2522 uses various techniques such as pattern recognition and machine learning to detect the person to be tracked from the image just saved (step S20), generates an image in which the circumscribing rectangle of the detected person is superimposed on the image, and displays it on the screen display unit 23 (step S21). This allows the operator to check the tracking status of the specified person on the image displayed on the screen display unit 23 in real time.

Next, the tracking unit 2522 updates the tracking target movement trajectory information 245 (step S22). In updating the tracking target movement trajectory information 245 in step S22, the tracking unit 2522 first secures one free entry 2452, sets association information between the secured free entry 2452 and the entry of interest 2452, and then shifts attention to the secured free entry 2452. Next, the tracking unit 2522 sets the shooting time, person area, position information (image coordinate system), and image features in the newly focused entry 2452, and sets the position information (world coordinate system), movement vector, movement speed, and acceleration to NULL values.

Next, the coordinate conversion unit 2523 of the monitoring unit 252 converts the position information (image coordinate system) set in the entry of interest 2452 from the image coordinate system to the world coordinate system (step S23). The image corresponding to the entry of interest 2452 is an image taken by the camera 11 set to a PTZ value different from that of the reference image. Therefore, as described with reference to FIG. 5, the coordinate conversion unit 2523 first calculates a mapping function that converts the position information (image coordinate system) set in the entry of interest 2452 to position information on the reference image, and converts the position information (image coordinate system) to position information on the reference image using the calculated mapping function. Next, the coordinate conversion unit 2523 converts the position information on the reference image obtained by the conversion from the image coordinate system to the world coordinate system using the camera calibration information 242-1. In addition, in step S 23 , the coordinate conversion unit 2523 calculates a movement vector, a movement speed, and an acceleration based on the position information (world coordinate system) of the current entry 2452 and the previous entry 2452 , and sets them in the current entry 2452 .

Next, the tracking unit 2522 of the monitoring unit 252 determines whether or not to end tracking (step S24). For example, the tracking unit 2522 may determine to end tracking when it detects that the person to be tracked is no longer present in the monitoring areas 13A, 13B, and 13C. Alternatively, the tracking unit 2522 may end tracking when it detects that the person to be tracked is no longer in a state where it cannot be tracked by the camera 11. If the tracking unit 2522 does not determine to end tracking, it returns to step S18 and repeats the same process as the above-mentioned process. As a result, the tracking of the person to be tracked by the camera 11 continues, and the tracking target movement trajectory information 245 is further updated accordingly.

If the tracking unit 2522 determines to end tracking, it determines whether or not to end monitoring (step S25). For example, the tracking unit 2522 determines to end monitoring when an end monitoring command is input from the operator via the operation input unit 22. If it is not determined to end monitoring, the monitoring unit 252 returns to step S11 and repeats the same processing as described above. If it is determined to end monitoring, the monitoring unit 252 ends the processing shown in FIG. 6.

Figure 7 is a flowchart showing an example of a tracking process performed using the camera 12. Below, the tracking process performed by the control device 20 using the camera 12 will be explained with reference to Figure 7.

First, the matching unit 2531 in the monitoring unit 253 of the control device 20 performs initialization (step S31). In this initialization, the matching unit 2531 sets the camera 12 to a reference PTZ value. In addition, in the initialization, the matching unit 2531 also clears all entries of the matched movement trajectory information 246. Next, the matching unit 2531 acquires an image of the monitoring area 13C taken by the camera 12 set to the reference PTZ value, stores it in the image DB 244-2, and displays it on the monitor of the screen display unit 23 (step S32). Next, the matching unit 2531 detects all people from the image stored this time using various methods such as pattern recognition and machine learning (step S33). Next, the matching unit 2531 displays the detection result on the screen display unit 23 (step S34).

Next, the matching unit 2531 updates the matched movement trajectory information 246 based on the person detection result (step S35). Since all people detected from the image of the camera 12 acquired first after initialization are people detected for the first time, the matching unit 2531 assigns one piece of matched movement trajectory information 246 to each detected person in step S35, and performs the following process for each piece of assigned matched movement trajectory information 246.

First, the matching unit 2531 sets the person ID assigned to the detected person and the camera ID of the camera 12 in entry 2461 of the matched movement trajectory information 246. Next, the matching unit 2531 secures one free entry 2462, sets information that correlates the secured free entry 2462 with entry 2461, and then focuses on the secured free entry 2462. Next, the matching unit 2531 sets the shooting time, person area, position information (image coordinate system), and image features in the focused entry 2462, and sets the position information (world coordinate system), movement vector, movement speed, and acceleration to NULL values.

Next, the coordinate conversion unit 2533 of the monitoring unit 253 performs the following processing on the entry of interest 2462 (step S36). First, the coordinate conversion unit 2533 uses the camera calibration information 442-2 to convert the position information (image coordinate system) set in the entry of interest 2462 from the image coordinate system to the world coordinate system. Next, the coordinate conversion unit 2533 sets the position information in the world coordinate system obtained by the above conversion as the position information (world coordinate system) of the entry of interest 2462.

Next, the matching unit 2531 of the monitoring unit 253 matches the tracking target movement trajectory information 245 with one or more pieces of matched movement trajectory information 246 (step S37). The matching unit 2531 determines, through this matching, which person among the people moving within the monitoring area 13C photographed by the camera 12 is the person being tracked by the camera 11. Next, the matching unit 2531 determines whether or not the detection of the person being tracked by the camera 11 has been successful (step S38). If the above detection is not successful, the matching unit 2531 returns to step S32 and repeats the same processing as the above-mentioned processing. On the other hand, if the above detection is successful, the matching unit 2531 transmits the person ID of the person determined to be the person being tracked by the camera 11 to the tracking unit 2532.

The tracking unit 2532 focuses on the last entry 2462 of the matched movement trajectory information 246 in which the person ID received from the matching unit 2531 is set in the entry 2461, and tracks the tracking target person using the camera 12. For example, the tracking unit 2532 generates a command to control the PTZ value of the camera 12 according to the position information (image coordinate system) set in the entry under focus 2462, and transmits it to the camera 12 through the communication I/F unit 21. At this time, for example, the tracking unit 2532 may adjust the pan and tilt so that the center of gravity of the circumscribed rectangle of the tracking target person represented by the position information (image coordinate system) of the entry under focus 2462 is displayed at the center of the image, and adjust the zoom so that the entire circumscribed rectangle fits within a predetermined angle of view. The camera 12 changes the imaging range by changing the pan, tilt, and zoom in response to the command. Furthermore, in step S39, the tracking unit 2532 acquires an image captured by the camera 12 after the PTZ change, and performs processes such as detection of a person to be tracked and monitor display on the screen display unit 23.

Furthermore, the tracking unit 2532 judges whether or not to end tracking (step S40). For example, the tracking unit 2532 may judge to end tracking when it detects that the person to be tracked is no longer present in the monitoring areas 13C and 13D. Alternatively, the tracking unit 2532 may end tracking when it detects that the person to be tracked is in a state where it cannot be tracked by the camera 12. If the tracking unit 2532 does not judge to end tracking, it returns to step S38 and repeats the same processing as the above-mentioned processing. As a result, the tracking unit 2532 continues tracking the person to be tracked by the camera 12. If the tracking unit 2532 judges to end tracking, it judges whether or not to end monitoring (step S41). For example, the tracking unit 2532 judges to end monitoring when a monitoring end command is input from the operator through the operation input unit 22. If it is not judged to end monitoring, the monitoring unit 253 returns to step S31 and repeats the same processing as the above-mentioned processing. If it is determined that monitoring should be ended, the monitoring unit 253 ends the process shown in FIG.

Thus, according to this embodiment, the tracking unit 2522 acquires position information (image coordinate system) of the tracking target person from multiple images captured continuously while tracking the moving tracking target person with the camera 11 having a tracking function that captures the tracking target person within the shooting range, and calculates each entry 2452 of the tracking target movement trajectory information 245. In addition, the coordinate conversion unit 2523 calculates a mapping function from the image corresponding to each entry 2452 captured by the camera 11 to the reference image, converts the position information (image coordinate system) of each entry 2452 of the tracking target movement trajectory information 245 to position information on the reference image using this calculated mapping function, and converts the position information on the reference image obtained by this conversion from the image coordinate system to the world coordinate system using the camera calibration information 242-1. Unlike the camera calibration information, the above mapping function can be calculated without manual intervention. Therefore, according to this embodiment, it is possible to calculate movement trajectory information, which is a time series of the position of the person in real space, while tracking the person with a PTZ camera having a tracking function that captures the moving person within the shooting range. As a result, the movement trajectories of a person can be collated in real time between multiple PTZ cameras, enabling automatic collation of moving objects between PTZ cameras.

Next, we will explain a variation of this embodiment.

The coordinate transformation unit 2523 may calculate a planar projection transformation matrix from an image captured by the camera 11 to a reference image based on the PTZ value of the camera 11 when the image was captured and the reference PTZ value.

The camera 11 may be configured to be equipped with a computer, and all or part of the functions of the monitoring unit 252 may be installed on that computer. Similarly, the camera 12 may be configured to be equipped with a computer, and all or part of the functions of the monitoring unit 253 may be installed on that computer. In addition, the operation input unit 22, screen display unit 23, and memory unit 24 may be installed on a computer connected to the cameras 11 and 12 via a network.

The monitored area may be anywhere other than a corridor, such as a shop, a factory, a station platform, a playground, a gymnasium, etc.

The object to be tracked may be a moving object other than a person, such as an animal, a car, a walking robot, etc.

The number of cameras that share a portion of the camera field of view is not limited to two, camera 11 and camera 12, but may be three or more.

[Second embodiment]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

Referring to Figure 8, the movement trajectory information processing device 30 of this embodiment comprises a calculation means 31 and a conversion means 32.

The calculation means 31 is configured to calculate movement trajectory information, which is a time series of position information on an image of an object, from a plurality of images continuously captured by a camera capturing a moving object within a capture range. The calculation means 31 can be configured, for example, in the same manner as the tracking unit 2522 in FIG. 2, but is not limited thereto.

The conversion means 32 is configured to calculate a mapping function from the image to a reference image, calculate position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and convert the calculated position information on the reference image into position information in the world coordinate system using camera calibration information. The conversion means 32 can be configured, for example, in the same manner as the coordinate conversion unit 2523 in FIG. 2, but is not limited thereto.

The movement trajectory information processing device 30 configured as described above operates as follows. That is, the calculation means 31 calculates movement trajectory information, which is a time series of position information on the image of the object, from multiple images of the object captured continuously by a camera that captures the moving object within its shooting range. Next, the conversion means 32 calculates a mapping function from the above image to a reference image, calculates position information on the reference image corresponding to the position information on the image of the object using this calculated mapping function, and converts the calculated position information on the reference image into position information in the world coordinate system using the camera calibration information.

The movement trajectory information processing device 30 configured and operated as described above can calculate movement trajectory information, which is a time series of the position of an object in real space, from multiple images of the object captured continuously by a camera that captures the moving object within its shooting range. This is because the mapping function can be calculated without manual intervention, unlike camera calibration information.

Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

For example, the movement trajectory of an object calculated by the present invention may be used for purposes other than monitoring and tracking, such as personal authentication and abnormal behavior detection based on the movement trajectory. Furthermore, the camera used in the present invention does not necessarily need to have a tracking function. For example, the present invention can also be applied to cases where the photographer manually changes the camera's attitude to capture a moving object within the shooting range, and calculates the movement trajectory information of the object from multiple images captured in succession while tracking the object.

It can be used in systems that detect and track moving objects from multiple images taken continuously with a camera that captures moving objects within its shooting range, such as a PTZ camera.

A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
[Appendix 1]
a calculation means for calculating movement trajectory information, which is a time series of position information of an object on an image, from a plurality of images continuously captured by a camera capturing a moving object within a capturing range of the object;
a conversion means for calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
A movement trajectory information processing device comprising:
[Appendix 2]
the conversion means calculates a planar projective transformation matrix from the image to the reference image, and calculates the mapping function using the calculated planar projective transformation matrix;
2. The movement trajectory information processing device according to claim 1.
[Appendix 3]
Further comprising a matching means for matching the movement trajectory information with other movement trajectory information.
3. The movement trajectory information processing device according to claim 1 or 2.
[Appendix 4]
The other movement trajectory information is movement trajectory information calculated from a plurality of images continuously captured by another camera installed so that a part of a camera field of view overlaps with the camera.
4. The movement trajectory information processing device according to claim 3.
[Appendix 5]
The camera is a PTZ camera.
5. A movement trajectory information processing device according to any one of claims 1 to 4.
[Appendix 6]
Calculating movement trajectory information, which is a time series of position information of the object on the image, from a plurality of images continuously captured by a camera capturing the moving object within a shooting range of the object;
calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
A method for processing movement trajectory information.
[Appendix 7]
In the conversion, a planar projective transformation matrix from the image to the reference image is calculated, and the mapping function is calculated using the calculated planar projective transformation matrix.
7. A movement trajectory information processing method according to claim 6.
[Appendix 8]
Furthermore, the movement trajectory information is collated with other movement trajectory information.
8. The movement trajectory information processing method according to claim 6 or 7.
[Appendix 9]
The other movement trajectory information is movement trajectory information calculated from a plurality of images continuously captured by another camera installed so that a part of a camera field of view overlaps with the camera.
9. The movement trajectory information processing method according to claim 8.
[Appendix 10]
On the computer,
A process of calculating movement trajectory information, which is a time series of position information on an image of a moving object, from a plurality of images continuously captured by a camera capturing the object within a shooting range of the object;
a process of calculating a mapping function from the image to a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using camera calibration information;
A computer-readable recording medium having a program recorded thereon for causing a computer to carry out the above.

10 Tracking system 11, 12 PTZ cameras 13A to 13D Monitoring area 20 Control device 21 Communication I/F unit 22 Operation input unit 23 Screen display unit 24 Memory unit 25 Calculation processing unit 30 Movement trajectory information processing device 31 Calculation means 32 Conversion means 241 Program 242-1, 242-2 Camera calibration information 243-1, 243-2 Reference image 244-1, 244-2 Image DB
245 Tracking target movement trajectory information 246 Matched movement trajectory information 251 Camera calibration unit 252, 253 Monitoring unit 2521 Detection unit 2522, 2532 Tracking unit 2523, 2533 Coordinate conversion unit 2531 Matching unit

Claims (9)

a calculation means for calculating movement trajectory information, which is a time series of position information on an image of an object, from a plurality of images continuously captured by a PTZ camera that captures a moving object within a shooting range of the object;
a conversion means for calculating a mapping function from the image of the shooting range captured by the PTZ camera set to the pan value, tilt value, and zoom value when the camera calibration information was obtained as a reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using the camera calibration information;
A movement trajectory information processing device comprising: the conversion means calculates a planar projective transformation matrix from the image to the reference image, and calculates the mapping function using the calculated planar projective transformation matrix;
The movement trajectory information processing device according to claim 1 . The system further includes a matching unit for matching the movement trajectory information with other movement trajectory information.
The movement trajectory information processing device according to claim 1 . The other movement trajectory information is movement trajectory information calculated from a plurality of images continuously captured by another PTZ camera installed so that a part of a camera field of view overlaps with the PTZ camera.
The movement trajectory information processing device according to claim 3 . Calculating movement trajectory information, which is a time series of position information of the object on the image, from a plurality of images continuously captured by a PTZ camera that captures a moving object within a shooting range of the object;
an image of the shooting range captured by the PTZ camera set to the pan value, tilt value, and zoom value when the camera calibration information was obtained is used as a reference image, a mapping function is calculated from the image to the reference image, position information on the reference image corresponding to position information on the image of the object is calculated using the calculated mapping function, and the calculated position information on the reference image is converted to position information in a world coordinate system using the camera calibration information;
A method for processing movement trajectory information. In the conversion, a planar projective transformation matrix from the image to the reference image is calculated, and the mapping function is calculated using the calculated planar projective transformation matrix.
The movement trajectory information processing method according to claim 5 . Furthermore, the movement trajectory information is collated with other movement trajectory information.
The movement trajectory information processing method according to claim 5 or 6 . The other movement trajectory information is movement trajectory information calculated from a plurality of images continuously captured by another PTZ camera installed so that a part of a camera field of view overlaps with the PTZ camera.
The movement trajectory information processing method according to claim 7 . On the computer,
A process of calculating movement trajectory information, which is a time series of position information of an object on an image, from a plurality of images continuously captured by a PTZ camera that captures a moving object within a shooting range of the object;
a process of setting an image of the shooting range captured by the PTZ camera set to the pan value, tilt value, and zoom value when the camera calibration information was obtained as a reference image, calculating a mapping function from the image to the reference image, calculating position information on the reference image corresponding to position information on the image of the object using the calculated mapping function, and converting the calculated position information on the reference image into position information in a world coordinate system using the camera calibration information;
A program to carry out the above.