JP7837684B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to object tracking technology.

In situations where multiple subjects (objects) are simultaneously present in an imaging scene and each object moves independently, proximity between objects often causes tracking errors. Patent Document 1 discloses a technique for detecting humans in each frame of an image sequence (a series of continuously captured images) of a scene in which multiple humans are moving, and switching the method of updating the tracking model depending on whether the humans are in close proximity or not. This technique makes it possible to stably track objects even when occlusion relationships occur between them.

International Publication No. 2017/199840

The technology described in Patent Document 1 above detects objects in each frame beforehand, determining their positions and number. It then determines proximity based on the distance between objects within a frame, enabling tracking of each object. In other words, this method requires pre-determining the attributes of the objects to be tracked (human, dog, or specific body parts such as head or feet) and designing a model suitable for tracking them in advance. Therefore, it could not be applied to image sequences of arbitrary imaging scenes where the object attributes were unknown.

The image processing apparatus according to this disclosure includes: acquisition means for acquiring sequence data including shape data of objects in a plurality of consecutive frames; matching means for matching a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame consecutive to the first frame, based on the distance between the point clouds; and tracking means for tracking the object across the plurality of consecutive frames by assigning an ID to the point cloud corresponding to the shape data contained in each frame based on the result of the matching, wherein the matching means performs the matching using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different .

According to the technology disclosed herein, stable object tracking becomes possible even when the attributes of objects within the imaging scene are unknown.

(a) and (b) are diagrams illustrating ideal object tracking. Hardware configuration diagram of an image processing device. Software configuration diagram of an image processing device. A flowchart illustrating the object tracking process. A flowchart detailing the process of mapping point clouds between adjacent frames. (a) is a schematic representation of the point cloud between adjacent frames, and (b) is a diagram showing the distance between the centroids of the point clouds shown in (a). A diagram showing an example of the results of the linking process. A flowchart showing the details of the object tracking process according to Embodiment 1. (a) and (b) are diagrams illustrating the determination of the shape attributes of an object. Figures (a) to (c) illustrate specific examples of cases where the shape attribute determination results are duplicated. (a) to (c) are diagrams illustrating how the point cloud is tracked. (a) to (c) are diagrams illustrating asymmetric distance. A diagram showing how objects of similar size come into contact and then separate. A flowchart showing the details of the object tracking process according to Embodiment 2.

The embodiments for implementing this model will be described below with reference to the drawings and other materials. Note that the following embodiments are not intended to limit the technology of this disclosure, and not all configurations described in the following embodiments are necessarily essential for solving the problem.

[Embodiment 1]
First, we will explain ideal object tracking with a concrete example. Figure 1(a) shows three frames extracted from a scene in which a person is juggling a ball, corresponding to specific times (t-1, t, t+1). In this case, a binary image (silhouette image) is given as shape data representing the two-dimensional shape of the object, in which the foreground region corresponding to the silhouettes of the person and the ball is represented by black pixels (pixel value "1") and the rest of the region by white pixels (pixel value "0"). Here, the center of the black pixels in the silhouette image is considered a point, and the point cloud, which is a collection of points, is treated as representing the two-dimensional shape of the object. In the frame at time t-1, the silhouette of the person and the silhouette of the ball are separated, so there are two point clouds, PC_0 and PC_1, in that frame. On the other hand, in the frame at time t, the silhouette of the person and the silhouette of the ball overlap, so there is only one point cloud, PC_0. Note that here, the centers of all pixels constituting the foreground region are considered as point clouds, but it is also acceptable to consider only the centers of pixels in the contour portion of the foreground region as point clouds.

In this embodiment, the target of processing is an image sequence consisting of multiple consecutive frames, each containing shape data representing the two-dimensional shape of an object (hereinafter referred to as "sequence data"). The objective is to achieve highly accurate object tracking by correctly assigning object IDs to the point cloud representing the object shape in each frame of the input sequence data, as shown in Figure 1(b).

Furthermore, the shape data included in each frame is not limited to representing the two-dimensional shape of an object; it may also be volume data representing a three-dimensional shape. For example, if the volume data is composed of voxels, the set of center points of each voxel can be considered as a point cloud representing the object's shape. Alternatively, only the point cloud distributed on the object's surface may be used. Also, if the shape data is mesh data representing the object's surface, for example, a set of polygons formed by topologically connecting point clouds distributed on the surface, then the faces of the mesh can be considered as a set of an infinite number of point clouds.

<Hardware Configuration>
Figure 2 shows an example of the hardware configuration of the image processing device 100. The image processing device 100, which is an information processing device, has a CPU 211, ROM 212, RAM 213, auxiliary storage device 214, operation unit 215, communication I/F 216, and bus 217.

The CPU 211 controls the entire image processing device 100 using computer programs and data stored in the ROM 212 or RAM 213, thereby realizing each function of the image processing device 100. The image processing device 100 may also have one or more dedicated hardware components or a GPU (Graphics Processing Unit) separate from the CPU 211. Furthermore, at least a portion of the processing performed by the CPU 211 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

ROM 212 stores programs and other data that do not require modification. RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, as well as data supplied from external sources via the communication interface 217. The auxiliary storage device 214 is composed of, for example, a hard disk drive and stores various types of data, such as input image sequences.

The operation unit 215 is composed of, for example, a keyboard or mouse, and receives various instructions from the user and inputs them to the CPU 211. The CPU 211 operates as a display control unit that controls the display device 104, and as an operation control unit that controls the operation unit 215. The communication interface 216 is used for communication between the image processing device 100 and external devices. For example, if the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication interface 216. If the image processing device 100 has a function for wireless communication with an external device, the communication interface 216 is equipped with an antenna.

Bus 217 connects the various parts of the image processing device 100 to transmit information.

<Software Configuration>
Figure 3 is a diagram showing an example of the software configuration of the image processing device 100. As shown in Figure 3, the image processing device 100 has the following functional units: a data input unit 301, a mapping unit 302, a tracking unit 303, and a data output unit 304. The tracking unit 303 also has an attribute determination unit 303a and an ID assignment unit 303b. Figure 4 is a flowchart showing the object tracking flow according to this embodiment. The operation of each functional unit of the image processing device 100 will be described below with reference to the flowchart in Figure 4. In the following description, the symbol "S" means step.

In S401, the data input unit 301 receives sequence data to be tracked from an external device.

In S402, the mapping unit 302 analyzes the point clouds representing object shapes based on the shape data contained in each frame of the frame group constituting the input sequence data, and performs a process to map point clouds between adjacent frames. The mapping relationship between point clouds between adjacent frames is determined by direction and type. Directions include "forward direction," which is the direction in which time progresses, and "reverse direction," which is the direction in which time goes backward. There are also three types: 1. "One-to-one correspondence," where point clouds in adjacent frames have a one-to-one relationship. 2. "Combined correspondence," where one point cloud in the frame of interest corresponds to multiple point clouds in the previous frame. 3. "Separated correspondence," where one point cloud in the frame of interest corresponds to multiple point clouds in the next frame. The determined point cloud mapping relationships are output, for example, in the form of a graph composed of nodes and edges, and stored in RAM 213. Details of the mapping process will be described later.

In S403, the tracking unit 303 performs a process to track each object across multiple consecutive frames that constitute the sequence data, based on the correspondence between point clouds determined by the correspondence unit 302. In this tracking process, first, the attribute determination unit 303a determines whether the point cloud corresponding to the shape data of each frame represents the shape of a single object or the combined shape of multiple objects. Here, the point cloud representing the shape of a single object is called a "single point cloud," and the point cloud representing the combined shape of multiple objects that corresponds to a group of multiple objects is called a "non-single point cloud." Following this determination of shape attributes, the ID assignment unit 303b assigns an object ID to the point cloud of each frame and tracks the object based on the correspondence between point clouds obtained in S402 and the result of the shape attribute determination above. At this time, using the characteristics of the object being tracked, it selects a point cloud with similar characteristics from the point cloud of the next frame. In this embodiment, the total number of points constituting the point cloud is used as the characteristics of the object. This is equivalent to using the number of pixels in the foreground region and its contours in a silhouette image as its features. Furthermore, if the shape data is volume data representing the three-dimensional shape of an object, this is equivalent to using the object's volume (or the object's surface area in the case of a point cloud of a surface) as its features. Details of the tracking process will be described later.

In S404, the data output unit 305 outputs the results of the tracking process performed by the tracking unit 303 as tracking data.

In this embodiment, the above-described functional units are explained as being realized by the CPU 211 executing a predetermined program; however, some of these may be realized by hardware.

<Matching process>
Figure 5 is a flowchart detailing the process of associating point clouds between adjacent frames in S402 described above. The following explanation will be given in detail with reference to the flowchart in Figure 5.

In S501, point clouds that are close together are linked between adjacent frames in the sequence data. This linking is performed in both the forward direction (time advancing) and the reverse direction (time reversing). For example, in the forward direction, between frames at time t and t+1, the point cloud at time t is found to have the smallest distance from the point cloud at time t+1, and that distance is below a threshold. This is designated as the "forward correspondence." In the reverse direction, the point cloud at time t+1 is found to have the smallest distance from the point cloud at time t, and that distance is below a threshold. This is designated as the "reverse correspondence." Let's explain this using a concrete example. Figure 6(a) is a schematic representation of point clouds between adjacent frames. Currently, in the frame at time t, there is a large circular point group 601 and a small circular point group 602, shown by solid lines. In the frame at time t+1, there is a single point group 603, shown by a dashed line, where the large and small circles are in contact. Figure 6(b) shows the distance between the centroids of the point groups shown in Figure 6(a), with black circles representing the centroids of each point group. The double arrow 604 represents the distance between the centroids of point group 601 at time t and point group 603 at time t+1, and the double arrow 605 represents the distance between the centroids of point group 602 at time t and point group 603 at time t+1. At this time, point group 601 has a forward correspondence with point group 603, and point group 603 has an inverse correspondence with point group 601. Similarly, point group 602 has a forward correspondence with point group 603, and point group 603 has an inverse correspondence with point group 602. Here, if the preset threshold is smaller than the distance 605, it will be determined that point cloud 602 and point cloud 603 do not correspond to each other. Therefore, in this embodiment, it is necessary to set a threshold that allows a distance of the magnitude indicated by the double arrow 605. Figure 7 is a graph structure representing an example of the result of linking when sequence data of a scene in which three objects exist in the imaging space is input. The sequence data contains frames for four time points t0, t1, t2, and t3, and point clouds are linked to each pair of t0 and t1, t1 and t2, and t2 and t3. In the graph of Figure 7, the vertical direction represents the difference in the position of point clouds within the same frame, and the horizontal direction represents the time axis. The arrows connecting point clouds between frames indicate the direction and type of linking. For example, point cloud 1 in the frame at time t0 is connected to point cloud 3 in the frame at time t1 by a double arrow, and to point cloud 4 by a single arrow pointing to itself. This means that point cloud 1 corresponds directly to point cloud 3, point cloud 3 corresponds inversely to point cloud 1, and furthermore, point cloud 4 corresponds inversely to point cloud 1. This process is performed on all frames within the sequence data in this step.

In S502, the processing to proceed to the next step is determined based on the number of point clouds in the next frame linked in S501 to the point cloud of interest in the frame of interest within the frame group that constitutes the sequence data. The frames are set as frames of interest in order from the first frame of the sequence data. If the number of point clouds linked to the point cloud of interest in the set frame of interest (the number of arrows connecting the point clouds in Figure 7 above) is 0, the process proceeds to S507; if it is 1, the process proceeds to S503; and if it is 2 or more, the process proceeds to S506.

In S503, the next processing step is determined based on the number of inverse correspondences associated with the point group of interest. If there is one inverse correspondence associated with the point group of interest, the process proceeds to S504; if there are two, it proceeds to S505.

In S504, the transition states of the point group of interest are determined in a "one-to-one correspondence" for both the forward and reverse directions.

In S505, the transition state of the point group of interest is determined to be "joined" in the forward direction and "separated" in the reverse direction.

In S506, the transition state of the point group of interest is determined as "separated correspondence" in the forward direction and "joined correspondence" in the reverse direction.

In S507, it is determined whether all point clouds in the frame of interest have been processed. If there are any unprocessed point clouds, the next point cloud of interest is selected from them and processing continues. If all point clouds have been processed, the process proceeds to S508.

In S508, it is determined whether all frames constituting the sequence data have been processed. If there are unprocessed frames, the next frame to focus on is selected from among them, and processing continues. If all frames have been processed, this process ends.

The above describes the mapping process. Now, referring to Figure 7 mentioned earlier, we will show a concrete example of the result of the mapping process, using the case where the frame of interest is the frame at time t0.

≪When the point group of interest is point group 1≫
The number of arrows connecting the point cloud in the next frame, t1, to point cloud 3 is two: a double arrow to point cloud 3 and a single arrow to point cloud 4. Therefore, the determination in S502 leads to proceeding to S506. In S506, it is determined that the forward direction corresponds to "separation" and the reverse direction corresponds to "combination". Looking at the forward direction, point cloud 1 is separated into point cloud 3 and point cloud 4 in the next frame (frame t1), so the determination result of "separation" is accurate. Also, regarding the reverse direction, point cloud 3 and point cloud 4 in frame t1 are combined into point cloud 1 in frame t0, so the determination result of "combination" is also accurate.

≪When the point group of interest is point group 2≫
The number of arrows connecting the point cloud in the next frame, t1, to point cloud 5 is one double-headed arrow, so the decision in S502 leads to proceeding to S503. Then, the number of inverse correspondences is only one, from point cloud 5 to point cloud 2, so the decision in S503 leads to proceeding to S504. Then, in S504, a "one-to-one correspondence" is determined for both the forward and reverse directions. Point cloud 2 has a one-to-one correspondence with point cloud 5 in the next frame (frame t1), so the "one-to-one correspondence" is accurate.

<Tracking process>
Figure 8 is a flowchart detailing the process of tracking an object across multiple consecutive frames in S403 described above. The following explanation will be given in detail with reference to the flowchart in Figure 8.

In S801, the attribute determination unit 303a performs a shape attribute determination for all point clouds in the sequence data, determining whether they are single point clouds or multi-point clouds. This shape attribute determination consists of a primary process and a secondary process.

<<Primary Processing>>
In the primary processing, point clouds determined to be connected in either the forward or reverse direction in relation to the preceding and succeeding frames in the aforementioned mapping process are determined to be "non-single point clouds," and point clouds determined to be separated are determined to be "single point clouds." Figure 9(a) shows the results of the primary determination performed on each point cloud included in the four frames shown in Figure 7 above. Point clouds determined to be single point clouds are marked with the symbol "S," and point clouds determined to be non-single point clouds are marked with the symbol "M."

≪Secondary Processing≫
Next, in the secondary processing, for point clouds that were determined to have a one-to-one correspondence with the preceding and succeeding frames in the aforementioned mapping process, the results of their initial determination are propagated to the point clouds of the preceding and succeeding frames that are associated with them. As a result, as shown in Figure 9(b), all point clouds are assigned a determination result of whether they are single point clouds or not.

It is important to note here that point clouds that are not linked to any point clouds in the preceding or following frames (point cloud 6 in frame t1 of Figure 7) are deleted without being determined to be either a standalone point cloud or a non-standalone point cloud. This is equivalent to determining that such a point cloud is noise, preventing the generation and assignment of object IDs to point clouds that contain false shapes of non-existent objects due to errors in generating shape data.

≪Duplicate judgment results≫
In addition, if a single point cloud and a non-single point cloud overlap in the primary and secondary processing, the determination of the non-single point cloud takes precedence in both cases. Figures 10(a) to (c) illustrate specific examples of cases where the shape attribute determination results overlap. Figure 10(a) shows the state before the shape attribute determination, (b) shows the result of the primary processing, and (c) shows the result of the secondary processing. As shown by the dashed box in Figure 10(b), in the primary processing, point cloud 1 of the frame at time t1 is determined to be a separate correspondence in relation to point cloud 0 of the frame at time t0, and is therefore determined to be a single point cloud (S). However, since it is determined to be a pair correspondence in relation to point cloud 3 of the frame at time t2, which was determined to be a non-single point cloud, it is rewritten to a non-single point cloud (M) in the secondary processing, as shown by the dashed box in Figure 10(c).

Returning to the explanation of the flow in Figure 8, the processing from S802 onward is executed by the ID assignment unit 303b.

In S802, an object ID is assigned to the point cloud of interest at the starting frame in a predetermined tracking direction. At this time, the number of points constituting the point cloud, which indicates the size of the object being tracked, is stored in RAM 213 as characteristic information of the object being tracked. By setting a lower limit on the number of points that can be stored as characteristic information, it is possible to prevent point clouds (noise) that are too small to be objects in the imaging scene from being selected as the point cloud of interest. Note that the starting frame when tracking in the forward direction is the first frame of the sequence data, and the starting frame when tracking in the reverse direction is the last frame of the sequence data. Either forward or reverse tracking can be performed first; once one time axis direction is completed, the remaining time axis direction can be tracked next. The object ID assigned to the point cloud of interest (hereinafter simply referred to as "ID") can be any information that uniquely identifies the object.

In S803, the next processing step is determined based on whether the point group of interest, to which an ID has been assigned, is a single point group or not. If the point group of interest is a single point group, the process proceeds to S804; otherwise, it proceeds to S805.

In S804, the feature information of the object being tracked is updated. Specifically, the average number of points in the single point cloud of interest for the same ID up to the present time is calculated, and the obtained average value is stored in RAM 213 as a new feature of the object being tracked. In this way, by updating the information indicating the object's features only when the point cloud of interest is a single point cloud, it is possible to avoid distortion of the original features of the object being tracked due to contact with other objects. If the shape data is, for example, mesh data representing the three-dimensional shape of the object, the sum of the polygon areas (surface area) can be used as the object's feature information.

In S805, it is determined whether there is a next frame for the frame to which the point group of interest belongs. If the time axis direction of the processing target is forward, the next frame for the target frame at time t is the frame at time t+1. If the time axis direction of the processing target is reverse, the next frame for the target frame at time t is the frame at time t-1. If a next frame exists, the process proceeds to S806; otherwise, it proceeds to S810.

In S806, based on the object's feature information, a point cloud similar to the point cloud of interest is selected from the point clouds of the next frame. In this embodiment, the point cloud with the number of points closest to the number of points held as feature information is selected from the point clouds of the next frame. Let's explain this using a specific example. Suppose the number of points in the feature information is 100. If there are two point clouds in the next frame, one consisting of 95 points and the other of 102 points, the latter point cloud will be selected. Now, suppose there are two point clouds in the next frame: point cloud A consisting of 98 points and point cloud B consisting of 102 points. In this case, the difference between the number of points held ("100") and the number of points is 2 in both cases, so it is impossible to determine which to select as is. Therefore, in such cases, the ratio of each value to the number of points held is calculated as the size evaluation value, and the one whose calculated value is closest to 1 is selected. In the example above, point cloud A has a size evaluation value of approximately 100 ÷ 99 ≈ 1.0204, while point cloud B has a size evaluation value of approximately 0.9804. Therefore, point cloud B, which has a size evaluation value closer to 1, is selected. Alternatively, a threshold can be set to prevent the selection of point clouds that are too small as objects (noise), and the above selection can be made from point clouds with a number of points exceeding this threshold. Thus, in this embodiment, point clouds similar to the point cloud of interest are selected from the next frame based on their size. This prevents the IDs from being swapped between multiple objects, even if, for example, multiple objects of clearly different sizes, such as a ball and a person, come into contact at one time and separate at another.

In S807, the process is divided based on whether or not a point cloud similar to the point cloud of interest can be selected from the point cloud of the next frame. If a similar point cloud can be selected, the process proceeds to S808; otherwise, it proceeds to S810.

In S808, the point cloud of the next frame selected in S806 is assigned the same ID as the ID assigned to the point cloud of interest. In the following S809, an update process is performed to set the point cloud of the next frame, which now has the same ID, as the next point cloud of interest. After the update is complete, the process returns to S803 and is repeated.

In S810, it is determined whether the above processing has been completed for all point clouds included in the starting frame. If there are any unprocessed point clouds in the starting frame, the process returns to S802 to determine the next point cloud of interest and continues processing. On the other hand, if all point clouds in the starting frame have been processed, the process proceeds to S811.

In S811, it is determined whether point cloud tracking is complete in both the forward and reverse directions. If there are any unprocessed directions, the process proceeds to S812, where the time axis direction of the processing target is changed to the opposite direction, and the processing from S802 onwards is restarted.

The above describes the tracking process according to this embodiment. Now, referring to Figures 11(a) to (c), we will explain how the point cloud is tracked in chronological order.

In Figure 11(a), point cloud 1101 is the first point cloud of interest in the frame at time t0, which is the starting frame, when the time axis direction of the first processing target is forward. First, point cloud 1101 is assigned the ID "0" (S802). Then, point cloud 1101 is not a single point cloud (No in S803), and there is a next frame (Yes in S805). Therefore, point cloud 1102, which has the closest object features, is selected from the next frame at time t1 (S806), and the same ID "0" is assigned to it (S808). Similar processing is performed up to the final frame at time t3, and the same ID "0" is assigned to point cloud 1103 in the frame at time t2 and point cloud 1104 in the frame at time t3.

Figure 11(b) shows the results of a similar tracking process, starting from the second point cloud 1105 in the frame at time t0, which serves as the starting frame. The same ID "1" assigned to point cloud 1105 is also assigned to the corresponding point clouds 1106, 1107, and 1108 in the remaining frames. At this point, tracking of all point clouds in the starting frame in the forward direction is complete. Looking at Figure 11(b), we can see that point clouds 1009 and 1110, which have not been assigned IDs, remain in the frames at time t1 and t3, respectively. This is because, although there are originally three objects in the imaging space, in the starting frame in the forward direction, two of the three objects merged, resulting in only two point clouds representing the object shapes. Considering the possibility of such situations, in this embodiment, after tracking in the forward direction is completed, tracking in the reverse direction is performed.

Figure 11(c) shows the result of performing a similar trace in reverse, starting with the frame at time t3 as the starting frame. By tracing in reverse, starting from point cloud 1109 in the frame at time t3, the common ID "2" is assigned to point clouds 1109 and 1110, which were not assigned IDs in the forward trace. In this way, IDs are assigned to every point cloud within the multiple frames that make up the sequence data without any omissions.

<Different example>
Simply performing tracking processes in both forward and reverse directions, starting from the first and last frames of the sequence data, may result in some point clouds remaining without assigned IDs. Therefore, it is also possible to perform further tracking processes in both forward and reverse directions, starting from the point clouds of intermediate frames. This reduces the risk of point clouds remaining without assigned IDs.

As described above, according to this embodiment, in cases where multiple objects within the imaging scene repeatedly come into contact with and separate from each other, it is possible to accurately track each object without having to pre-determine whether it is a person or an object.

[Embodiment 2]
In Embodiment 1, the distance between the centroids of the point clouds was used to determine the correspondence between point clouds in adjacent frames. This distance between centroids is called the "symmetric distance" because the distance from point cloud A to point cloud B is equal to the distance from point cloud B to point cloud A. Next, Embodiment 2 describes a method in which the distance between point clouds is defined by an "asymmetric distance," where the distance from point cloud A to point cloud B is different to the distance from point cloud B to point cloud A, and the correspondence between point clouds is determined using this asymmetric distance. Note that the explanation of content common to Embodiment 1 will be omitted, and the following explanation will focus on the differences.

<About asymmetrical distance>
First, let's explain the fundamental concept of asymmetric distance in this embodiment. Let X be the set of points in point cloud A, and Y be the set of points in point cloud B. The distance d from point cloud A to point cloud B is given by the following equation (1).

In equation (1) above, the distance d can be determined by finding the distance from a point x, which is an element of X, to the nearest element of Y, and then obtaining the maximum value of this distance over the entire X. In Figure 12(a), the point groups 1201 and 1202, shown by solid lines, belong to the frame at time t, and the point group 1203, shown by a dashed line, belongs to the frame at time t+1. In this case, the distance d from point group 1203 to point group 1201 (point group 1201 → point group 1203) and the distance d from point group 1203 to point group 1201 (point group 1203 → point group 1201) are represented by arrows 1204 and 1205, respectively, in Figure 12(b). Furthermore, the distance d(1202→1203) between point cloud 1202 and point cloud 1203 is represented by arrow 1207, and the distance d(1203→1202) between point cloud 1203 and point cloud 1202 is represented by arrow 1206. Figures 12(b) and (c) show that the value differs depending on the direction in which the distance is calculated.

The advantage of using the asymmetric distance described above is that when multiple objects transition from a state of contact to a state of separation, the correspondence between larger and smaller objects remains stable. In Embodiment 1, it was necessary to set a large threshold for correspondence so that the point cloud 1202 in the frame at time t and the point cloud 1203 in the frame at time t+1 were linked. However, setting a large threshold can lead to incorrect correspondences, increasing the risk of tracking errors. By using asymmetric distance, it is possible to obtain a graph that accurately represents the correspondence without breaking the correspondence relationship, even with a small threshold.

<Tracking process>
Figure 14 is a flowchart detailing the process of tracking an object over multiple consecutive frames according to this embodiment. The following explanation will focus on the differences between this embodiment and the flowchart in Figure 8 of Embodiment 1, referring to the flowchart in Figure 14.

In S1401, similar to S801, the shape attributes are determined for each point cloud within each frame. In the following S1402, similar to S803, the process is divided based on whether the point cloud of interest in the target frame is a single point cloud or not. If the point cloud of interest is a single point cloud, the process proceeds to S1403; otherwise, it proceeds to S1404.

In S1403, the feature information of the object being tracked, corresponding to the point cloud of interest, is updated. In Embodiment 1, the total number of points in the point cloud, which indicates the size of the object, was used as feature information. However, in this embodiment, information indicating the position of the object is also used. Specifically, information indicating how far the current position is from the position when the point cloud of interest was last (most recently) a single point cloud (travel distance) is used as feature information. If the point cloud of interest merges with another point cloud and then separates, the distance between the position of the point cloud of interest before merging and the positions of each candidate point cloud is calculated, and the point cloud with the shorter distance (specifically, the one with a larger value obtained by dividing the size evaluation value by the distance) is selected. By selecting the point cloud in the next frame while taking the position of the object into consideration in this way, stable tracking becomes possible even if multiple objects of similar size come into contact and then separate, as shown in Figure 13. For example, in the specific example used in the explanation of S806 above (point clouds A and B with only a slight difference in size evaluation values), suppose the distance from the position where the point cloud of interest was last a single point cloud was 0.5m for point cloud A and 0.8m for point cloud B. Now, the size evaluation value of point cloud A is 1.0204 and the size evaluation value of point cloud B is 0.9804. Dividing each size evaluation value by the distance gives 2.0408 for point cloud A and 1.2255 for point cloud B. Therefore, since the value after division is larger for point cloud A, point cloud A is selected.

In S1404, similar to S805, it is determined whether there is a next frame for the target frame to which the group of points of interest belongs. If a next frame exists, the process proceeds to S1405; otherwise, it proceeds to S1409.

In S1405, based on the object's feature information, a point cloud similar to the point cloud of interest is selected from the point cloud of the next frame. In this embodiment, in addition to information indicating the size of the point cloud, a point cloud similar to the point cloud of interest is selected from the next frame based on the position of the most recent single point cloud.

In S1406, similar to S807, the process to proceed to the next step is determined by whether or not a point cloud similar to the point cloud of interest can be selected from the next frame. If a similar point cloud can be selected, the process proceeds to S1407; otherwise, it proceeds to S1409.

In S1407, the count value of a counter (hereinafter referred to as the "tracking counter") that counts the number of times the corresponding point cloud was selected from the next frame is incremented (+1). In the following S1408, an update process is performed to set the point cloud in the next frame selected in S1405 as the next point cloud of interest. After the update is complete, the process returns to S1402 and the same process is repeated.

In S1409, it is determined whether the tracking counter value (i.e., the number of consecutive tracking attempts) is above a threshold. If it is above the threshold, the process proceeds to S1410; otherwise, it proceeds to S1411. In S1410, a common ID is assigned to the point cloud of each frame that was successfully tracked consecutively. This process ensures that only objects whose tracking count exceeds the threshold are assigned an ID, while objects whose tracking count falls below the threshold are not.

In S1411, similar to S810, it is determined whether the above processing has been completed for all point clouds included in the starting frame. If there are unprocessed point clouds in the starting frame, the process returns to S1402 to determine the next point cloud of interest and continues. On the other hand, if all point clouds in the starting frame have been processed, the process proceeds to S1412.

S1412 and S1413 correspond to S811 and S812, respectively. That is, it is determined whether point cloud tracking is complete in both the forward and reverse directions (S1412), and if there are any unprocessed directions, processing is started in the opposite direction (S1413).

The above describes the tracking process according to this embodiment.

<Variation 1>
Even if the number of times tracking is successful exceeds a threshold, if the object being tracked moves out of (disappears from) the image space midway through, it may be judged as noise and an ID may not be assigned. For example, suppose the threshold is set to "3" on the premise that an ID will be assigned if tracking is successful for three consecutive frames or more. In this case, if an object that should be in the center of the field suddenly disappears in the fourth frame, it is highly likely that the object is a false object that does not actually exist, so an ID will not be assigned even if tracking is successful more times than the threshold. This cancellation of ID assignment is determined, for example, by whether the position of the point cloud in the frame at the end of tracking is more than a certain distance from the edge of the image space. Specifically, if the distance from the centroid of the point cloud at the end of tracking to the edge of the captured image is greater than or equal to the threshold, the ID assignment will be canceled. Note that point clouds for which ID assignment has been canceled will not be used as a new starting point for tracking, and tracking will be terminated.

<Modified Example 2>
Even if the object's position is included as feature information for the object being tracked, accurate tracking becomes difficult in cases where the object's position changes frame by frame (for example, when two people are holding hands and spinning around). To address such cases, it is conceivable to use the color information (pixel values) from the image captured in the last frame where the object was a single point cloud as feature information. Specifically, the image region corresponding to the point cloud in the captured image is divided into multiple parts (for example, the upper half and the lower half), the median pixel value is calculated for each divided region, and the color difference between the divided regions is used as the selection criterion. This is intended to use the colors of the upper and lower body clothing (uniforms) as features when the object in the captured scene is a human, but it can also be applied to objects other than humans. The use of color information described above is just one example; the image region corresponding to the point cloud can also be divided into the left half and the right half, or further divided into smaller parts. In addition, text information such as the number on the back of the uniform can be obtained through image analysis and combined.

As described above, this embodiment makes it possible to suppress the issuance of IDs for shape data of objects that do not actually exist but are mixed into the sequence data. In other words, it has the effect of reducing noise in the sequence data. As a result, it becomes possible to achieve more accurate tracking.

(Other examples)
This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

100 Image processing device 301 Data input unit 302 Correspondence unit 303 Tracking unit

Claims (21)

Acquisition means for acquiring sequence data including shape data of objects in multiple consecutive frames,
A correspondence means that associates a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame continuous with the first frame, using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different .
Based on the results of the aforementioned correspondence, a determination means is provided to determine whether the point cloud corresponding to the shape data of the frames included in multiple frames corresponds to a single object or to a group of multiple objects.
A tracking means that tracks the object across multiple consecutive frames by assigning an ID to the point cloud corresponding to the shape data contained in each frame based on the results of the correspondence and the results of the determination,
An image processing apparatus characterized by having The image processing apparatus according to claim 1, characterized in that the matching means matches the second point cloud, which is closest to the first point cloud, to the first point cloud. The image processing apparatus according to claim 2, characterized in that the mapping means does not map the second point cloud to the first point cloud if the distance of the nearest second point cloud to the first point cloud is greater than a threshold. The image processing apparatus according to any one of claims 1 to 3, characterized in that the tracking means tracks the object by sequentially performing a process for a plurality of consecutive frames, starting from a point cloud of interest included in a starting frame in a predetermined time axis direction, and selecting a point cloud similar to the point cloud of interest from among the point clouds included in the next frame based on feature information indicating the characteristics of the object. The aforementioned tracking means is
If a point cloud representing the individual shape of the object appears during the aforementioned tracking process, the feature information of the object is updated.
The image processing apparatus according to any one of claims 1 to 4 . The aforementioned correspondence means is,
In the sequence data, for both the forward direction (time moving forward) and the reverse direction (time moving backward), the point clouds that are closest to each other in adjacent frames within the multiple consecutive frames are linked together.
If the number of point clouds associated with the point cloud of interest in the frame of interest among the aforementioned consecutive frames is one, and the number of point clouds associated with the point cloud of interest in the reverse direction is one, then a one-to-one correspondence is determined for both the forward and reverse directions.
If the number of point clouds associated with the point cloud of interest in the frame of interest is two or more, the correspondence is determined to be separated in the forward direction and joined in the reverse direction.
If the number of point clouds associated with the point cloud of interest in the frame of interest among the aforementioned consecutive frames is two or more, and the number of point clouds associated with the point cloud of interest in the reverse direction is two or more, then the forward direction is determined to be a combined correspondence, and the reverse direction is determined to be a separated correspondence.
The determination means determines that a point group determined to be a combined correspondence in either the forward or reverse direction is not a point group representing a single shape, and determines that a point group determined to be a separated correspondence is a point group representing a single shape.
The image processing apparatus according to any one of claims 1 to 5 . The image processing apparatus according to any one of claims 1 to 6, characterized in that, if the determination means determines that it is a point cloud representing a single shape and determines that it is not a single shape, the determination means prioritizes the determination that it is not a single shape. The image processing apparatus according to any one of claims 1 to 7, further characterized in that the determination means propagates the result of the determination concerning itself to the point clouds in the preceding and succeeding frames associated with the point cloud that has been determined to be a paired pair. The image processing apparatus according to any one of claims 1 to 8, characterized in that, as a result of performing a process to propagate the result of the determination, if the determination result that it is a point cloud representing a single shape and the determination result that it is not a single shape overlap, the determination result that it is not a single shape is given priority. The image processing apparatus according to any one of claims 1 to 9 , characterized in that the determination means deletes point clouds that are not associated with other point clouds after the process of propagating the result of the determination has been performed. The image processing apparatus according to claim 4 or 5 , characterized in that the aforementioned feature information is information indicating the size of the object. The image processing apparatus according to claim 4 or 5 , characterized in that the characteristic information is information indicating the volume of the object. The image processing apparatus according to claim 4 or 5 , characterized in that the aforementioned feature information is information indicating the surface area of the object. The image processing apparatus according to any one of claims 11 to 13 , characterized in that the feature information is information on the number of points constituting a point cloud corresponding to the shape data. The image processing apparatus according to claim 5 , characterized in that the feature information is information indicating the position of the point cloud that was last determined to represent the individual shape of the object in the tracking. The image processing apparatus according to claim 5 , characterized in that the feature information is color information in the captured image of the frame to which the point cloud that was last determined to represent the individual shape of the object in the tracking belongs. The image processing apparatus according to any one of claims 1 to 16 , characterized in that the tracking means performs the tracking in both the forward direction (time advancing) and the reverse direction (time reversing) of the sequence data. The image processing apparatus according to any one of claims 1 to 17 , characterized in that the tracking means does not assign the ID if the number of frames that can be tracked is less than or equal to a threshold. The image processing apparatus according to any one of claims 1 to 18 , characterized in that the tracking means does not assign an ID if the position of the point cloud at the time the tracking is completed is more than a certain distance from the edge of the imaging space. A step of obtaining sequence data containing the shape data of objects in multiple consecutive frames,
The steps include: associating a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame continuous with the first frame, using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different;
Based on the results of the aforementioned correspondence, the step of determining whether the point cloud corresponding to the shape data of the frames included in multiple frames corresponds to a single object or to a group of multiple objects,
Based on the results of the correspondence and the results of the determination, an ID is assigned to the point cloud corresponding to the shape data contained in each frame, and the object is tracked across a plurality of consecutive frames.
An image processing method characterized by including A program for causing a computer to function as an image processing device according to any one of claims 1 to 19 .