JP7707643B2 - Identification program, identification method, and information processing device - Google Patents

Description

The present invention relates to an identification program, an identification method, and an information processing device.

In recent years, for example, changes in lifestyles and labor shortages have created a demand for automating and streamlining store operations in the retail industry. For example, there is a growing interest in using video data captured inside stores by surveillance cameras to analyze purchasing behavior and suspicious behavior.

For example, attempts are being made to analyze consumer shopping behavior within a store from video data and use this information to attract new customers and improve the efficiency of store operations. There are also attempts to use the results of behavioral analysis using video data to detect suspicious behavior, such as leaving a store without scanning items, in stores that use unmanned registers.

In this regard, there are known technologies relating to detecting people from images and identifying people (for example, Patent Documents 1 to 3).

JP 2010-239992 A JP 2019-144830 A JP 2020-198053 A

When analyzing the behavior of a person captured in a video, for example, the person is detected from the video, the person is identified, and their movements are tracked. However, at least a portion of a person captured in a video may be hidden behind an obstacle, resulting in failure to identify and track the person.

In one aspect, the present invention aims to perform person identification with high accuracy even when at least a portion of a person in a video is hidden behind an obstacle.

An identification program according to one embodiment of the present invention includes a process for detecting a human region from a plurality of frame images included in video data, a process for performing skeletal detection on a person appearing in the human region detected from the plurality of frame images to obtain skeletal information of the person, a process for correcting the human region to match the size of the person's whole body based on the skeletal information of the person to generate a full-body-sized human region, and a process for correcting the human region to match the size of half of the person based on the skeletal information of the person, and a process for correcting the human region to match the size of half of the person based on the skeletal information of the person, and if a similarity between at least one of the full-body-sized human region and the half-body-sized human region between a first human region detected from a first frame image among the plurality of frame images and a second human region detected from a second frame image satisfies a predetermined condition. and a process of identifying the first and second human regions as human regions of the same person, assuming that the person reflected in the first human region and the person reflected in the second human region are the same person, wherein the process of identifying includes, when a person is hidden in at least one of the first human region detected from the first frame image and the second human region detected from the second frame image by an obstacle that blocks part of an object behind the person, the similarity between the half-body-sized human region corresponding to the first human region and the half-body-sized human region corresponding to the second human region, and the similarity between the full-body-sized human region corresponding to the first human region and the full-body-sized human region corresponding to the second human region, is evaluated based on a similarity between the first human region and the second human region.

People can be identified with high accuracy even when at least a portion of the person in the video is hidden behind an obstacle.

FIG. 1 illustrates a motion detection system according to an embodiment. FIG. 1 is a diagram illustrating a block configuration of an information processing device according to an embodiment. 1A and 1B are diagrams showing frame images of video data captured by an exemplary imaging device. FIG. 13 illustrates an exemplary human region detection result. FIG. 1 illustrates an exemplary person identification. 11A and 11B are diagrams illustrating correction of a human region according to an embodiment; FIG. 11 is a diagram illustrating evaluation target information according to the embodiment. 10A and 10B are diagrams illustrating obstacle information according to the embodiment; FIG. 2 shows an exemplary two frame image. 11A and 11B are diagrams illustrating examples of human regions detected by performing human object detection on two frame images. 13 is a diagram showing an example in which a human region is corrected to match the size of the entire body based on skeletal information; FIG. 13 is a diagram showing an example in which a human region is corrected to match the size of half the body based on skeletal information; FIG. FIG. 13 is a diagram illustrating an example of a similarity evaluation according to the embodiment. FIG. 13 is a diagram illustrating human regions of the same person linked using weighted matching. FIG. 11 is a diagram illustrating an example of an operation flow of a person's movement detection process according to the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer for implementing the information processing device according to the embodiment.

Several embodiments of the present invention will be described in detail below with reference to the drawings. Note that the same reference numerals are used to designate corresponding elements in multiple drawings.

FIG. 1 is a diagram illustrating a motion detection system 100 according to an embodiment. The motion detection system 100 may include, for example, an information processing device 101 and an imaging device 102. The information processing device 101 may be, for example, a computer with a computing function, such as a server computer, a personal computer (PC), a mobile PC, or a tablet terminal.

The image capture device 102 is, for example, a camera that generates video data. In one example, the image capture device 102 may be, for example, a surveillance camera installed in a store in the retail industry, etc., which captures people visiting the store, etc., and generates video data. The information processing device 101 may, for example, perform human movement detection on the video data captured by the image capture device 102. Note that, for example, multiple image capture devices 102 may be installed in a store. Also, the information processing device 101 may, for example, receive video data from the image capture device 102, or may obtain video data captured by the image capture device 102 via another device.

2 is a diagram illustrating a block configuration of the information processing device 101 according to the embodiment. The information processing device 101 includes, for example, a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 includes, for example, a detection unit 211, an acquisition unit 212, a generation unit 213, and an identification unit 214, and may also include other functional units. The storage unit 202 of the information processing device 101 stores, for example, video data captured by the imaging device 102 and information such as obstacle information 800. The communication unit 203 communicates with other devices according to, for example, instructions from the control unit 201. For example, the control unit 201 may acquire video data from the imaging device 102 via the communication unit 203. Details of each of these units and details of the information stored in the storage unit 202 will be described later.

As described above, when analyzing the behavior of a person appearing in a video, for example, the movement of the person appearing in the video is tracked. In tracking the movement of a person, the control unit 201 performs object detection of a person on the image of each frame of the video, for example. In one example, the control unit 201 may detect a human region from the image of each frame of the video using a machine learning-based technique such as deep learning. For example, the object detection of a person may be performed using techniques such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), and R-CNN (Region Convolutional Neural Network). In one example, the human region is a bounding box.

The control unit 201 may then use, for example, a same person determination model (e.g., person re-identification) to identify human regions of the same person in multiple temporally consecutive frame images of the video, and track the movement of the person. Note that the identification of the same person may be performed, for example, by obtaining an evaluation value indicating a similarity or distance measure that indicates how similar a human region detected from a certain frame image is to a comparison human region detected from another frame image. In one example, the control unit 201 may identify human regions that are similar and satisfy a predetermined condition in a certain frame image and another frame image as human regions of the same person based on the evaluation value, and generate a track that represents the trajectory of the human region over time.

For example, the control unit 201 may input a human region in a certain frame image and a human region in another frame image into a same person determination model. The same person determination model may be generated by learning using deep learning so as to output the similarity (or distance measure) of the images of the two human regions. Then, the control unit 201 may use the same person determination model to calculate the similarity between the human region in a certain frame image and the human region in another frame image. Next, the control unit 201 may generate a track indicating the trajectory of the person's movement by linking the human regions of the same person using the similarity using weighted matching such as the Hungarian method.

However, for example, at least a portion of a person in a video may be hidden behind an obstacle, causing the range of the person included in the generated person region to be uneven. As a result, the accuracy of person identification and tracking may decrease.

Figure 3 is a diagram showing a frame image of video data captured by an exemplary imaging device 102. As shown in Figure 3, the frame image shows multiple people. Among the people captured in the frame image, some people's entire bodies are captured, while others have parts of their bodies hidden by obstacles such as shelves and carts. For example, when object detection of people is performed on such a frame image, a human region that surrounds various areas of the person may be detected.

Figure 4 shows an example of the detection result of a human region 401. In Figure 4(a), the entire body of the person is visible, so the human region 401 is set to surround the entire body of the person.

On the other hand, in Figures 4(b) and 4(c), part of the person's body is hidden by a shelf 402, and as a result, a human region 401 is set for the person's upper body. Note that the size of the human region 401 set for the person may vary depending on the extent to which the person's body is visible and the person's posture, etc. For example, human regions 401 of different sizes are set in Figures 4(b) and 4(c).

In addition, in Figures 4(d) and 4(e), part of the person's body is hidden by a cart 403. The cart 403 has a frame structure, and part of the person's body is visible through the cart 403. For example, suppose that the person's body is hidden by an obstacle that does not completely block the person's figure, but rather blocks the person's body so that part of an object behind it is visible. In this case, the size of the person region 401 set for the person may change depending on how the person's body appears through the obstacle. For example, in Figure 4(d), the person region 401 is set for the entire person's body, but in Figure 4(e), the person region 401 is set for the upper half of the person's body.

If the human region 401 is set unevenly in this way across various areas of the person's body, then when trying to identify the same person between frame images, there may be differences in the features even for the same person, which may result in failed identification.

FIG. 5 is a diagram showing an exemplary person identification. FIG. 5(a) illustrates an example of an evaluation of the similarity between human regions 401 of the same person detected in a certain frame image i and another frame image i+1. For example, the control unit 201 inputs two images, an image of a human region 401 detected in a certain frame image i and an image of a human region 401 detected in another frame image i+1 to be compared, to the same person determination model. The same person determination model generates a feature vector representing the characteristics of the person shown in the image for each image. Note that the same person determination model may be executed by, for example, the control unit 201. Then, the same person determination model may determine that the people shown in the two human regions 401 are the same person when the similarity of the obtained feature vectors is similar enough to satisfy a predetermined condition.

In the example of FIG. 5(a), a human region 401 including the entire body is detected in both a certain frame image i and another frame image i+1. Therefore, when a feature vector that characterizes a person is extracted from the image in the human region 401, similar feature vectors can be obtained if the person is the same. Therefore, human regions 401 with high similarity in feature vectors between a certain frame image i and another frame image i+1 can be linked as belonging to the same person.

However, for example, when a part of a person is hidden by an obstacle such as a cart, the whole body may be detected as the human region 401, or only a part of the body may be detected as the human region 401. For example, in FIG. 5B, in a certain frame image i, the whole body can be detected as the human region 401 even though a cart is present, but in another frame image i+1, the human region 401 is detected for a part of the body due to the body being hidden by the cart. For example, even when the human region 401 is detected for a part of the body in this way, the feature vector is generated based on the human region 401, so that even if it is the same person, there may be differences in the feature vector. As a result, the similarity between the two human regions may decrease even if they are the same person, and tracking of the same person may not work well. In this way, the non-uniformity of the range of the person included in the human region 401 may have a negative effect on person identification.

Therefore, in the embodiment described below, the control unit 201 corrects the size of the human area based on the person's skeletal information. For example, when the inside of a store is photographed with the image capture device 102, the image capture device 102 is often installed near the ceiling of the store and captures the inside of the store from above diagonally downward. Also, for example, obstacles in the store such as shelves and carts often hide the lower half of a person's body, but the upper half of the person's body can often be photographed.

In this case, for example, by performing person detection and skeletal detection on a frame image showing the upper body, it is possible to obtain skeletal information of the person's upper body. Then, it is possible to correct the size of the person area from the skeletal information of the upper body so that a specified range of the upper body is included. It is also possible to correct the size of the person area from the skeletal information of the upper body so that the entire body of the person is included.

In this way, by correcting the size of the human region so that it surrounds a specified area of the person's body, it becomes possible to uniformize the area of the person's body contained in the human region and evaluate the similarity. This makes it possible to improve the accuracy of person identification and person tracking.

The embodiment will be described in more detail below. In the embodiment, the control unit 201 corrects the size of the human region based on skeletal information detected from the human region.

[Human area correction]
The control unit 201 can specify the position of the person's skeleton, such as the position of the person's waist or the position of the head, by performing skeleton detection on the detected human region, for example. Note that the skeleton detection may be performed using a method such as OpenPose, Mask R-CNN, DeepPose, or PoseNet.

Figure 6 is a diagram illustrating an example of human region size correction according to an embodiment. Figure 6(a) shows an example in which the lower half of a person's body is hidden behind a cart, and a human region has been detected in an area including the upper body and part of the lower body. Additionally, information on the position of the skeleton as a result of performing skeleton detection on the human region is shown by a circle, and the position of the head 601 and the position of the waist 602 can be identified from the information on the position of the skeleton.

In the embodiment, the control unit 201 corrects the size of the human region based on the position of the detected skeleton. For example, the height direction length of the person's upper body is defined as the length from the head position 601 to the waist position 602. In this case, for example, as shown in FIG. 6(b), the size of the human region is corrected so that the height direction length of the person's body is the range of the upper body from the head position 601 to the waist position 602, thereby making it possible to set the human region according to the size of the person's upper body. This makes it possible to make the range of the person's body included in the human region uniform, thereby improving the comparison accuracy of the human region. Note that the height direction of the person's body may be, for example, the direction of height.

The control unit 201 may also correct the size of the human region so that the human region is set for the entire body. For example, it is estimated that the length of the upper body and the length of the lower body are approximately equal. Therefore, for example, the control unit 201 may obtain the length of the upper body from the head position 601 to the waist position 602, and set a human region of a size corresponding to the entire body of the person by doubling that length and extending the human region in the direction of the lower body. For example, as shown in FIG. 6(c), the size of the human region can be set to twice the size of the range from the head position 601 to the waist position 602 and extending the human region in the direction of the lower body, so that the human region can be set to match the size of the entire body of the person.

As described above, the control unit 201 can correct the human region based on skeletal information so that the range of the person's body included in the human region becomes a predetermined range, such as the upper body and the entire body. This makes it possible to make the range of the person's body included in the human region used for person identification uniform, thereby improving the accuracy of person identification.

Note that the correction of the human region is not limited to the upper body and the entire body, and the control unit 201 may correct the size of the human region for a person based on skeletal information so as to encompass other areas. For example, in another embodiment, the size of the human region may be set to match the size of other halves of the body, such as the lower body instead of the upper body.

In an embodiment, the control unit 201 may, for example, use the corrected human region to evaluate the similarity and perform person identification. In one example, the control unit 201 may identify a human region of the same person as a certain human region based on a full-body-sized human region obtained by correction and a human region of an upper-body-sized human region.

In this case, the control unit 201 may determine the corrected human area to be used in the similarity evaluation depending on the relationship between the two human areas for which similarity is evaluated and the obstacle.

FIG. 7 is a diagram illustrating evaluation target information 700 according to an embodiment. In the evaluation target information 700, two human regions for which similarity is evaluated and information specifying a corrected human region used in the similarity evaluation depending on the relationship with an obstacle are registered. Note that in the evaluation target information 700, the query human region may be, for example, a human region in which a certain person appears among a plurality of human regions detected from a certain frame image. Note that here, the certain frame image may be, for example, referred to as a first frame image. Also, the query human region may be, for example, referred to as a first human region.

The human region to be compared may be a human region to be compared that is evaluated for similarity with a human region that shows a certain person. The human region to be compared may be, for example, one of a plurality of human regions detected from another frame image. Note that here, the other frame image may be called, for example, a second frame image. The human region to be compared may also be called, for example, a second human region.

Then, for example, the similarity between two human regions, the query human region and the comparison human region, is evaluated. Here, if both human regions are not hidden by obstacles and both have their entire bodies visible (no obstacles in FIG. 7), the control unit 201 may evaluate the similarity using a human region of full body size, as shown in the evaluation target information 700.

On the other hand, for example, at least one of the human regions may be partially hidden by an obstacle, such as a cart, such that the person is partially visible through the obstacle. In this case, information about the person partially visible through the obstacle may be useful in identifying the same person. On the other hand, depending on the visibility of the region partially visible through the obstacle, it may be preferable to evaluate the similarity using only the upper body without including the lower body. Thus, for example, when a person is hidden by an obstacle that blocks part of an object behind the person, such as a cart, so that part of the object is visible, the control unit 201 may evaluate the similarity using both the whole body and half of the body (e.g., the upper body) human regions.

Also, for example, when at least one human region is hidden by an obstacle that completely blocks an object behind a shelf or the like, the control unit 201 may evaluate the similarity using only half of the human region (e.g., the upper body).

In this way, by setting the human region used to evaluate the similarity according to the relationship between the query human region and the comparison human region, and the obstacle, it is possible to evaluate the similarity using human regions that contain the same area of a person. This improves the accuracy of identifying the same person.

Also, for example, in an embodiment, when a person is hidden by an obstacle that obscures part of an object behind the person, the similarity may be evaluated using both the whole body and half of the body (e.g., the upper body). This can improve the accuracy of identifying the same person.

Below, an example of the process for identifying the same person according to the embodiment will be described with reference to Figures 8 to 15.

FIG. 8 is a diagram illustrating obstacle information 800 according to an embodiment. The obstacle information 800 may include information indicating an area in which an obstacle is captured within the imaging range of the imaging device 102. In the example of FIG. 8, the obstacle information 800 includes, for example, an obstacle ID (identifier) for identifying an obstacle and information indicating an area in a frame image in which the obstacle is captured, which are associated with each other. Note that the obstacle information 800 may include information about obstacles that are not moved very often, such as shelves. The obstacle information 800 may also include information about the front-to-back relationship indicating whether the obstacle is in front of or behind another area, such as an aisle.

For example, for obstacles that move, such as carts, the control unit 201 may use other techniques, such as object detection, to identify the position of the obstacle.

Figure 9 is a diagram showing two exemplary frame images. Figure 9 shows two frame images, a certain frame image i (Figure 9(a)) included in video data, and another frame image i+1 (Figure 9(b)). The control unit 201 may perform human object detection on these two frame images and detect human regions.

Figure 10 shows the human regions detected by performing human object detection on the two frame images in Figure 9. Note that in Figure 10(a), the control unit 201 assigns IDs a to d to the human regions in order to identify each human region. Also, in Figure 10(b), the control unit 201 assigns IDs 1 to 4 to the human regions in order to identify each human region.

In the example of FIG. 10, the frame image includes a person with parts of his/her body hidden by obstacles such as a shelf and a cart, and as a result, human regions are set for various ranges of the person's body, such as the whole body or the upper half of the body. For example, in a certain frame image i in FIG. 10(a), the lower halves of the bodies of persons with ID=b and ID=d are hidden by a shelf or a cart, so human regions are detected for the upper halves of the bodies. Also, for example, in another frame image i+1 in FIG. 10(b), the lower half of the body of person with ID=4 is hidden by a shelf and cannot be seen at all, so human regions are detected for the upper half of the body.

For example, if the range of a person's body included in a detected human region is uneven like this, attempting to identify the same person by evaluating the similarity as is may result in identifying the wrong person as the same person. For this reason, in the embodiment, the range in which the human region is set for the person's body is corrected based on information about the person's skeleton included in the human region.

Figure 11 shows an example where the size of the human region is corrected based on skeletal information so that the human region includes the entire body of a person. Also, Figure 12 shows an example where the size of the human region is corrected based on skeletal information so that the human region includes the upper half of the person's body.

Then, the control unit 201 calculates the similarity using at least one of the full-body size human region and the upper-body size human region based on the relationship between the two human regions to be evaluated for similarity and the obstacles, as shown in the evaluation target information 700.

Figure 13 is a diagram showing an example of similarity evaluation according to the embodiment. In Figure 13 (a), the control unit 201 sets human region: a in a certain frame image i as the query human region, and calculates the similarity between the human regions of ID: 1 to ID: 4 in another frame image i+1 as comparison target human regions. Note that the query human region: a does not overlap with the position of an obstacle, and the entire body of the person is included in the human region. Meanwhile, the comparison target human region of ID: 1 does not overlap with an obstacle. Therefore, the control unit 201 may evaluate the similarity between the two evaluation target human regions using the whole-body size human regions generated in Figure 11, as shown in the evaluation target information 700.

Furthermore, since the human region of ID: 2 to be compared does not overlap with an obstacle, the control unit 201 may similarly evaluate the similarity between the full-body size human regions generated in FIG. 11.

The comparison target human region with ID: 3 overlaps with an obstacle to the cart. Therefore, the control unit 201 may evaluate the similarity using both the similarity between the full-body size human regions generated in FIG. 11 and the similarity between the upper body size human regions generated in FIG. 12, as shown in the evaluation target information 700. In one example, the control unit 201 may use the average value of the similarity between the full-body size human regions and the similarity between the upper body size human regions as the similarity between the query human region: a and the comparison target human region: 3, as shown in FIG. 13(a).

The human region of ID: 4 to be compared overlaps with an obstacle on the shelf. Therefore, the control unit 201 may evaluate the similarity between the upper body size human regions generated in FIG. 12.

Similarly, the similarity can be evaluated for the person region: b of the query to obtain the similarity shown in FIG. 13(b). The similarity can be evaluated for the person region: c of the query to obtain the similarity shown in FIG. 13(c). The similarity can be evaluated for the person region: d of the query to obtain the similarity shown in FIG. 13(d).

Then, by performing weighted matching using the similarity between the obtained query human region and the comparison human region, it is possible to identify the human region of the same person as the human region detected in a certain frame image i from among the human regions detected in another frame image i+1.

Figure 14 is a diagram illustrating human regions of the same person linked using weighted matching. As shown in Figure 14, the correct ID can be identified from the human regions of the same person linked using weighted matching. In this way, the size of the human regions is corrected and the similarity is evaluated so that the query human region and the comparison human region include the same area of the person's body, thereby improving the accuracy of identifying the same person.

Furthermore, when evaluating the similarity only based on the upper body, the amount of information is reduced by the amount of information on the lower body. In this case, for example, if the clothes on the upper body happen to be similar, the accuracy of identifying the same person may decrease. Also, when a person's body is hidden by an obstacle such as a cart that does not completely block the person's figure but blocks the person's body so that part of the object behind it is visible, the accuracy of identifying the same person may be improved by also using information on the visible area. In the above embodiment, when a person's body is hidden by an obstacle such as a cart that blocks the person's body so that part of the object behind it is visible, the similarity is evaluated by integrating the similarity evaluated in the human area of the whole body size and the similarity evaluated in the upper body size. This improves the accuracy of identifying the same person.

As described above, according to the embodiment, person identification can be performed with high accuracy even when at least a portion of a person in a video is hidden behind an obstacle.

FIG. 15 is a diagram illustrating an example of an operational flow of a person's motion detection process according to an embodiment. For example, the control unit 201 may start the process of the operational flow in FIG. 15 when an instruction to execute the person's motion detection process is input.

In step 1501 (hereinafter, step is abbreviated as "S", e.g., S1501), the control unit 201 executes, for example, human detection. For example, the control unit 201 may execute human detection on the image of each frame of the shooting data captured by the image capture device 102 to identify a human area from the image. Note that human detection may be executed using, for example, techniques such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), and R-CNN (Region Convolutional Neural Network).

In S1502, the control unit 201 performs skeleton detection on the human region detected from the image of each frame. For example, the control unit 201 may detect the skeleton of a person appearing in the human region using a method such as OpenPose, Mask R-CNN, DeepPose, or PoseNet.

In S1503, the control unit 201 corrects the detected human region. For example, as illustrated in FIG. 6, the control unit 201 may correct the human region to fit the upper body of the person, or may correct the human region to fit the entire body of the person, based on information about the person's skeleton detected from the human region.

In S1504, the control unit 201 identifies the position of the obstacle. For example, for an obstacle that is fixed at a predetermined position such as a shelf, the control unit 201 may obtain the position of the obstacle from information registered in the obstacle information 800.

For example, for a moving obstacle such as a cart, the control unit 201 may obtain the position of the obstacle using a technology such as object detection. For example, technology such as SSD, YOLO, and R-CNN may be used for object detection. For example, the control unit 201 may obtain the area in which the obstacle appears from each frame image of the video data using a trained model that has been machine-learned to detect a detection target obstacle such as a cart.

In S1505, the control unit 201 determines whether or not there is an obstacle to the person area. For example, if the person area detected in S1501 is in a position where it is hidden by an obstacle in front of it, the control unit 201 may determine that the person in the person area is hidden by an obstacle. On the other hand, if the person area detected in S1501 is not in a position where it is hidden by an obstacle in front of it, the control unit 201 may determine that the person in the person area is not hidden by an obstacle.

In S1506, the control unit 201 evaluates the similarity between the human region of a certain frame image and the human region of another frame image depending on the presence or absence of the identified obstacle. For example, the control unit 201 may evaluate the similarity between human regions that are not hidden by obstacles using the whole-body size human region as shown in the evaluation target information 700. In addition, when at least one of the two human regions to be evaluated for similarity is hidden by an obstacle such as a shelf that completely hides a person behind the obstacle, the control unit 201 may evaluate the similarity using the upper-body size human region. In addition, for example, at least one of the two human regions to be evaluated for similarity is hidden by an obstacle such as a cart that shows part of a person behind the obstacle. In this case, the control unit 201 may evaluate the similarity in both the whole-body size human region and the upper-body size human region, and may integrate the evaluation values by averaging them, etc., and use them as the similarity.

In S1507, the control unit 201 links the human regions of the same person using weighted matching based on the similarity, thereby performing person tracking, and this operation flow ends.

As described above, according to the embodiment, person identification can be performed with high accuracy even when at least a portion of a person in a video is hidden behind an obstacle.

For example, the control unit 201 evaluates the similarity by correcting the size of the human region so that the query human region and the comparison human region contain the same areas of the person's body, thereby improving the accuracy of identifying the same person.

In addition, when a person's body is hidden by an obstacle such as a cart that obscures part of an object behind the person, the control unit 201 acquires a similarity by integrating the similarity evaluation results of the human regions of the whole body and the upper body. This can improve the accuracy of identifying the same person.

In addition, in the above-described embodiment, the human area used to evaluate the similarity is changed depending on whether at least one of the two human areas to be evaluated for similarity is hidden by an obstacle. Therefore, it is possible to evaluate the similarity with high accuracy depending on how the person appears, and the accuracy of identifying the same person can be improved.

In addition, in the above-described embodiment, the size of the human region is corrected to match the size of the person's upper body and entire body based on skeletal information. Therefore, the size of the human region can be set with high accuracy to match the size of the person's body.

Although the above describes an embodiment, the embodiment is not limited to this. For example, the above-described operation flow is an example, and the embodiment is not limited to this. If possible, the operation flow may be executed by changing the order of the processes, may include additional processes, or may omit some processes.

For example, in one embodiment, when identifying the same person from video data in an environment where the lower half of a person's body is frequently hidden, the control unit 201 may evaluate the similarity using only the upper half of the person's body to identify the same person. In this case, in the process of S1503, the control unit 201 may correct the human area to match the upper half of the body based on the skeletal information, and the processes of S1504 and S1505 may be omitted. In another embodiment, after the operation flow of FIG. 15, further processing such as a behavioral analysis of the person may be performed on the track that tracks the movements of the person determined to be the same person.

In addition, in the above embodiment, an example is described in which similarity is used to determine whether two human regions represent the same person, but the embodiment is not limited to this. For example, in another embodiment, the control unit 201 may determine whether the human regions represent the same person based on a distance measure.

In addition, in the above embodiment, the upper body is used as an example of the half of a person's body for which the size of the human region is adjusted, but the embodiment is not limited to this. In another embodiment, the control unit 201 may correct the size of the human region to match another specified body region, such as the lower body, and use it for evaluating the similarity.

Also, for example, when the whole body of a person is visible in two human regions to be compared in person identification, the similarity determination result is estimated to be more reliable than when a part of the person's body is not visible due to an obstacle. Similarly, when a person is hidden by an obstacle that completely hides the person behind the obstacle, it is estimated that the amount of information is greater and the reliability of the similarity is higher when the person is hidden by an obstacle that blocks a part of an object behind the obstacle so that it can be seen, than when the person is hidden by an obstacle that completely hides the person behind the obstacle. Therefore, when the control unit 201 matches the human regions of the same person between frame images by weighted matching or the like in S1507, the control unit 201 may match the human regions so that the similarity evaluated using the whole body is given priority over other similarities, as shown below. Similarly, when the control unit 201 matches the human regions of the same person between frame images by weighted matching or the like in S1507, the control unit 201 may match the human regions so that the similarity evaluated using the human region of the whole body + upper body is given priority over the similarity evaluated using only the upper body.
Full body: High priority Full body + upper body: Medium priority Upper body only: Low priority

In the above embodiment, for example, in the process of S1501, the control unit 201 operates as the detection unit 211. Also, for example, in the process of S1502, the control unit 201 operates as the acquisition unit 212. In the process of S1503, the control unit 201 operates as the generation unit 213. In the processes from S1504 to S1507, the control unit 201 operates as the identification unit 214.

FIG. 16 is a diagram illustrating an example of the hardware configuration of a computer 1600 for realizing the information processing device 101 according to the embodiment. The hardware configuration for realizing the information processing device 101 in FIG. 16 includes, for example, a processor 1601, a memory 1602, a storage device 1603, a reading device 1604, a communication interface 1606, and an input/output interface 1607. The processor 1601, the memory 1602, the storage device 1603, the reading device 1604, the communication interface 1606, and the input/output interface 1607 are connected to each other, for example, via a bus 1608.

The processor 1601 may be, for example, a single processor, or a multi-processor or multi-core. The processor 1601 provides some or all of the functions of the control unit 201 described above by using the memory 1602 to execute a program that describes, for example, the steps of the above-mentioned operation flow. For example, the processor 1601 of the information processing device 101 operates as the detection unit 211, the acquisition unit 212, the generation unit 213, and the identification unit 214 by reading and executing a program stored in the storage device 1603.

Memory 1602 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. Storage device 1603 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device. Note that RAM is an abbreviation for Random Access Memory. Also, ROM is an abbreviation for Read Only Memory.

The reader 1604 accesses the removable storage medium 1605 according to instructions from the processor 1601. The removable storage medium 1605 is realized by, for example, a semiconductor device, a medium where information is input and output by magnetic action, or a medium where information is input and output by optical action. The semiconductor device is, for example, a Universal Serial Bus (USB) memory. The medium where information is input and output by magnetic action is, for example, a magnetic disk. The medium where information is input and output by optical action is, for example, a CD-ROM, DVD, Blu-ray Disc, etc. (Blu-ray is a registered trademark). CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk.

The storage unit 202 includes, for example, a memory 1602, a storage device 1603, and a removable storage medium 1605. For example, the storage device 1603 of the information processing device 101 stores, for example, video data captured by the imaging device 102 and obstacle information 800.

The communication interface 1606 communicates with other devices, such as the image capture device 102, according to instructions from the processor 1601. The communication interface 1606 is an example of the communication unit 203 described above.

The input/output interface 1607 may be, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, a mouse, or a touch panel that accepts instructions from a user. The output device is, for example, a display device such as a display, and an audio device such as a speaker.

Each program according to the embodiment is provided to the information processing device 101 in the following form, for example.
(1) It is pre-installed in the storage device 1603.
(2) Provided by a removable storage medium 1605.
(3) Provided from a server such as a program server.

The hardware configuration of the computer 1600 for realizing the information processing device 101 described with reference to FIG. 16 is an example, and the embodiment is not limited to this. For example, part of the above-mentioned configuration may be deleted, or a new configuration may be added. In another embodiment, for example, part or all of the functions of the above-mentioned control unit 201 may be implemented as hardware using an FPGA, SoC, ASIC, PLD, or the like. Note that FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip. ASIC is an abbreviation for Application Specific Integrated Circuit. PLD is an abbreviation for Programmable Logic Device.

In the above, several embodiments are described. However, the embodiments are not limited to the above embodiments, and should be understood to include various modified and alternative forms of the above-mentioned embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments.

100 Motion detection system 101 Information processing device 102 Shooting device 201 Control unit 202 Storage unit 203 Communication unit 211 Detection unit 212 Acquisition unit 213 Generation unit 214 Identification unit 401 Person area 402 Shelf 403 Cart 700 Evaluation target information 800 Obstacle information 1600 Computer 1601 Processor 1602 Memory 1603 Storage device 1604 Reading device 1605 Removable storage medium 1606 Communication interface 1607 Input/output interface 1608 Bus

Claims (7)

A process of detecting a human region from a plurality of frame images included in the video data;
A process of performing skeletal detection on a person appearing in the human region detected from the plurality of frame images to obtain skeletal information of the person;
a process of correcting the human region to match the size of the whole body of the person based on the skeletal information of the person, thereby generating a human region of full body size, and a process of correcting the human region to match the size of half the body of the person based on the skeletal information of the person, thereby generating a human region of half body size;
when a similarity between at least one of a full-body size human region and a half-body size human region in a first human region detected from a first frame image among the plurality of frame images and a second human region detected from a second frame image satisfies a predetermined condition, the first human region and the second human region are determined to be the same person, and the first and second human regions are identified as human regions of the same person ;
The identification process includes an identification program that, when a person is hidden by an obstacle that blocks part of an object behind at least one of a first human region detected from the first frame image and a second human region detected from the second frame image, evaluates a similarity between the first human region and the second human region based on a similarity between the half-body-sized human region corresponding to the first human region and the half-body-sized human region corresponding to the second human region, and a similarity between the full-body-sized human region corresponding to the first human region and the full-body-sized human region corresponding to the second human region . The identification program according to claim 1, wherein the process of identifying includes, when a person is not hidden by an obstacle in either the first human region detected from the first frame image or the second human region detected from the second frame image, evaluating the similarity between the full-body-sized human region corresponding to the first human region and the full-body-sized human region corresponding to the second human region. The identification program according to claim 1, wherein the process of identifying evaluates the similarity between the first human region and the second human region based on the similarity between the half-body size human region corresponding to the first human region and the half-body size human region corresponding to the second human region when a person is hidden by an obstacle that completely blocks an object behind at least one of the first human region detected from the first frame image and the second human region detected from the second frame image. The identification program according to any one of claims 1 to 3, wherein the generating process sets the height length of the half-body size human area to be the length from the head to the waist of the person identified based on the skeletal information. 5. The identification program according to claim 1, wherein the generating process sets the height direction length of the full-body size human area to be twice the length from the head to the waist of the person identified based on the skeletal information. 1. A computer implemented method for identifying a
The computer,
A step of detecting a human region from a plurality of frame images included in video data;
A step of performing skeletal detection on a person appearing in the human region detected from the plurality of frame images to obtain skeletal information of the person;
correcting the human region to match the size of the whole body of the person based on the skeletal information of the person, thereby generating a human region of full body size; and correcting the human region to match the size of half the body of the person based on the skeletal information of the person , thereby generating a human region of half body size;
and when a similarity between at least one of a full-body size human region and a half-body size human region in a first human region detected from a first frame image among the plurality of frame images and a second human region detected from a second frame image satisfies a predetermined condition, determining that a person appearing in the first human region and a person appearing in the second human region are the same person, and specifying the first and second human regions as human regions of the same person ,
an identification method in which, in the identification step, when a person is hidden by an obstacle that blocks part of an object behind at least one of a first human region detected from the first frame image and a second human region detected from the second frame image, the similarity between the first human region and the second human region is evaluated based on the similarity between the half-body-sized human region corresponding to the first human region and the half-body-sized human region corresponding to the second human region, and the similarity between the full-body-sized human region corresponding to the first human region and the full-body-sized human region corresponding to the second human region . A detection unit that detects a human region from a plurality of frame images included in the video data;
an acquisition unit that executes skeletal detection on a person appearing in the human region detected from the plurality of frame images to acquire skeletal information of the person;
a generation unit that corrects the human region to match the size of the whole body of the person based on the skeletal information of the person, thereby generating a human region of full body size, and corrects the human region to match the size of half the body of the person based on the skeletal information of the person, thereby generating a human region of half body size;
an identification unit that, when a similarity between at least one of a full-body size human region and a half-body size human region in a first human region detected from a first frame image among the plurality of frame images and a second human region detected from a second frame image satisfies a predetermined condition, determines that a person appearing in the first human region and a person appearing in the second human region are the same person, and identifies the first and second human regions as human regions of the same person ,
The information processing device, when a person is hidden by an obstacle that blocks part of an object behind at least one of a first human region detected from the first frame image and a second human region detected from the second frame image, the identification unit evaluates the similarity between the first human region and the second human region based on the similarity between the half-body-sized human region corresponding to the first human region and the half-body-sized human region corresponding to the second human region, and the similarity between the full-body-sized human region corresponding to the first human region and the full-body-sized human region corresponding to the second human region .