JP6922768B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device that identifies a person in a captured image.

As a technique for identifying a person in an captured image, there is a technique for acquiring a person's facial features from the captured image and identifying the person from the acquired facial features.

In the technique disclosed in Patent Document 1, the feature amount of one face of one image and the feature amount of one face of another image are close to each other, and the feature amount of one face of one image and another It is determined whether or not there are a plurality of relationships in which the features of all other faces other than one face in the image are not close to each other. Then, when a plurality of such relationships exist, it is determined that one face of one image and one face of another image are the faces of the same person.

Japanese Unexamined Patent Publication No. 2015-225567

When the identification result is used for locking / unlocking a door or the like, the person actively turns his / her face toward the camera, so that an accurate facial feature amount of the person can be obtained from the captured image. However, when the identification result is used for other purposes, the person may not turn his / her face toward the camera, and the accurate facial feature amount of the person may not be obtained from the captured image. .. Therefore, in the conventional technique for identifying a person from the amount of facial features, the person may not be identified with high accuracy.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an information processing device capable of identifying a person appearing in an captured image even when the face is not imaged.

In order to achieve the above object, in the present invention, a method of identifying a person based on at least one of the posture, gesture, silhouette, flow line, and place of stay of the person in the captured image. Is adopted.

Specifically, the first aspect of the present invention is an image acquisition means for acquiring an captured image, and a posture, gesture, silhouette, movement line, and staying place of a person appearing in the image from the image. Provided is an information processing apparatus characterized by having an information acquisition means for acquiring characteristic information indicating at least one of the characteristics of the person, and an identification means for identifying the person based on the characteristic information. ..

According to this configuration, it is possible to identify the person in the captured image even when the face is not captured. Specifically, the posture, gesture, silhouette, flow line, place of stay, etc. of the person in the captured image can be acquired even if the face of the person is not captured. And since those characteristics are unique to a person, the person can be identified from those characteristics.

The information acquisition means may acquire skeleton information indicating the skeleton of the person from the image and acquire the feature information based on the skeleton information. For each of the plurality of persons, the storage means for storing the reference information indicating the characteristics of the person is further provided, and the identification means compares the characteristic information with each reference information and shows the person in the image. May be identified. It may further have an imaging means for capturing the image.

When two or more people are shown in the image, the identification means identifies a part of the two or more people based on the feature information corresponding to the person, and the above 2 It is preferable to identify the remaining person among the person or more based on the feature information corresponding to the person and the identification result of the part of the person. The storage means stores two or more reference information corresponding to each of two or more combinations of the person and the other person for each of the plurality of persons, and the image shows the two or more persons. In this case, the identification means identifies a part of the two or more persons by comparing the feature information corresponding to the person with the reference information, and identifies the two or more persons. The remaining person of the person may be identified by comparing the feature information corresponding to the person with the reference information corresponding to the combination of the identified part of the person.

When one person is with another person, the characteristics of one person (posture, gesture, flow line, place of stay, etc.) may change depending on the other person. For example, the attitude or behavior of one person between when another person includes a specific person (father, mother, brother, brother, sister, sister, boss, subordinate, etc.) and when it does not. May be different. Therefore, one person can be identified with higher accuracy by considering the identification result of another person.

The present invention can be regarded as an information processing system having at least a part of the above-mentioned configuration or function. Further, the present invention provides a method for controlling an information processing method or an information processing system including at least a part of the above processing, a program for causing a computer to execute these methods, or a program such as such for non-temporary purposes. It can also be regarded as a recorded computer-readable recording medium. Each of the above configurations and processes can be combined with each other to construct the present invention as long as there is no technical contradiction.

According to the present invention, it is possible to identify a person appearing in an image captured even when the face is not imaged.

FIG. 1 is a block diagram showing a configuration example of an information processing apparatus to which the present invention is applied. FIG. 2 is a block diagram showing a configuration example of the information processing system according to the first embodiment. FIG. 3 is a flowchart showing an example of a processing flow of the information processing apparatus according to the first embodiment. FIG. 4 is a diagram showing an example of an image captured by the surveillance camera according to the first embodiment. FIG. 5 is a diagram showing an example of a difference in posture according to the first embodiment. FIG. 6 is a diagram showing an example of reference information according to the first embodiment. FIG. 7 is a block diagram showing a configuration example of the communication robot according to the second embodiment. FIG. 8 is a flowchart showing an example of a processing flow of the information processing apparatus according to the third embodiment. FIG. 9A is a diagram showing an example of reference information according to the third embodiment, and FIG. 9B is a diagram showing an example of the silhouette shown by the mask image according to the third embodiment. Is. FIG. 10 is a diagram showing an example of a silhouette image and a mask image according to the third embodiment. FIG. 11 is a flowchart showing an example of a processing flow of the information processing apparatus according to the fourth embodiment. FIG. 12 is a diagram showing an example of a stay map (feature information) according to the fourth embodiment. FIG. 13 (A) is a diagram showing an example of reference information according to the fourth embodiment, and FIG. 13 (B) is a diagram showing an example of a stay map (reference information) according to the fourth embodiment. be. FIG. 14 is a diagram showing an example of a flow line according to a modified example of the present invention. FIG. 15 is a flowchart showing an example of a processing flow of the information processing apparatus according to the fifth embodiment. FIG. 16 is a flowchart showing an example of a processing flow of the information processing apparatus according to the sixth embodiment. FIG. 17 is a diagram showing an example of reference information according to the sixth embodiment.

<Application example>
An application example of the present invention will be described. In the conventional technique for identifying a person appearing in an captured image, the facial feature amount of the person is acquired from the captured image, and the person is identified from the acquired facial feature amount. When the identification result is used for locking / unlocking a door or the like, the person actively turns his / her face toward the camera, so that an accurate facial feature amount of the person can be obtained from the captured image. However, when the identification result is used for other purposes, the person may not turn his / her face toward the camera, and the accurate facial feature amount of the person may not be obtained from the captured image. .. Therefore, in the above-mentioned conventional technique, a person may not be identified with high accuracy.

FIG. 1 is a block diagram showing a configuration example of an information processing apparatus 100 to which the present invention is applied. The information processing device 100 includes an image input unit 101, a control unit 102, a storage unit 103, and an output unit 104. The control unit 102 has an information acquisition unit 111 and an identification unit 112.

The image input unit 101 acquires the captured image (image data). For example, the image input unit 101 is an input terminal into which image data is input. The image input unit 101 is an example of the image acquisition means of the present invention.

The control unit 102 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and performs various information processing and control of each component.

The information acquisition unit 111 is a feature that indicates at least one of the posture, gesture, silhouette, flow line, and staying place of the person in the image from the image acquired by the image input unit 101 as the characteristics of the person. Get information. The information acquisition unit 111 is an example of the information acquisition means of the present invention.

The identification unit 112 identifies the person in the image acquired by the image input unit 101 based on the feature information acquired by the information acquisition unit 111. The identification unit 112 is an example of the identification means of the present invention.

The storage unit 103 stores programs executed by the control unit 102, various data used by the control unit 102, and the like. For example, the storage unit 103 is an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 103 is an example of the storage means of the present invention.

The output unit 104 outputs the identification result of the identification unit 112 to an external device, a component (not shown) of the information processing device 100, and the like. The output unit 104 is, for example, an output terminal that outputs identification result data.

According to the above configuration of the information processing device 100, it is possible to identify a person in the captured image even when the face is not captured. Specifically, the posture, gesture, silhouette, flow line, place of stay, etc. of the person in the captured image can be acquired even if the face of the person is not captured. And since those characteristics are unique to a person, the person can be identified from those characteristics.

<First Embodiment>
The first embodiment of the present invention will be described. Surveillance cameras may be used to manage employees, residents (residents), etc., and to improve security in workplaces, communities, homes, etc. People rarely turn their faces to surveillance cameras. In the first embodiment, an example of identifying a person in an image captured by a surveillance camera will be described.

FIG. 2 is a block diagram showing a configuration example of the information processing system according to the first embodiment. In the information processing system of FIG. 2, the surveillance camera 200 and the management device 300 are connected to the information processing device 100 of FIG. At least one of the surveillance camera 200 and the management device 300 may be a part of the information processing device 100.

The surveillance camera 200 is an image pickup device that takes an image and outputs the captured image to the information processing device 100. The imaging range of the surveillance camera 200 may be fixed or variable. The image captured by the surveillance camera 200 is input to the image input unit 101 of the information processing device 100. When the image captured by the surveillance camera 200 is input to the image input unit 101, the person in the image is identified by the processing of the control unit 102 (information acquisition unit 111 and identification unit 112) of the information processing device 100. Will be done. Then, the output unit 104 of the information processing device 100 outputs the identification result of the control unit 102 (identification unit 112) to the management device 300. The management device 300 performs various processes using the identification result output from the information processing device 100. For example, the management device 300 generates statistical data of a plurality of identification results in a predetermined period, and notifies the administrator (user of the management device 300) of the statistical data and the identification results. The notification to the administrator is realized by, for example, a screen display using a display unit such as a liquid crystal monitor, an audio output using a speaker, or the like. In the first embodiment, the information acquisition unit 111 acquires characteristic information indicating a posture and a gesture.

FIG. 3 is a flowchart showing an example of a processing flow of the information processing apparatus 100 according to the first embodiment.

First, the image input unit 101 acquires an image captured by the surveillance camera 200 from the surveillance camera 200 (step S301). FIG. 4 shows an example of an image captured by the surveillance camera 200. In FIG. 4, the person 401 is shown in the image 400 captured by the surveillance camera 200.

Next, the information acquisition unit 111 acquires skeleton information indicating the skeleton of the person in the image from the image acquired in step S301 (step S302). The skeleton information is acquired using, for example, OpenPose. The skeletal information is information indicating the human body and information indicating parts of the human body (head, neck, shoulders, elbows, hands, hips, knees, ankles, eyes, ears, fingertips, etc.). Therefore, the acquisition of skeletal information can be said to be "human body detection" or "site detection". In FIG. 4, the skeleton (skeleton information) 402 of the person 401 is detected from the image 400.

Then, the information acquisition unit 111 detects the posture and gesture of the person in the image acquired in step S301 based on the skeleton information acquired in step S302 (steps S303 and S304). The combination of the detection result (posture) of step S303 and the detection result (gesture) of step S304 is the characteristic information of the person shown in the image acquired in step S301. The process of step S303 (posture detection) and the process of step S304 (gesture detection) may be performed in parallel or in order. The order of posture detection and gesture detection is not particularly limited.

In step S303, as the posture detection result, for example, information indicating upright, stoop, O-leg, X-leg, and the like can be obtained. As shown in FIG. 5, the shape of the skeleton is different between the case of stoop and the case of normal. Thus, the shape of the skeleton depends on the posture. Therefore, the posture can be detected from the shape of the skeleton based on the skeleton information.

In step S304, as a result of detecting the gesture, information indicating, for example, walking, bending and stretching, lying down, arms folded, and the like can be obtained. The positional relationship between the upper arm and the forearm differs between the case where the arms are folded and the case where the arms are not folded. In this way, the positional relationship of each part depends on the gesture. Therefore, the gesture can be detected from the positional relationship of each part based on the skeleton information. Gestures accompanied by movements such as walking and bending and stretching may be detected by using a plurality of skeletal information corresponding to a plurality of images captured at different times. For walking, information indicating the ratio of stride length to shoulder width may be obtained.

The posture and gesture may be detected by another method that does not use skeletal information. For example, posture detection or gesture detection using pattern matching or the like may be performed.

Next, the identification unit 112 identifies the person in the image acquired in step S301 based on the feature information obtained by the processes of steps S303 and S304 (step S305). For example, the storage unit 103 stores in advance reference information indicating the characteristics of each of the plurality of persons. Then, the identification unit 112 compares the feature information with each reference information to identify the person in the image. The reference information may be manually registered in the information processing apparatus 100, or may be automatically registered using the image captured by the surveillance camera 200.

FIG. 6 shows an example of reference information according to the first embodiment. In FIG. 6, reference information 601 to 603 is registered in advance. Reference information 601 is characterized by "posture (back): normal", "posture (leg): normal", and "step ratio (ratio of stride when shoulder width is 1): 1. .5 ”is shown. Reference information 602 shows "posture (back): stoop", "posture (leg): O-leg", and "step ratio: 1.3" as the characteristics of "Mr. B". The reference information 603 indicates "posture (back): stoop", "posture (leg): O-leg", and "step ratio: 1.0" as the characteristics of "Mr. C".

Here, the reference information 601 to 603 is registered in advance, and by the processing of steps S303 and S304, "posture (back): normal", "posture (leg): normal", and "step ratio: 1. Consider the case where the feature information indicating "4" is acquired. In this case, the information most similar to the feature information among the reference information 601 to 603 is the reference information 601. Therefore, the identification unit 112 determines that the person in the image acquired in step S301 is "Mr. A".

Next, the output unit 104 outputs the identification result of step S305 to the management device 300 (step S306).

As described above, according to the first embodiment, even when the face is not imaged by the surveillance camera, it is based on the feature information indicating the posture and gesture of the person in the image captured by the surveillance camera. , The person can be identified.

<Second embodiment>
A second embodiment of the present invention will be described. Communication robots that provide meaningful information to users (employees, customers, residents, etc.) and have conversations with users may identify users for proper communication with them. preferable. For example, the communication robot acquires a face feature amount of a person from an image captured by its own imaging unit, and identifies the person from the acquired face feature amount.

However, the person does not always turn his face to the communication robot (imaging unit) before identification. In particular, when the communication robot voluntarily communicates with the user, it is unlikely that the user turns his face to the communication robot before identification. In addition, due to the influence of heredity, the faces of people in the family may be similar. Therefore, even if a certain person in the family is facing the communication robot and an accurate facial feature amount of the person is obtained, the communication robot may not be able to identify the person from the facial feature amount. ..

In the second embodiment, an example in which the present invention is applied to a communication robot will be described.

FIG. 7 is a block diagram showing a configuration example of the robot (communication robot) 700 according to the second embodiment. The robot 700 includes an image pickup unit 701, an image input unit 101, a control unit 102, a storage unit 103, an output unit 104, and a communication unit 702. The control unit 102 has an information acquisition unit 111 and an identification unit 112.

The image pickup unit 701 takes an image and outputs the captured image to the image input unit 101. For example, the image pickup unit 701 is an image pickup sensor such as a CCD or CMOS sensor. The image input unit 101, the control unit 102, the storage unit 103, and the output unit 104 are as described in the first embodiment. However, the image input unit 101 acquires the image captured by the image pickup unit 701 from the image pickup unit 701, and the output unit 104 outputs the identification result to the communication unit 702. Based on the identification result output from the output unit 104, the communication unit 702 performs processing for communication with the person in the image captured by the image pickup unit 701. For example, the communication unit 702 includes a display unit such as a liquid crystal monitor, a speaker, and the like. Then, the communication unit 702 provides a screen display using a display unit such as a liquid crystal monitor and a speaker based on the identification result for providing information to the person of the identification result, talking with the person of the identification result, and the like. Perform the audio output, etc. used.

The processing flow example of the robot 700 is the same as that of the first embodiment (FIG. 3). However, in step S301, the image input unit 101 acquires the image captured by the image pickup unit 701 from the image pickup unit 701. In step S306, the output unit 104 outputs the identification result of step S305 to the communication unit 702.

As described above, according to the second embodiment, even when the face is not imaged in the communication robot, the communication robot is said to be based on the characteristic information indicating the posture and gesture of the person in the captured image. Can identify a person. In addition, there is a high possibility that postures and behaviors will differ even among family members due to differences in actual age and mental age. Therefore, by considering the posture and gesture, each person in the family can be identified with high accuracy. Similarly, there is a high possibility that silhouettes, flow lines, places of stay, etc. will differ even among family members. An example of considering the silhouette, flow line, place of stay, etc. will be described later.

<Third embodiment>
A third embodiment of the present invention will be described. In the first and second embodiments, an example in which feature information indicating a posture and a gesture is acquired has been described. In the third embodiment, an example of acquiring feature information further indicating the silhouette of a person in the captured image will be described.

The configuration of the information processing device according to the third embodiment is the configuration of the information processing device 100 according to the first embodiment (FIGS. 1 and 2) or the configuration of the robot 700 according to the second embodiment (FIG. 7). ).

FIG. 8 is a flowchart showing an example of a processing flow of the information processing apparatus according to the third embodiment.

First, similarly to the first embodiment and the second embodiment, the image input unit 101 acquires the captured image (step S301), and the information acquisition unit 111 acquires the skeleton information (step S302). ..

Next, the information acquisition unit 111 determines the posture, gesture, and silhouette of the person in the image acquired in step S301 based on the image acquired in step S301 and the skeleton information acquired in step S302. Detect (steps S303, S304, S800). The combination of the detection result (posture) of step S303, the detection result (gesture) of step S304, and the detection result (silhouette) of step S800 is the characteristic information of the person shown in the image acquired in step S301. The process of step S303 (posture detection), the process of step S304 (gesture detection), and the process of step S800 (silhouette detection) may be performed in parallel or in order. The order of posture detection, gesture detection, and silhouette detection is not particularly limited.

The process of step S303 (posture detection) and the process of step S304 (gesture detection) are as described in the first embodiment. In step S800, for example, Mask
The silhouette of a person is detected from the image using R-CNN or the like.

Next, as in the first embodiment and the second embodiment, the identification unit 112 provides the obtained feature information (specifically, the feature information obtained by the processing of steps S303, S304, S800). Based on this, the person in the image acquired in step S301 is identified (step S305).

FIG. 9A shows an example of reference information according to the third embodiment. In FIG. 9A, reference information 901 to 903 is registered in advance. Reference information 901 shows "posture (back): normal", "step ratio: 1.5", and "mask image: I1" as features of "Mr. A". Reference information 902 shows "posture (back): stoop", "step ratio: 1.3", and "mask image: I2" as features of "Mr. B". Then, the reference information 903 shows "posture (back): stoop", "step ratio: 1.3", and "mask image: I3" as the characteristics of "Mr. C".

The mask images I1 to I3 are images showing silhouettes. FIG. 9B shows an example of silhouettes corresponding to mask images I1 to I3. "Mr. A", "Mr. B", and "Mr. C" have different body shapes. Therefore, as shown in FIG. 9B, the silhouette is different between the mask images I1 to I3.

Here, the reference information 901 to 903 is registered in advance, and steps S303 and SS
Consider the case where the feature information indicating "posture (back): stoop", "step ratio: 1.3", and "silhouette image: Ip" is acquired by the processing of 304 and S800. The silhouette image is an image showing the silhouette detected in step S800. In this case, "posture (back): stoop" and "step ratio: 1.3" match those of reference information 902 and 903. Therefore, from the feature information "posture (back): stoop" and "step ratio: 1.3", the person in the image acquired in step S301 is "Mr. B" or "Mr. C". I can't tell if it's there.

In the third embodiment, for example, the identification unit 112 compares the silhouette image Ip of the feature information with the mask image I2 of the reference information 902 and the mask image I3 of the reference information 903. As a result, the identification unit 112 can determine that the similarity between the detected silhouette and the silhouette of "Mr. B" is higher than the similarity between the silhouette detected in step S800 and the silhouette of "Mr. C". As a result, the identification unit 112 can determine that the person in the image acquired in step S301 is "Mr. B". As described above, in the third embodiment, by further considering the silhouette, a person who cannot be identified by considering only the posture and the gesture can be identified.

The size of the silhouette of a person depends on the distance between the camera (imaging unit) and the person. Therefore, the detected silhouette is size-normalized to generate a silhouette image, or a mask image corresponding to the size-normalized silhouette is prepared in advance. The silhouette is normalized so that, for example, the size from the tip of the head to the tip of the foot becomes a predetermined value.

An example of how to acquire the similarity of silhouettes will be described in more detail. FIG. 10 shows an example of a silhouette image and a mask image. Although the mask image corresponding to a part of the arm is shown in FIG. 10, the mask image may be an image corresponding to the whole person or an image corresponding to a part of the person. good. As shown in FIG. 10, the mask image is composed of a plurality of regions (plurality of silhouette determination regions). The attribute "1" corresponding to the silhouette is assigned to the silhouette determination area corresponding to the silhouette shown by the mask image. Then, the attribute "0" corresponding to the background is assigned to the silhouette determination area corresponding to the background indicated by the mask image.

The identification unit 112 determines whether or not the attribute (silhouette / background) of the silhouette image matches the attribute (1/0; silhouette / background) of the mask image for each silhouette determination area. Then, the identification unit 112 calculates the ratio of the total number of matching areas (silhouette determination areas determined to match) to the total number of silhouette determination areas, the total number of matching areas, and the like as the similarity of silhouettes.

The size of the silhouette determination area is not particularly limited. The silhouette determination area may be an area of one pixel or an area of a plurality of pixels. When the silhouette determination area is an area for a plurality of pixels, the silhouette determination area may include both the silhouette and the background indicated by the silhouette image. In that case, the attributes of the silhouette image in the silhouette determination area may be determined based on the size of the silhouette (silhouette indicated by the silhouette image) in the silhouette determination area. For example, the attribute "silhouette" of the silhouette image is acquired for the silhouette determination area where the size of the silhouette indicated by the silhouette image is equal to or larger than the threshold value, and the attribute "background" of the silhouette image is acquired for the silhouette determination area which is not. May be obtained.

Next, as in the first embodiment and the second embodiment, the output unit 104 outputs the identification result of step S305 (step S306).

As described above, according to the third embodiment, the person can be identified with higher accuracy than the first embodiment and the second embodiment by using the feature information further indicating the silhouette.

<Fourth Embodiment>
A fourth embodiment of the present invention will be described. In the fourth embodiment, an example of acquiring characteristic information indicating the posture, gesture, and staying place of the person in the captured image will be described.

The configuration of the information processing device according to the fourth embodiment is the configuration of the information processing device 100 according to the first embodiment (FIGS. 1 and 2) or the configuration of the robot 700 according to the second embodiment (FIG. 7). ).

FIG. 11 is a flowchart showing an example of a processing flow of the information processing apparatus according to the fourth embodiment.

First, similarly to the first embodiment and the second embodiment, the image input unit 101 acquires the captured image (step S301), and the information acquisition unit 111 acquires the skeleton information (step S302). ..

Next, the information acquisition unit 111 detects the posture, gesture, and staying place of the person in the image acquired in step S301 based on the skeleton information acquired in step S302 (steps S303 and S304). , S1100). The combination of the detection result (posture) of step S303, the detection result (gesture) of step S304, and the detection result (stay location) of step S1100 is the characteristic information of the person shown in the image acquired in step S301. .. The process of step S303 (posture detection), the process of step S304 (gesture detection), and the process of step S1100 (stay location detection) may be performed in parallel or in sequence. The order of posture detection, gesture detection, and stay location detection is not particularly limited.

The process of step S303 (posture detection) and the process of step S304 (gesture detection) are as described in the first embodiment. In step S1100, the information acquisition unit 111 detects the staying place of the person based on, for example, the position of the skeleton (position of the person; the position of the person) indicated by the skeleton information. Specifically, a plurality of stay determination areas constituting the imaging range are predetermined. The information acquisition unit 111 calculates the ratio of the staying time to the length of the predetermined period (stay rate) for each stay determination area based on the time change of the person's position in the predetermined period such as the past several minutes. As a result, as a result of detecting the place of stay, a stay map (heat map) showing the stay rate of each stay determination area is obtained. For example, the stay map Mp shown in FIG. 12 can be obtained.

Next, as in the first embodiment and the second embodiment, the identification unit 112 provides the obtained feature information (specifically, the feature information obtained by the processing of steps S303, S304, and S1100). Based on this, the person in the image acquired in step S301 is identified (step S305).

FIG. 13A shows an example of reference information according to the fourth embodiment. In FIG. 13A, reference information 1301 to 1303 is registered in advance. Reference information 1301 indicates "posture (back): normal", "step ratio: 1.5", and "stay map: M1" as the characteristics of the "father". Reference information 1302 indicates "posture (back): stoop", "step ratio: 1.3", and "stay map: M2" as the characteristics of the "mother". Then, the reference information 1303 indicates "posture (back): stoop", "step ratio: 1.3", and "stay map: M3" as the characteristics of the "sister".

The stay maps M1 to M3 of the reference information 1301 to 1303 show the stay rate in a predetermined period such as the past one month. FIG. 13B shows an example of stay maps M1 to M3. For example, my father is often near the aisle in the living room, my mother is often in the kitchen, my sister is often near the wall in the living room, and so on. do. Therefore, as shown in FIG. 13B, the distribution of the stay rate differs between the stay maps M1 to M3.

Here, the reference information 1301 to 1303 is registered in advance, and by the processing of steps S303, S304, and S1100, "posture (back): stoop", "step ratio: 1.3", and "stay map: Consider the case where the feature information indicating "Mp (FIG. 12)" is acquired. In this case, "posture (back): stoop" and "step ratio: 1.3" match those of reference information 1302 and 1303. Therefore, from the feature information "posture (back): stoop" and "step ratio: 1.3", whether the person in the image acquired in step S301 is the "mother" or the "sister". Cannot be determined.

In the fourth embodiment, for example, the identification unit 112 compares the stay map Mp of the feature information with the stay map M2 of the reference information 1302 (mother) and the stay map M3 of the reference information 1303 (sister). As a result, the identification unit 112 can determine that the degree of similarity between the stay map Mp and the "sister" stay map M3 is higher than the degree of similarity between the stay map Mp and the "mother" stay map M2. As a result, the identification unit 112 can determine that the person in the image acquired in step S301 is the "sister". As described above, in the fourth embodiment, by further considering the place of stay in addition to the posture and gesture, a person who cannot be identified by considering only the posture and gesture can be identified.

The processing related to the place of stay is not limited to the above processing. For example, in step S1100, the information acquisition unit 111 may detect the stay determination area having the highest stay rate as the stay place. In the reference information, the stay determination area having the highest stay rate, the stay determination area designated by the user, and the like may be indicated as the stay place. Then, in step S305, the identification unit 112 may determine a match / mismatch between the staying place detected in step S1100 and the staying place of the reference information.

Next, as in the first embodiment and the second embodiment, the output unit 104 outputs the identification result of step S305 (step S306).

As described above, according to the fourth embodiment, by using the characteristic information indicating the place of stay in addition to the posture and gesture, the person can be displayed with higher accuracy than the first embodiment and the second embodiment. Can be identified.

The flow line (movement path) of the person may be taken into consideration. The flow line is detected, for example, in the same manner as the stay map. FIG. 14 shows an example of the flow lines of the father, mother, and sister. Just as the place of stay (stay rate) depends on the person, the flow line also depends on the person. Therefore, as shown in FIG. 14, the flow lines differ between the father, mother, and sister.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described. In the fifth embodiment, an example of acquiring characteristic information indicating the posture, gesture, silhouette, and staying place of the person in the captured image will be described. That is, in the fifth embodiment, an example of the combination of the third embodiment and the fourth embodiment will be described.

The configuration of the information processing device according to the fifth embodiment is the configuration of the information processing device 100 according to the first embodiment (FIGS. 1 and 2) or the configuration of the robot 700 according to the second embodiment (FIG. 7). ).

FIG. 15 is a flowchart showing an example of a processing flow of the information processing apparatus according to the fifth embodiment.

First, similarly to the first embodiment and the second embodiment, the image input unit 101 acquires the captured image (step S301), and the information acquisition unit 111 acquires the skeleton information (step S302). ..

Next, the information acquisition unit 111 detects the posture, gesture, silhouette, and staying place of the person in the image acquired in step S301 based on the skeleton information acquired in step S302 (step S303). , S304, S800, S1100). The combination of the detection result (posture) of step S303, the detection result (gesture) of step S304, the detection result (silhouette) of step S800, and the detection result (stay place) of step S1100 is added to the image acquired in step S301. It is the characteristic information of the person in the picture. The process of step S303 (posture detection), the process of step S304 (gesture detection), the process of step S800 (silhouette detection), and the process of step S1100 (stay location detection) may be performed in parallel, or in order. May be done in. The order of posture detection, gesture detection, silhouette detection, and stay location detection is not particularly limited.

The process of step S303 (posture detection) and the process of step S304 (gesture detection) are as described in the first embodiment. The process (silhouette detection) of step S800 is as described in the third embodiment. The process of step S1100 (detection of the place of stay) is as described in the fourth embodiment.

Next, as in the first embodiment and the second embodiment, the identification unit 112 obtains the feature information (specifically, the feature information obtained by the processing of steps S303, S304, S800, S1100). ), The person in the image acquired in step S301 is identified (step S305).

As described in the third embodiment, the accuracy of identifying a person is improved by further considering the silhouette in addition to the posture and gesture. Then, as described in the fourth embodiment, the accuracy of identifying a person is improved by further considering the place of stay in addition to the posture and gesture. Therefore, by further considering both the silhouette and the place of stay in addition to the posture and gesture, the person can be identified with higher accuracy than in the third embodiment and the fourth embodiment. For example, even if the identification based on the posture, the gesture, the silhouette, and the place of stay cannot be succeeded, the identification based on the posture, the gesture, the silhouette, and the rest of the place of stay may be successful. Therefore, the probability of successful identification increases.

Next, as in the first embodiment and the second embodiment, the output unit 104 outputs the identification result of step S305 (step S306).

As described above, according to the fifth embodiment, the person is identified with higher accuracy than the first to fourth embodiments by using the characteristic information indicating the posture, the gesture, the silhouette, and the place of stay. can.

<Sixth Embodiment>
A sixth embodiment of the present invention will be described. In the sixth embodiment, an example in which two or more people are shown in the captured image will be described.

The configuration of the information processing device according to the sixth embodiment is the configuration of the information processing device 100 according to the first embodiment (FIGS. 1 and 2) or the configuration of the robot 700 according to the second embodiment (FIG. 7). ).

FIG. 16 is a flowchart showing an example of a processing flow of the information processing apparatus according to the sixth embodiment.

First, similarly to the first embodiment and the second embodiment, the image input unit 101 acquires the captured image (step S301), and the information acquisition unit 111 acquires the skeleton information (step S302). .. When two or more people are shown in the captured image, the skeleton information of each person is acquired.

Next, as in the first to fifth embodiments, the information acquisition unit 111 acquires feature information based on the skeleton information acquired in step S302 (step S1601). In step S1601, for example, processes such as steps S303, S304, S800, and S1100 of FIG. 15 are performed. When two or more people are shown in the captured image, the feature information of each person is acquired.

Then, as in the first to fifth embodiments, the identification unit 112 identifies the person in the image acquired in step S301 based on the feature information acquired in step S1601 (step S305). .. When two or more people are shown in the captured image, each person is identified. In step S305, the person corresponding to the feature information may not be identified because the plurality of reference information corresponding to the plurality of persons are similar to one feature information. Here, it is assumed that only a part of the two or more persons in the captured image is identified in step S305, and the remaining person among the two or more persons is not identified. .. The remaining person may be a person who failed to be identified in step S305, or may be a person who was not targeted for identification in step S305.

Next, the identification unit 112 determines the feature information corresponding to the remaining person (feature information acquired in step S1601) and the identification result of some of the persons (identification result in step S305). Identify the remaining person (step S1602). For example, the storage unit 103 stores in advance two or more reference information corresponding to each of two or more combinations of the person and the other person for each of the plurality of persons. Then, the identification unit 112 identifies the remaining person by comparing the feature information corresponding to the remaining person with the reference information corresponding to the combination with the part of the person.

When one person is with another person, the characteristics of one person (posture, gesture, flow line, place of stay, etc.) may change depending on the other person. For example, the attitude or behavior of one person between when another person includes a specific person (father, mother, brother, brother, sister, sister, boss, subordinate, etc.) and when it does not. May be different. Therefore, one person can be identified with higher accuracy by considering the identification result of another person.

FIG. 17 shows an example of reference information according to the sixth embodiment. Here, it is assumed that the reference information 1701 to 1713 of FIG. 17 is registered in advance, and the feature information of the two persons A and B is acquired in step S1601. The characteristic information of person A indicates "posture (back): normal", "step ratio: 1.5", and "stay map: M1", and the characteristic information of person B is "posture (back): stoop". , "Step ratio: 1.3", and "Stay map: M2".

In this case, since the characteristic information of the person A is similar to the reference information 1701 to 1704 of the "father", it can be determined in step S305 that the person A is the "father". On the other hand, since the characteristic information of the person B is similar to the reference information 1705 to 1709 of the "mother" and the reference information 1710 of the "sister", in step S305, whether the person B is the "mother" or the "sister". I can't tell if it is.

In the third embodiment, when the person A is determined to be the "father" in step S305, in step S1602, among the reference information 1701 to 1713, the reference information 1706 of the "mother" together with the "father" is obtained. , The reference information 1711 of the "sister" together with the "father" will be referred to. Since the characteristic information of the person B is more similar to the reference information 1706 of the "mother" than the reference information 1711 of the "sister", it can be determined in step S1602 that the person B is the "mother".

Although FIG. 17 shows an example in which the place of stay (stay map) changes depending on the same person, the posture, gesture, flow line, etc. may change depending on the same person. ..

Next, as in the first to fifth embodiments, the output unit 104 outputs the identification results of steps S305 and S1602 (step S306).

As described above, according to the sixth embodiment, by considering the person identified based on the feature information, it is possible to identify another person with higher accuracy than in the first to fifth embodiments. For example, by considering the person identified based on the feature information, the person who could not be identified can be identified.

<Others>
Each of the above embodiments is merely an example of the present invention. The present invention is not limited to the above-mentioned specific form, and various modifications can be made within the scope of its technical idea. It is also possible to appropriately combine each of the above-mentioned configurations, each configuration described below, and the like. For example, if at least one of the posture, gesture, silhouette, flow line, and place of stay is indicated by the feature information, the feature indicated by the feature information is not particularly limited. For example, the feature information indicates one, two, three, four, or five of the posture, gesture, silhouette, flow line, and place of stay. The feature information may show features different from the posture, gesture, silhouette, flow line, and place of stay. The same applies to the reference information.

<Additional notes>
An image acquisition means (101) for acquiring an captured image, and
Information acquisition means (111) for acquiring characteristic information indicating at least one of the posture, gesture, silhouette, flow line, and staying place of the person shown in the image as the characteristic of the person from the image.
An identification means (112) for identifying the person based on the feature information, and
(100), an information processing device (100).

100: Information processing device 101: Image input unit 102: Control unit 103: Storage unit 104: Output unit 111: Information acquisition unit 112: Identification unit 200: Surveillance camera 300: Management device 700: Communication robot 701: Imaging unit 702: Communication Part 400: Image 401: Person 402: Skeleton (skeleton information)
601 to 603, 901 to 903: Reference information 1301 to 1303, 1701 to 1713: Reference information I1 to I3, Ip: Mask image M1 to M3, Mp: Stay map (heat map)

Claims (7)

An image acquisition means for acquiring an captured image, and
Information acquisition means for acquiring characteristic information indicating the place of stay of the person shown in the image as a characteristic of the person from the image, and
An identification means for identifying the person based on the feature information,
Have a,
The feature information includes the stay rate of the person in each of the plurality of images to be imaged as the information of the stay place.
The information processing device is characterized in that the stay rate is the ratio of the stay time to the length of a predetermined period. The information processing apparatus according to claim 1, wherein the feature information further indicates at least one of the posture, gesture, silhouette, and flow line of the person as a feature of the person. The information processing device according to claim 1 or 2 , wherein the information acquisition means acquires skeleton information indicating the skeleton of the person from the image and acquires the feature information based on the skeleton information. For each of the plurality of persons, a storage means for storing reference information indicating the characteristics of the person is further provided.
The information processing apparatus according to any one of claims 1 to 3 , wherein the identification means identifies the person in the image by comparing the feature information with each reference information. When two or more people are shown in the image, the identification means is
A part of the two or more persons is identified based on the characteristic information corresponding to the person, and the person is identified.
Any of claims 1 to 4 , wherein the remaining person among the two or more persons is identified based on the characteristic information corresponding to the person and the identification result of the part of the persons. The information processing apparatus according to item 1. The storage means stores, for each of the plurality of persons, two or more reference information corresponding to each of the two or more combinations of the person and the other person.
When two or more people are shown in the image, the identification means is
A part of the two or more persons is identified by comparing the feature information corresponding to the person with the reference information.
The feature is that the remaining person among the two or more persons is identified by comparing the feature information corresponding to the person with the reference information corresponding to the combination of the identified part of the persons. The information processing apparatus according to claim 4. The information processing apparatus according to any one of claims 1 to 6 , further comprising an imaging means for capturing the image.

Images