JP7631769B2 - Information processing device, matching program, and matching method - Google Patents

Description

The present invention relates to an information processing device, etc.

In recent years, data has been utilized in a variety of fields. Personal data in particular is highly important because it can lead to services tailored to an individual's behavior and preferences. Examples of personal data include captured images of individuals.

Facial recognition technology is used to identify people and recognize their facial expressions in captured images. Facial recognition technology, for example, recognizes a person's facial expression from a facial image of a single person. Furthermore, posture estimation technology is used to recognize a person's posture in a captured image. Posture estimation technology, for example, estimates a person's posture from an image of a single person and estimates skeletal information from the posture.

Technologies using face recognition and posture estimation technologies have been disclosed (see, for example, Patent Documents 1 and 2). For example, in Patent Document 1, face information and skeletal information are extracted from an image of a subject, and the positional relationship between the face position based on the face information and the skeletal position based on the skeletal information is judged to be normal or abnormal, and if it is judged to be abnormal, a message is output indicating that the subject cannot be authenticated.

International Publication No. 2020/152917 JP 2014-053965 A

However, there is a problem in that it is difficult to accurately match face information recognized by face recognition with posture information estimated by posture estimation for an image that contains multiple people. For example, if the target image is an image in which multiple people are close together, such as a group photo, matching between face information and posture information may fail. Here, a case in which matching fails will be described with reference to FIG. 18.

Figure 18 is a diagram illustrating a case where matching fails. Figure 18 shows a group photograph in which multiple people are close together. From such a group photograph, for example, facial information indicated by the symbol a0 is recognized by facial recognition. Furthermore, personal information on which posture information indicated by the symbol b0 is based is estimated by posture estimation. However, in an image showing a group photograph, facial information indicated by the symbol a1 overlaps with other people's personal information, so matching between facial information and personal information may fail. In other words, matching between facial information and posture information may fail.

In one aspect, the present invention aims to accurately match face information recognized by face recognition with pose information estimated by pose estimation for an image containing multiple people.

In one aspect, the information processing device has a first detection unit that detects the faces of multiple people from a captured image including the multiple people, a second detection unit that detects people from the captured image, a posture estimation unit that estimates a posture from the captured image based on the person detection data for each of the multiple people detected by the second detection unit, a calculation unit that calculates face candidate data from skeletal information based on the posture data for the posture estimated by the posture estimation unit, and an identification unit that identifies, for the face candidate data calculated for each of the person detection data, face detection data with the largest overlap area among the face detection data of each person detected by the first detection unit as the same person.

According to one embodiment, for an image containing multiple people, face information recognized by face recognition and posture information estimated by posture estimation can be matched with high accuracy.

FIG. 1 is a block diagram illustrating a functional configuration of an information processing system including an information processing device according to an embodiment. FIG. 2 is a diagram showing an example of a target image. FIG. 3 is a diagram showing an example of face detection data. FIG. 4 is a diagram illustrating an example of human detection data. FIG. 5 is a diagram showing an example of face detection. FIG. 6 is a diagram illustrating an example of human detection. FIG. 7 is a diagram illustrating an example of the posture estimation data. FIG. 8 is a diagram showing an example of human body proportions. FIG. 9 is a diagram showing an example of face candidate data. FIG. 10 is a diagram for explaining coordinates according to the embodiment. FIG. 11A is a diagram (1) for explaining landscape orientation determination according to the embodiment. FIG. 11B is a diagram (2) for explaining landscape orientation determination according to the embodiment. FIG. 12A is a diagram (1) for explaining face candidate extraction according to the embodiment. FIG. 12B is a diagram (2) for explaining face candidate extraction according to the embodiment. FIG. 13A is a diagram (1) for explaining matching according to an embodiment. FIG. 13B is a diagram (2) for explaining matching according to the embodiment. FIG. 14 is a diagram illustrating an example of matching according to the embodiment. FIG. 15 is a diagram illustrating an example of an overall flowchart of information processing according to the embodiment. FIG. 16 is a diagram illustrating an example of a flowchart of the matching process according to the embodiment. FIG. 17 is a diagram illustrating an example of a computer that executes a matching program. FIG. 18 is a diagram for explaining a case where matching fails.

Below, examples of the information processing device, matching program, and matching method disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the examples.

[Configuration of a system including an information processing device]
1 is a block diagram showing a functional configuration of an information processing system including an information processing device according to an embodiment. The information processing system 9 includes an information processing device 1 and a camera 3. The information processing device 1 and the camera 3 are connected, for example, by a network 5. The network 5 may be wireless communication or wired communication, as long as image data can be transmitted from the camera 3 to the information processing device 1. As an example, in the case of wireless communication, the network 5 may be WiFi (registered trademark) or Bluetooth (registered trademark).

The information processing device 1 performs face detection and pose estimation from image data containing multiple people transmitted from the camera 3, and matches the face and pose for each person from the face detection and pose estimation information. An example of a target image used in the embodiment is shown in FIG. 2. FIG. 2 is a diagram showing an example of a target image. As shown in FIG. 2, the target image is a group photo containing multiple people. In such a target image, one person's face overlaps with another person's body, and multiple people are in close proximity.

The information processing device 1 has a communication unit 10, a control unit 20, and a storage unit 30. The communication unit 10 controls communication with other devices. For example, the communication unit 10 receives image data of an image of multiple people from the camera 3. The communication unit 10 may also receive video data of a video of multiple people from the camera 3. Note that in the embodiment, the video data is a bundle of image data, so individual image data will be described.

The memory unit 30 stores various data and programs executed by the control unit 20. For example, the memory unit 30 stores an image storage 31, face detection data 32, person detection data 33, posture estimation data 34, face candidate data 35, and a matching list 36.

The control unit 20 is a processing unit that controls the entire information processing device 1, and has a storage unit 21, a face detection unit 22, a person detection unit 23, a posture estimation unit 24, a landscape orientation determination unit 25, a face candidate extraction unit 26, and a matching unit 27. The storage unit 21 stores image data. For example, the storage unit 21 receives image data transmitted from the camera 3 via the communication unit 10, and stores it in the image storage 31.

The face detection unit 22 detects faces from a captured image in which multiple people are captured. For example, the face detection unit 22 acquires target image data from the image storage 31. The image data is image data of an image in which multiple people are captured. The face detection unit 22 detects individual faces from the acquired image data. One example of a tool used for face detection is OpenCV, but any tool capable of detecting faces will do. The face detection unit 22 generates face detection data 32 in which a face ID is assigned to the face coordinates and face range for each detected face.

Here, an example of the face detection data 32 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the face detection data. As shown in FIG. 3, the face detection data 32 is information that associates a face ID (identifier), an x coordinate, a y coordinate, a width, and a height. The face ID is an identifier that uniquely identifies a face. The x coordinate and the y coordinate are coordinates of a predetermined corner position when the detected face is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and height are the horizontal length and the height of the rectangle when the detected face is shown as a rectangle. This rectangle represents the face area. The face coordinates and the face area are called a bounding box.

As an example, if the face ID is "0", the x coordinate is stored as "1333", the y coordinate as "98", the width as "413", and the height as "413".

Returning to FIG. 1, the person detection unit 23 detects people from a captured image in which multiple people are captured. For example, the person detection unit 23 detects people from the target image data. One example of a tool used for person detection is YOLO (You Only Look Once), but any tool that can detect people will do. For each detected person, the person detection unit 23 generates person detection data 33 in which a body ID is assigned to the person's coordinates and the person's range.

Here, an example of the person detection data 33 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the person detection data. As shown in FIG. 4, the person detection data 33 is information that associates a body ID, an x coordinate, a y coordinate, a width, and a height. The body ID is an identifier that uniquely identifies a person. The x coordinate and the y coordinate are the coordinates of a predetermined corner position when the detected person is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and height are the horizontal length and the height of the rectangle when the detected person is shown as a rectangle. This rectangle represents the range of the person. The coordinates of the person and the range of the person are called a bounding box.

As an example, when the body ID is "0", the x coordinate is stored as "1176", the y coordinate as "109", the width as "754", and the height as "680".

Figure 5 is a diagram showing an example of face detection. As shown in Figure 5, a bounding box of a face detected by the face detection unit 22 is displayed for a target image. The face detection unit 22 detects individual faces for image data of the target image using a face detection tool. The face detection unit 22 then generates face detection data 32 in which a face ID is assigned to the face coordinates and face range for each detected face. The face coordinates are, for example, the bottom left coordinates of a rectangle when the upper left corner of the captured image is the origin, and the bounding box is represented by the face coordinates and face range (width, height).

Figure 6 is a diagram showing an example of person detection. As shown in Figure 6, the bounding box of a person detected by the person detection unit 23 is displayed for the target image. Note that in Figure 6, for convenience of explanation, the bounding box of only one person is displayed, but in reality, the bounding boxes of multiple detected people are displayed. The person detection unit 23 detects individual people using a person detection tool for the image data of the target image. Then, for each detected person, the person detection unit 23 generates person detection data 33 in which a body ID is assigned to the person's coordinates and the person's range. The person's coordinates are, for example, the lower left coordinates of a rectangle when the upper left of the captured image is the origin, and the bounding box is represented by the person's coordinates and the person's range (width, height).

Returning to FIG. 1, the posture estimation unit 24 estimates the posture of a person from the captured image based on the person detection data 33 for each of the person detection data 33 corresponding to the multiple people detected by the person detection unit 23. For example, the posture estimation unit 24 extracts an image (person detection image) corresponding to each person detection data 33 from the captured image. The posture estimation unit 24 estimates position information (skeletal information) of the person's joint points using a posture estimation tool based on each person detection image. An example of a posture estimation tool is DeepPose, but any tool that can estimate position information (skeletal information) of a person's joint points will do. The posture estimation unit 24 generates posture estimation data 34 by assigning the body ID of the person detection data 33 on which the estimated skeletal information is based.

Here, an example of the posture estimation data 34 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the posture estimation data. As shown in FIG. 7, the posture estimation data 34 is information in which a body ID and skeletal information are associated with each other. The body ID corresponds to the body ID of the person detection data 33. The skeletal information is information in which a joint number, an x coordinate, and a y coordinate are associated with each other. The joint number is a number corresponding to a joint point. As an example, the joint number "0" indicates the nose. "1" indicates the left eye. "2" indicates the right eye. "3" indicates the left ear. "4" indicates the right ear. "5" indicates the left shoulder. "6" indicates the right shoulder. "7" indicates the left elbow. "8" indicates the right elbow. "9" indicates the left wrist. "10" indicates the right wrist. "11" indicates the left hip. "12" indicates the right hip. "13" indicates the left knee. "14" indicates the right knee. "15" indicates the left ankle. "16" indicates the right ankle. The joint numbers correspond to the numbers in the joint model of the person in the right diagram of Figure 7. The x and y coordinates are the coordinates of the estimated joint point. The coordinates are those when the origin is the upper left corner of the captured image. Note that the definition of the joint numbers is not limited to that described above.

As an example, when the body ID is "0", the joint number is stored as "5" (left shoulder), the x coordinate as "1173", and the y coordinate as "481". The joint number is stored as "6" (right shoulder), the x coordinate as "953", and the y coordinate as "464".

Returning to FIG. 1, the landscape orientation determination unit 25 uses the human body ratio to determine whether the person's posture estimated by the posture estimation unit 24 is landscape orientation or not.

For example, the sideways determination unit 25 performs the following process for each posture of the person estimated by the posture estimation unit 24. The sideways determination unit 25 uses the human body ratio to determine whether the person is facing sideways from the skeletal information based on the posture estimation data 34 corresponding to the posture. The human body ratio is a calculation of the ratio (coefficient) of a part in the entire human body from the publicly known ratio of the length of each part of the human body. The details of the human body ratio will be described later. As an example, the sideways determination unit 25 calculates the length of a predetermined part using the skeletal information based on the posture estimation data 34. The predetermined part refers to, for example, the upper arm, lower arm, torso, thigh, and calf. The sideways determination unit 25 uses the human body ratio to multiply the length of the predetermined part by a ratio according to the part, and estimates the length of the torso from the length of the part. Then, the sideways determination unit 25 determines the length of the part with the maximum value as the length of the torso. Here, the part-specific ratios used to estimate the torso length are 9/5 for the upper arm, 9/4 for the lower arm, 1 for the torso, 3/2 for the thigh, and 3/2 for the calf. The part coefficients and part-specific ratios are determined from the human body proportions.

The sideways orientation determination unit 25 also calculates the length of the shoulder width using skeletal information based on the posture estimation data 34. The sideways orientation determination unit 25 then determines whether the shoulder width is shorter than 1/3 of the torso length, and if it is determined that the shoulder width is shorter, determines that the body is sideways. The sideways orientation determination unit 25 then associates a sideways orientation flag with the posture estimation data 34 of the person (body ID) whose body has been determined to be sideways. The sideways orientation determination unit 25 then passes the posture estimation data 34 of the person whose body has been determined not to be sideways to the face candidate extraction unit 26 in order to exclude the person whose body has been determined to be sideways from the target for face candidate extraction. The reason why the person whose body has been determined to be sideways is excluded from the target for face candidate extraction is that when the body is sideways, the x coordinate of the position of one shoulder overlaps with the x coordinate of the range of the face, and therefore the face candidate extraction unit 26 described later cannot correctly extract face candidates.

Here, an example of human body ratios will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of human body ratios. The left diagram of FIG. 8 shows the publicly known ratios of the lengths of each part of the human body. The right diagram of FIG. 8 shows the ratios (coefficients) of parts within the entire human body calculated from the ratios of the lengths of each part of the human body. As shown in the right diagram of FIG. 8, the coefficient of the torso is "9", and the coefficient of shoulder width is "6". The coefficients of the chest and waist within the torso are "5" and "4". The coefficients of the thighs and calves are "6" and "6".

Here, the reason why the body is determined to be sideways when the shoulder width is shorter than 1/3 of the torso length is as follows. That is, when the human body is facing forward (or backward), the ratio of the torso to shoulder width is approximately 3:2 based on the human body proportions. When the ratio of the torso to shoulder width is less than 3:1, it is presumed that the shoulder width is narrow and the body is no longer facing forward (or backward). That is, it is presumed that the x coordinate of the position of one shoulder overlaps with the x coordinate of the face range. For this reason, the sideways determination unit 25 determines that the body is sideways when the shoulder width is shorter than 1/3 of the torso length, and excludes it from the target for face candidate extraction.

The face candidate extraction unit 26 extracts data of face candidates using the human body ratio and posture estimation data 34. For example, the face candidate extraction unit 26 acquires the x-coordinate and y-coordinate of the left shoulder (joint No. is "5") and the x-coordinate and y-coordinate of the right shoulder (joint No. is "6") from the skeletal information for the body ID of the target. When the origin is set to the upper left of the captured image of the target, the face candidate extraction unit 26 sets the smaller of the x-coordinate of the left shoulder and the x-coordinate of the right shoulder as the x-coordinate of the rectangle of the face candidate. The face candidate extraction unit 26 sets the larger of the y-coordinate of the left shoulder and the y-coordinate of the right shoulder as the y-coordinate of the rectangle of the face candidate. Then, the face candidate extraction unit 26 sets the length of the horizontal component of the shoulder width as the width of the rectangle of the face candidate. The face candidate extraction unit 26 sets 5/9 of the length of the torso as the height of the rectangle of the face candidate using the human body ratio. The height of the face candidate rectangle is set to 9/5 of the torso length because, according to the human body proportions shown in Figure 8, the ratio of the torso to the head including the neck is 9:5. Then, the face candidate extraction unit 26 generates the coordinates, width, and height of the face candidate rectangle as face candidate data 35 for each target body ID.

Here, an example of the face candidate data 35 will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the face candidate data. As shown in FIG. 9, the face candidate data 35 is information that associates a body ID, an x coordinate, a y coordinate, a width, and a height. The body ID corresponds to the body ID of the person detection data 33. The x coordinate and the y coordinate are the coordinates of a predetermined corner position when the extracted face candidate is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and the height are the horizontal length and the height of the rectangle when the extracted face candidate is shown as a rectangle. This rectangle represents the range of the face candidate. The coordinates of the face candidate and the range of the face candidate are called a bounding box.

Returning to FIG. 1, the matching unit 27 matches (links) the face detection data 32 with the largest overlapping area among the multiple face detection data 32 with the face candidate data 35 as the same person. For example, the matching unit 27 performs the following process for each face candidate data 35. The matching unit 27 calculates the overlapping area between the bounding box obtained from the target face candidate data 35 and the bounding box obtained from the face detection data 32 of all faces detected by the face detection unit 22. The matching unit 27 determines that the body ID of the target face candidate data 35 is matched with the face ID of the face detection data 32 with the largest overlapping area. In other words, the matching unit 27 identifies the matched face ID and body ID as the same person. The matching unit 28 then outputs the identified face ID and body ID to the matching list 36.

Here, the coordinates according to the embodiment will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining the coordinates according to the embodiment. For convenience of explanation, FIG. 10 uses an image in which a part of the captured image to be taken is extracted. The upper left corner of the captured image to be taken is the coordinates (0,0) of the origin. The right direction from the origin is the x coordinate, and the downward direction from the origin is the y coordinate. Here, for example, the coordinates (skeleton information) of the joint points of a person estimated by the posture estimation unit 24 are shown. As an example, the coordinates of the right shoulder are (14,34). The coordinates of the right shoulder are the coordinates corresponding to the joint number "6" in the joint model of the person shown in FIG. 7. The coordinates of the left shoulder are (28,24). The coordinates of the left shoulder are the coordinates corresponding to the joint number "5" in the joint model of the person shown in FIG. 7. The coordinates of the right hip are (28,55). The coordinates of the right hip are the coordinates corresponding to the joint number "12" in the joint model of the person shown in FIG. 7. The coordinates of the left hip are (44, 52). The coordinates of the left hip correspond to joint number "11" in the human joint model shown in Figure 7.

Figures 11A and 11B are diagrams illustrating sideways orientation determination according to an embodiment. Figure 11A shows the bounding boxes of face candidates for a person whose body is turned to the side. The black circles represent the coordinates of the left and right shoulders. When the body is turned to the side, the x-coordinate of the position of one shoulder (here, the right shoulder) overlaps with the x-coordinate of the face range, and face candidates extracted by the face candidate extraction unit 26 are not extracted correctly. Therefore, the sideways orientation determination unit 25 excludes people whose body is turned to the side from the targets for face candidate extraction.

As shown in FIG. 11B, some of the coordinates (skeletal information) of a person's joint points estimated by the posture estimation unit 24 are shown. The coordinates of the right shoulder are (14, 34). The coordinates of the left shoulder are (28, 24). The coordinates of the right hip are (28, 55). The coordinates of the left hip are (44, 52). Although not shown, there are also the coordinates of the left elbow and the left wrist. The space between the right shoulder and right hip, and the space between the left shoulder and left hip, respectively, indicate the torso. The space between the left elbow and left wrist indicates the lower arm.

The sideways orientation determination unit 25 calculates the length of each part using the skeletal information of the posture estimation data 34. Here, as an example, the length of the torso and the lower arm are calculated. Using the coordinates of the right shoulder (14, 34) and the coordinates of the right hip (28, 55), the length of the torso is calculated to be "25". Using the coordinates of the left shoulder (28, 24) and the coordinates of the left hip (44, 52), the length of the torso is calculated to be "32". The length of the lower arm is calculated to be "8" using the coordinates of the left elbow and the left wrist.

The sideways orientation determination unit 25 uses the human body ratio to multiply the length of a specific part by a ratio according to the part, and determines the maximum multiplication result as the length of the torso. Here, the length of the torso calculated from the right shoulder and right hip is calculated as "25" because the ratio is 1. The length of the torso calculated from the left shoulder and left hip is calculated as "32" because the ratio is 1. The length of the torso calculated from the lower arms is calculated as "18" (= 8 x 9/4) because the ratio is 9/4. The sideways orientation determination unit 25 then calculates max (25, 32, 18) and determines the length of the torso as "32". Note that here, as an example, the sideways orientation determination unit 25 determines the length of the torso from the torso and lower arms, but in reality, the length of the torso is determined by adding the upper arms, thighs, calves, etc.

The sideways orientation determination unit 25 also calculates the shoulder width using the coordinates of the right shoulder and the left shoulder. In this case, the coordinates of the right shoulder are (14, 34) and the coordinates of the left shoulder are (28, 24), so the shoulder width is calculated to be "17."

Then, the sideways orientation determination unit 25 determines whether the shoulder width is shorter than 1/3 of the torso length, and if it is determined that the shoulder width is shorter, it determines that the body is sideways. Here, the shoulder width is "17" and the torso length is "32", so the shoulder width "17" is longer than 1/3 of the torso length (approximately 10). Therefore, it is determined that the body is not sideways. As a result, the sideways orientation determination unit 25 makes the posture estimation data 34 of this person a target for face candidate extraction by the face candidate extraction unit 26. Note that, if the sideways orientation determination unit 25 determines that the body is sideways, it does not make the posture estimation data 34 of the person whose body has been determined to be sideways a target for face candidate extraction by the face candidate extraction unit 26.

12A and 12B are diagrams for explaining face candidate extraction according to an embodiment. In FIG. 12A, the bounding box of a person's face candidate is shown. The large black circle indicates the position of the left shoulder and the right shoulder. The small black circle indicates the position of the left hip and the right hip. The length between the shoulder and the hip indicated by the arrow is the length of the torso. The face candidate extraction unit 26 determines the length of the horizontal component of the shoulder width as the width of the bounding box of the face candidate. Using the human body ratio, the face candidate extraction unit 26 determines the height of the bounding box of the face candidate to be 5/9 of the torso length. This is because, according to the human body ratio in FIG. 8, the ratio between the length of the torso and the length of the head and neck is 9:5.

As shown in FIG. 12B, a part of the coordinates (skeletal information) of the joint points of the person estimated by the posture estimation unit 24 is shown. The coordinates of the right shoulder are (14, 34). The coordinates of the left shoulder are (28, 24). The face candidate extraction unit 26 sets the smaller of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the bounding box of the face candidate. Here, the x coordinate of the bounding box indicates "14". The face candidate extraction unit 26 sets the larger of the y coordinate of the left shoulder and the y coordinate of the right shoulder as the y coordinate of the bounding box of the face candidate. Here, the y coordinate of the bounding box indicates "34". The face candidate extraction unit 26 then sets the length of the horizontal component of the shoulder width as the width of the bounding box of the face candidate. Here, the width of the bounding box indicates "14" since the length of the horizontal component of the shoulder width is calculated to be 14. Then, using the human body ratio, the face candidate extraction unit 26 sets the height of the bounding box of the face candidate to 5/9 of the length of the torso. In this case, the length of the torso is "32" as calculated in FIG. 11B, so the height of the bounding box is 18 (= 32 x 5/9). As a result, the bounding box of the face candidate is extracted with the coordinates of the face candidate being (14, 34), the width of the face candidate range being 14, and the height of the face candidate range being 18. The dotted rectangle is the bounding box of the face candidate.

Figures 13A and 13B are diagrams for explaining matching according to an embodiment. Figure 13A shows the overlap between a bounding box bbox1 obtained from one face candidate data 35 and a bounding box bbox2 obtained from one face detection data 32 detected by the face detection unit 22. The coordinates of the face candidate in bbox1 are (bbox1_x, bbox1_y). The width of bbox1 is bbox1_w. The height of bbox1 is bbox1_h. The coordinates of the face candidate in bbox2 are (bbox2_x, bbox2_y). The width of bbox2 is bbox2_w. The height of bbox2 is bbox2_h.

The matching unit 27 determines the four points of the overlapping rectangle between bounding boxes bbox1 and bbox2, the bottom left, top left, bottom right, and top right, as follows: "left" indicates the x coordinate values of the top and bottom of the left of the overlap; "right" indicates the x coordinate values of the top and bottom of the right of the overlap; "bottom" indicates the y coordinate values of the left and right of the bottom of the overlap; and "top" indicates the y coordinate values of the left and right of the top of the overlap.
left=max(bbox1_x, bbox2_x)
top=max(bbox1_y-bbox1_h, bbox2_y-bbox2_h)
right=min(bbox1_x+bbox1_w, bbox2_x+bbox2_w)
bottom=min(bbox1_y, bbox2_y)
That is, the coordinates of the bottom left corner of the overlapping rectangle are (left, bottom). The coordinates of the top left corner of the overlapping rectangle are (left, top). The coordinates of the bottom right corner of the overlapping rectangle are (right, bottom). The coordinates of the top right corner of the overlapping rectangle are (right, top).

Then, if an overlapping rectangle exists, the matching unit 27 calculates the area of the overlap. Here, if the left is smaller than the right and the top is smaller than the bottom, it is determined that an overlapping rectangle exists. Then, the area of the overlap is calculated as follows.
Overlap area = (right-left) x (bottom-top)

As shown in FIG. 13B, a bounding box (dashed line) obtained from face candidate data 35 and a bounding box (solid line) obtained from face detection data 32 are shown. For the bounding box obtained from face candidate data 35, the coordinates of the face candidate are represented as (14, 34), and the width and height of the face candidate are represented as 14 and 18, respectively. For the bounding box obtained from face detection data 32, the coordinates of the face are represented as (14, 28), and the width and height of the face are represented as 12 and 13, respectively.

Then, the matching unit 27 finds the four points (lower left, upper left, lower right, and upper right) of the overlapping rectangle as follows.
left=max(14,14)=14
top=max(28-13,34-18)=max(15,16)=16
right=min(26,28)=26
bottom=min(28,34)=28
That is, the coordinates of the bottom left corner of the overlapping rectangle are (14, 28). The coordinates of the top left corner of the overlapping rectangle are (14, 16). The coordinates of the bottom right corner of the overlapping rectangle are (26, 28). The coordinates of the top right corner of the overlapping rectangle are (26, 16).

Then, since left (14) is smaller than right (26) and top (16) is smaller than bottom (28), an overlapping rectangle exists, so the matching unit 27 calculates the overlapping area. The overlapping area is calculated as 144 (= (26-14) x (28-16)).

Figure 14 is a diagram showing an example of matching according to the embodiment. The dashed rectangle shown in Figure 14 is assumed to be a bounding box obtained from face candidate data 35 showing a body ID of "1".

The matching unit 27 finds the overlap area between the bounding box obtained from the target face candidate data 35 and the bounding boxes obtained from the face detection data 32 of all faces detected by the face detection unit 22. Here, the bounding boxes obtained from the face detection data 32 that overlap the bounding box of the face candidate indicating a body ID of "1" are shown. The overlapping face detection data 32 face IDs are "1", "2", and "3". The overlap areas between the body ID "1" and the face IDs "1", "2", and "3" are assumed to be "20", "30", and "10".

The matching unit 27 then determines that the body ID of the target face candidate data 35 is matched with the face ID of the face detection data 32 with the largest overlapping area. Here, the overlapping area between body ID "1" and face ID "2" is the largest. Therefore, it is determined that body ID "1" and face ID "2" are matched. The matching unit 27 then links body ID "1" and face ID "2" together and outputs them to the matching list 36.

Fig. 15 is a diagram showing an example of an overall flowchart of information processing according to an embodiment. As shown in Fig. 15, the control unit 20 inputs a video image showing multiple people (step S11). The control unit 20 performs subsequent processing on the input video image on a frame-by-frame basis (step S12).

The face detection unit 22 detects faces in the image data, assigns a face ID to the face coordinates and face area for each detected face, and generates face detection data 32 (step S13). The face detection unit 22 then proceeds to step S20 to perform matching processing.

Meanwhile, the person detection unit 23 detects people from the image data, assigns a body ID to the coordinates and range of each detected person, and generates person detection data 33 (step S14).

Next, the posture estimation unit 24 extracts from the image an image (person detection image) corresponding to each person detection data 33 (step S15). Then, the posture estimation unit 24 performs posture estimation for all extracted person detection images, and estimates skeletal information obtained by the posture estimation. The posture estimation unit 24 assigns the body ID of the person detection data 33 on which it is based to the skeletal information, and generates posture estimation data 34 (step S16). Skeletal information here refers to position information of a person's joint points.

Next, the sideways orientation determination unit 25 uses the human body ratio and posture estimation data 34 to determine whether or not a person's body is sideways (step S17). For example, the sideways orientation determination unit 25 calculates the shoulder width using the skeletal information of the posture estimation data 34, determines whether or not the shoulder width is shorter than 1/3 of the torso length, and if it is determined to be shorter, determines that the body is sideways. The torso length is calculated from the lengths of the upper arm, lower arm, torso, thigh, and calf calculated from the skeletal information and the human body ratio.

If it is determined that there is no person with a sideways body (step S17; No), the sideways orientation determination unit 25 proceeds to step S19 to generate face candidates. On the other hand, if it is determined that there is a person with a sideways body (step S17; Yes), the sideways orientation determination unit 25 associates a sideways orientation flag with the posture estimation data 34 of the person with a sideways body (step S18). Then, the sideways orientation determination unit 25 proceeds to step S19 to generate face candidates.

In step S19, the face candidate extraction unit 26 calculates the coordinates and range of the face candidate from the posture estimation data 34 (skeletal information) without the correspondence between the human body ratio and the landscape flag to generate face candidate data 35 (step S19). For example, the face candidate extraction unit 26 acquires the x and y coordinates of the left shoulder and the x and y coordinates of the right shoulder from the skeleton information for the body ID of the posture estimation data 34. The face candidate extraction unit 26 sets the smaller value of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the rectangle of the face candidate, and sets the larger value of the y coordinate of the left shoulder and the y coordinate of the right shoulder as the y coordinate of the rectangle of the face candidate. Then, the face candidate extraction unit 26 sets the length of the horizontal component of the shoulder width as the width of the rectangle of the face candidate. The face candidate extraction unit 26 uses the human body ratio to set 5/9 of the length of the torso as the height of the rectangle of the face candidate. The length of the torso is calculated by the landscape determination unit 25. Then, the face candidate extraction unit 26 generates the coordinates, width, and height of the face candidate for each body ID in the posture estimation data 34 as face candidate data 35.

Then, the matching unit 27 executes a matching process for each body ID in the posture estimation data 34 (step S20). Note that a flowchart of the matching process will be described later. Then, if there are unprocessed images, the control unit 20 proceeds to step S12 (step S21). On the other hand, if there are no unprocessed images, the control unit 20 ends the information processing.

FIG. 16 is a diagram showing an example of a flowchart of the matching process according to the embodiment. As shown in FIG. 16, the matching unit 27 executes the process on a body ID basis (step S31). The matching unit 27 determines whether or not a landscape flag is associated with the posture estimation data 34 corresponding to the body ID (step S32). If it is determined that a landscape flag is associated with the posture estimation data 34 corresponding to the body ID (step S32; Yes), the matching unit 27 proceeds to step S37 to execute the process corresponding to the next body ID.

On the other hand, if it is determined that the landscape flag is not associated with the posture estimation data 34 corresponding to the body ID (step S32; No), the matching unit 27 executes processing on a face ID basis (step S33). The matching unit 27 calculates the overlapping area of the bounding boxes (coordinates and range) of the body ID and the face ID (step S34). If there is an unprocessed face ID, the matching unit 27 proceeds to step S33 (step S35). On the other hand, if there is no unprocessed face ID, the matching unit 27 proceeds to step S36 to match the body ID and the face ID.

In step S36, the matching unit 27 matches the body ID and face ID with the largest overlapping area of the bounding boxes (coordinates and range) as a pair (step S36). If there is an unprocessed body ID, the matching unit 27 proceeds to step S31 (step S37).

On the other hand, if there is no unprocessed body ID, the matching unit 27 outputs the matching result to the matching list 36 (step S38). Then, the matching unit 27 ends the matching process.

The matching process executed by the information processing device 1 according to the embodiment can be applied to a video distribution service. For example, the video distribution service matches the faces and bodies (postures) of multiple people in a video. In addition, the video distribution service considers a change in facial expression or body movement to be a characteristic of the moment when a person's emotions change, and distributes an image of that moment and video before and after that moment to the user. One example is a video of a part of a bowling game where everyone gets a strike and high-fives. The video distribution service may also highlight the target person and distribute an image of that moment and video before and after that moment to the user.

[Effects of the embodiment]
In this way, in the above embodiment, the information processing device 1 detects the faces of each person from a captured image including multiple people. The information processing device 1 detects people from a captured image. For each person detection data corresponding to the multiple detected people, the information processing device 1 estimates a posture from the captured image based on the person detection data. The information processing device 1 calculates face candidate data from skeletal information based on posture data for the estimated posture. For the face candidate data calculated for each person detection data, the information processing device 1 identifies face detection data with the largest overlap area among the face detection data of each detected person as the same person. With this configuration, the information processing device 1 can improve the matching accuracy between a face detected from face detection and a posture (body) estimated from posture estimation. That is, the information processing device 1 can match a face and a body with high accuracy even on an image where people are crowded together.

In addition, in the above embodiment, the information processing device 1 calculates face candidate data based on human body ratios that indicate the ratios of parts of the entire human body and skeletal information obtained from posture data. With this configuration, the information processing device 1 can calculate candidates for the position of the face from the posture data by using the human body ratios.

In addition, in the above embodiment, the information processing device 1 uses human body proportions to estimate the length of the torso from the length of a specific part obtained from the skeletal information. The information processing device 1 calculates face candidate data based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso. With this configuration, the information processing device 1 can calculate candidates for the position of the face by using the position of the shoulders and the length of the torso.

In the above embodiment, the information processing device 1 further determines whether the body is facing sideways using the shoulder width length and the torso length. If the information processing device 1 determines that the body is facing sideways, it does not calculate face candidate data from skeletal information. With this configuration, the information processing device 1 can determine whether the body is facing sideways by using human body proportions. Then, the information processing device 1 can avoid calculating face candidates with low accuracy by excluding skeletal information in which the body is facing sideways from the calculation of face candidates.

[others]
In the embodiment, the coordinates of the face detection data 32, the person detection data 33, the posture estimation data 34, and the face candidate data 35 are described as having the origin at the top left of the captured image. However, the origin may be located at the bottom left of the captured image. In such a case, for example, the face candidate extraction unit 26 may set the smaller of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the rectangle of the face candidate, and set the smaller of the y coordinate of the left shoulder and the y coordinate of the rectangle of the face candidate.

In addition, each component of the illustrated information processing device 1 does not necessarily have to be physically configured as illustrated. In other words, the specific manner in which each device is distributed and integrated is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. For example, the landscape determination unit 25 and the face candidate extraction unit 26 may be integrated into one unit. The landscape determination unit 25 may also be distributed into a determination unit that determines the length of the torso and a determination unit that determines landscape orientation using the length of the torso. In addition, the image storage 31, the storage unit 30 that stores the face detection data 32, etc. may be connected via a network as an external device of the information processing device 1. Furthermore, the control unit 20 and the storage unit 30 may be placed on a cloud, and the user may connect to the cloud via a network from a personal computer or the like to use the processing executed by the control unit 20. In this way, the processing of the present invention can be provided as a service.

The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, an example of a computer that executes a matching program that realizes the same functions as the information processing device 1 shown in FIG. 1 will be described below. FIG. 17 is a diagram showing an example of a computer that executes a matching program.

As shown in FIG. 17, computer 200 has a CPU 203 that executes various arithmetic processes, an input device 215 that accepts data input from a user, and a display device 209. Computer 200 also has a drive device 213 that reads programs and the like from a storage medium, and a communication I/F (Interface) 217 that transmits and receives data to and from other computers via a network. Computer 200 also has memory 201 that temporarily stores various information, and HDD (Hard Disk Drive) 205. Memory 201, CPU 203, HDD 205, display device 209, drive device 213, input device 215, and communication I/F 217 are connected by bus 219.

The drive device 213 is, for example, a device for the removable disk 211. The HDD 205 stores the matching program 205a and the matching process related information 205b. The communication I/F 217 is responsible for interfacing between the network and the inside of the device, and controls the input and output of data from other computers. For example, a modem or a LAN adapter can be used as the communication I/F 217.

The display device 209 is a display device that displays data such as a cursor, an icon, a tool box, documents, images, and function information. The display device 209 may be, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

The CPU 203 reads the matching program 205a, expands it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the information processing device 1. The matching process related information 205b corresponds to the image storage 31, face detection data 32, etc. And, for example, the removable disk 211 stores each piece of information such as the matching program 205a.

The matching program 205a does not necessarily have to be stored in the HDD 205 from the beginning. For example, the program can be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 200. The computer 200 can then read and execute the matching program 205a from the medium.

REFERENCE SIGNS LIST 1 Information processing device 3 Camera 5 Network 9 Information processing system 10 Communication unit 20 Control unit 21 Storage unit 22 Face detection unit 23 Person detection unit 24 Posture estimation unit 25 Landscape orientation determination unit 26 Face candidate extraction unit 27 Matching unit 30 Memory unit 31 Image storage 32 Face detection data 33 Person detection data 34 Posture estimation data 35 Face candidate data 36 Matching list

Claims (3)

a first detection unit that detects a face of each person from a captured image including a plurality of people;
a second detection unit that detects each person from the captured image;
a posture estimation unit that estimates, for each person detection data corresponding to each person detected by the second detection unit, a posture corresponding to the person detection data from the captured image;
a first calculation unit that uses a human body ratio indicating a ratio of a part in the entire human body to estimate a length of a torso according to a length of a predetermined part obtained from skeletal information based on a posture estimated by the posture estimation unit, calculates a shoulder width using the skeletal information, and calculates face candidate data from the skeletal information based on posture data for the posture estimated by the posture estimation unit;
a determination unit that determines whether or not the body is lying sideways using the shoulder width and the torso length;
a second calculation unit that calculates the face candidate data based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso when it is determined that the body is not turned sideways, and does not calculate the face candidate data from the skeletal information when it is determined that the body is turned sideways;
an identification unit that identifies, for face candidate data calculated for each of the person detection data, face detection data having a largest overlapping area among the face detection data of each person detected by the first detection unit, as the same person;
13. An information processing device comprising: Detecting the faces of multiple people from a captured image containing each person,
Detecting each person from the captured image;
For each person detection data corresponding to each of the detected people, estimating a posture corresponding to the person detection data from the captured image;
Using a human body ratio indicating the ratio of a part to the whole human body, a torso length is estimated according to the length of a predetermined part obtained from skeletal information based on the estimated posture, and a shoulder width is calculated using the skeletal information . Face candidate data is calculated from the skeletal information based on posture data for the estimated posture.
determining whether the body is lying sideways using the shoulder width and the torso length;
When it is determined that the body is not turned sideways, the face candidate data is calculated based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso, and when it is determined that the body is turned sideways, the face candidate data is not calculated from the skeletal information,
identifying, for the face candidate data calculated for each of the person detection data, the face detection data having the largest overlapping area among the face detection data of each of the detected persons as the same person;
A matching program that causes a computer to execute the processing. Detecting the faces of multiple people from a captured image containing each person,
Detecting each person from the captured image;
For each person detection data corresponding to each of the detected people, estimating a posture corresponding to the person detection data from the captured image;
Using a human body ratio indicating the ratio of a part to the whole human body, a torso length is estimated according to the length of a predetermined part obtained from skeletal information based on the estimated posture, and a shoulder width is calculated using the skeletal information . Face candidate data is calculated from the skeletal information based on posture data for the estimated posture.
determining whether the body is lying sideways using the shoulder width and the torso length;
When it is determined that the body is not turned sideways, the face candidate data is calculated based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso, and when it is determined that the body is turned sideways, the face candidate data is not calculated from the skeletal information,
identifying, for the face candidate data calculated for each of the person detection data, the face detection data having the largest overlapping area among the face detection data of each of the detected persons as the same person;
A matching method in which processing is performed by a computer.

Images