JP7743549B2 - Information processing device, program, system, and information processing method - Google Patents

Description

The present invention relates to an information processing device, a program, a system, and an information processing method.

Patent document 1 describes an image display device that cuts out a portion of a forward image and performs projective transformation on the cut-out image to generate a downward image corresponding to a rectangular road surface area located below a vehicle VC at a second timing.
[Prior art documents]
[Patent Documents]
[Patent Document 1] JP 2023-141792 A

According to one embodiment of the present invention, an information processing device is provided. The information processing device may include an image acquisition unit that acquires a first captured image of an object and a ground surface on which the object is grounded by a first imaging unit that captures an image of the object and the ground surface from a position other than directly above the object, and a second captured image including the object and the ground surface that is captured by a second imaging unit that captures an image of the object from directly above. The information processing device may include a rectangular area identification unit that analyzes the first captured image and identifies a first rectangular area that indicates the range of the object in the first captured image. The information processing device may include a ground surface area identification unit that identifies a ground surface area that indicates the range of the object within the ground surface in the first captured image based on the first rectangular area. The information processing device may include a projection transformation unit that projectively transforms the ground surface area in the first captured image into the second captured image using a plurality of predetermined reference points of the ground surface in the first captured image and the plurality of reference points of the ground surface in the second captured image. The information processing device may include a rectangular area determination unit that determines a second rectangular area indicating the area of the object in the second captured image based on the contact surface area projected into the second captured image by the projective transformation unit.

In the information processing device, the image acquisition unit may acquire the first captured image of the object and the ground surface captured by the first imaging unit, which captures images of the object and the ground surface from a position different from directly above the object and from an obliquely upward position substantially equal to directly above the object, and the second captured image of the object and the ground surface captured by the second imaging unit, which captures images of the object from directly above the object or from an obliquely upward position substantially equal to directly above the object. In the information processing device, the object may be a person. In any of the information processing devices, the ground contact area identification unit may identify a first point and a second point that are two lower vertices of the first rectangular area, a third point within the rectangular area that is the horizontal center of the rectangular area identified by the rectangular area identification unit and is located at a length from the top of the rectangular area that is a predetermined percentage of the vertical length of the rectangular area, a fourth point that is point-symmetrical to the first point with the third point as the center, and a fifth point that is point-symmetrical to the second point with the third point as the center, thereby identifying the quadrangular ground contact area formed by the first point, the second point, the fourth point, and the fifth point.The projective transformation unit may project the first point, the second point, the third point, the fourth point, and the fifth point in the first captured image into the second captured image using the multiple reference points of the ground contact area in the first captured image and the multiple reference points in the second captured image.

In any of the information processing devices, the rectangular area identification unit may estimate the posture of the subject in the first captured image, and based on the estimated posture of the subject, identify the midpoint between the ground contact point of the subject's right foot and the ground contact point of the subject's left foot as the center of gravity of the subject. The projection transformation unit may further projectively transform the center of gravity of the subject into the second captured image. The rectangular area determination unit may determine the size of the second rectangular area from the ground contact area projectively transformed into the second captured image by the projection transformation unit, and may set the center of gravity of the subject projected into the second captured image by the projection transformation unit as the center point of the second rectangular area.

In any of the information processing devices, the rectangular area determination unit may determine the size of the second rectangular area by taking the larger of the difference between the maximum and minimum horizontal coordinate values of the first point, the second point, the fourth point, and the fifth point and the difference between the maximum and minimum vertical coordinate values as the length of one side. The rectangular area identification unit may estimate the subject's skeleton including a point indicating the subject's right ankle, a point indicating the right knee, a point indicating the left ankle, and a point indicating the left knee, and identify the subject's center of gravity by adjusting the vertical position of the midpoint between the point indicating the right ankle and the point indicating the left ankle based on the length of the shin calculated from the point indicating the right ankle and the point indicating the right knee or the point indicating the left ankle and the point indicating the left knee.

In any of the information processing devices described above, the image acquisition unit may select, as the first captured image, an image in which the distance between the subject's right foot and left foot is the greatest from among a plurality of images of the subject captured continuously by the first imaging unit.

Any of the information processing devices may include an annotation unit that assigns metadata representing the object to the rectangular area determined by the rectangular area determination unit. In any of the information processing devices, the projective transformation unit may identify the plurality of reference points in the first captured image and the plurality of reference points in the second captured image by detecting a plurality of markers arranged on the ground surface. The information processing device may perform machine learning using training data including captured images to which metadata has been assigned by the annotation unit. The learning execution unit may use a plurality of pieces of training data to generate a learning model that, when a captured image includes a person captured from directly above, is capable of outputting that that part of the captured image is a person.

According to one embodiment of the present invention, an information processing device is provided. The information processing device may include an image acquisition unit that acquires a first captured image of a person captured by a first imaging unit with the person's torso, limbs, and head separated, and a second captured image of the person captured by a second imaging unit that images the person from a different direction from the first imaging unit with the person's torso, limbs, and head not separated. The information processing device may include a rectangular area identification unit that analyzes the first captured image and identifies a first rectangular area that indicates the area of the person in the first captured image. The information processing device may include a ground plane area identification unit that identifies a ground plane area that indicates the area of the person within the ground plane of the person in the first captured image based on the first rectangular area. The information processing device may include a projection transformation unit that projectively transforms the ground plane area in the first captured image into the second captured image using a plurality of predetermined reference points of the ground plane in the first captured image and the plurality of reference points of the ground plane in the second captured image. The information processing device may include a rectangular area determination unit that determines a second rectangular area indicating the area of the person in the second captured image based on the ground plane area projectively transformed into the second captured image by the projective transformation unit.

According to one embodiment of the present invention, a program is provided that, when executed by a computer, causes the computer to function as the information processing device.

According to one embodiment of the present invention, a system is provided that includes the information processing device, the first imaging unit, and the second imaging unit.

According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include an image acquisition step in which a first imaging unit, which images an object and a ground surface on which the object is in contact with the ground from a position other than directly above the object, acquires a first captured image of the object and the ground surface, and a second captured image including the object and the ground surface, which is captured by a second imaging unit, which images the object from directly above. The information processing method may include a rectangular range identification step in which the first captured image is analyzed to identify a first rectangular range indicating the range of the object in the first captured image. The information processing method may include a ground surface range identification step in which a ground surface range indicating the range of the object within the ground surface in the first captured image is identified based on the first rectangular range. The information processing method may include a projection transformation step of projecting the ground plane range in the first captured image into the second captured image using a plurality of predetermined reference points of the ground plane in the first captured image and the plurality of reference points of the ground plane in the second captured image, and a rectangular area determination step of determining a second rectangular area indicating the range of the object in the second captured image based on the ground plane range projected into the second captured image in the projection transformation step.

According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include an image acquisition step of acquiring a first captured image of a person captured by a first imaging unit with the person's torso, limbs, and head separated, and a second captured image of the person captured by a second imaging unit, which images the person from a different direction from the first imaging unit, with the person's torso, limbs, and head not separated. The information processing method may include a rectangular area identification step of analyzing the first captured image to identify a first rectangular area indicating the area of the person in the first captured image. The information processing method may include a ground surface area identification step of identifying a ground surface area indicating the area of the person within the ground surface of the person in the first captured image based on the first rectangular area. The information processing method may include a projection transformation step of projecting the ground surface area in the first captured image into the second captured image using a plurality of predetermined reference points of the ground surface in the first captured image and the plurality of reference points of the ground surface in the second captured image. The information processing method may include a rectangular area determination step of determining a second rectangular area indicating the area of the person in the second captured image based on the ground plane area projectively transformed into the second captured image in the projective transformation step.

Note that the above summary of the invention does not list all of the features of the present invention. Subcombinations of these features may also constitute inventions.

1 illustrates an example of a system 10 in schematic form. 10 is an explanatory diagram for explaining identification of a ground contact surface range by the information processing device 300. FIG. 10 is an explanatory diagram for explaining projective transformation of the contact surface range by the information processing device 300. FIG. FIG. 10 is an explanatory diagram illustrating how the information processing device 300 determines the size of a rectangular area. 2 shows an example of a functional configuration of an information processing device 300. 10 is an explanatory diagram for explaining identification of a center of gravity of an object by the information processing device 300. FIG. 10 shows an example of a processing flow of the information processing device 300. 1 shows an example of a hardware configuration of a computer 1200 that functions as the information processing device 300.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

When a person is photographed from directly above, from a diagonal angle substantially equal to directly above, directly below, or from a diagonal angle substantially equal to directly below (sometimes referred to as "directly above, etc."), it is difficult to automatically recognize that the person is a person. While many conventional annotated images are composed of images of people captured from the side, capturing the head, torso, legs, etc., when a person is photographed from directly above, etc., at least a portion of the torso may be hidden by the head, or at least a portion of the legs may be hidden by the torso, resulting in the head, torso, legs, etc. not being captured, resulting in a significant difference between the shape of the person photographed in the captured image and the shape of the person photographed in the annotated image. By performing machine learning using a large number of annotated images in which metadata indicating that the area of a person in an image captured from directly above, etc. is added to the area, the accuracy of detecting people from directly above, etc. can be improved. However, because it is difficult to automatically assign metadata to images of a person captured from directly above, annotated images must be prepared manually, making it practically difficult to prepare a large number of annotated images. In the system 10 according to the present embodiment, for example, in addition to a main camera that captures a person from directly above, an auxiliary camera that captures a person from a position other than directly above is used. For example, the system 10 uses an auxiliary camera that captures a person from the side. For example, the system 10 uses an auxiliary camera that captures a person from an obliquely above. As is well known, if a person is captured from the side or from an obliquely above, it is possible to recognize that the person is included in the captured image. Therefore, by using an auxiliary camera in addition to the main camera, a person can be recognized from the image captured by the auxiliary camera, and metadata representing the person can be assigned to an area of the image captured by the main camera that includes the person. As described above, the main camera is not limited to being installed so as to capture a person from directly above, but may also be installed so as to capture a person from directly below, from an obliquely above that is substantially equal to directly above, or from an obliquely below that is substantially equal to directly below. An angle diagonally above that is substantially equivalent to directly above may be within a range in which it is difficult to automatically recognize a person when an image of a person is captured from that angle. An angle diagonally below that is substantially equivalent to directly below may be within a range in which it is difficult to automatically recognize a person when an image of a person is captured from that angle. This can contribute to the automatic generation of annotated images. Note that system 10 is not limited to people, and may generate annotated images for any object that is difficult to automatically recognize from directly above, but can be automatically recognized from other angles. In this embodiment, the main example is a case in which the object is a person and the main camera is installed to capture an image of the person from directly above.

Figure 1 shows a schematic diagram of an example of a system 10. The system 10 includes an imaging unit 100, an imaging unit 200, and an information processing device 300. The imaging unit 100 may be an example of a first imaging unit. The imaging unit 100 may be used as an auxiliary camera. The imaging unit 200 may be an example of a second imaging unit. The imaging unit 200 may be used as a main camera.

The information processing device 300 communicates with the imaging unit 100 and the imaging unit 200. The information processing device 300 communicates with the imaging unit 100 and the imaging unit 200, for example, via the network 20.

The network 20 may include the Internet. The network 20 may include the cloud. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may be compliant with a 5G (5th Generation) communication system. The mobile communication network may be compliant with an LTE (Long Term Evolution) communication system. The mobile communication network may be compliant with a 3G (3rd Generation) communication system. The mobile communication network may be compliant with a 6G (6th Generation) communication system or later mobile communication system.

The information processing device 300 may be connected to the network 20 via a wired connection. The information processing device 300 may be connected to the network 20 wirelessly. The information processing device 300 may be connected to the network 20 via a wireless base station. The information processing device 300 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

The imaging unit 100 may be connected to the network 20 via a wired connection. The imaging unit 100 may be connected to the network 20 wirelessly. The imaging unit 100 may be connected to the network 20 via a wireless base station. The imaging unit 100 may be connected to the network 20 via a Wi-Fi access point.

The imaging unit 200 may be connected to the network 20 via a wired connection. The imaging unit 200 may be connected to the network 20 wirelessly. The imaging unit 200 may be connected to the network 20 via a wireless base station. The imaging unit 200 may be connected to the network 20 via a Wi-Fi access point.

The imaging unit 100 and the information processing device 300 may be directly connected via a communication cable. The imaging unit 200 and the information processing device 300 may be directly connected via a communication cable.

The imaging unit 100 may be any device capable of capturing an image of the target person 50 and providing the captured image 500 to the information processing device 300. For example, the imaging unit 100 may be a camera device such as a surveillance camera or network camera. The imaging unit 100 may also be a smartphone or tablet terminal. The imaging unit 100 may provide the information processing device 300 with the captured image 500, for example, capturing the target person 50 with the torso, limbs, and head of the target person 50 separated. For example, the imaging unit 100 is positioned in a position where it can capture an image of the target person 50 from a position other than directly above, and captures an image of the target person 50 and the ground surface 60 on which the target person 50 is standing. For example, the imaging unit 100 is positioned in a position where it can capture an image of the target person 50 from diagonally above, and captures an image of the target person 50 and the ground surface 60 from diagonally above. The captured image 500 may be an example of a first captured image.

The imaging unit 200 may be any device that can capture an image of the target person 50 and provide the captured image 600 to the information processing device 300. For example, the imaging unit 200 may be a camera device such as a surveillance camera or network camera. The imaging unit 200 may also be a smartphone or tablet terminal. The imaging unit 200 captures the target person 50 from a direction different from that of the imaging unit 100. The imaging unit 200 may provide the information processing device 300 with a captured image 600 in which the target person 50's torso, limbs, and head are not separated. For example, the imaging unit 200 is positioned in a position where it can capture an image of the target person 50 from directly above, and captures an image from above downward to capture the target person 50 and the ground surface 60 from directly above. The captured image 600 may be an example of a second captured image.

The information processing device 300 acquires a captured image 500 from the imaging unit 100, in which the imaging unit 100 captures an image of the target person 50 and the ground surface 60. The information processing device 300 acquires a captured image 600 from the imaging unit 200, in which the imaging unit 200 captures an image of the target person 50 and the ground surface 60. The information processing device 300, for example, analyzes the captured image 500 to identify a rectangular area indicating the range of the target person 50 in the captured image 500, identifies a ground surface area from the rectangular area indicating the range of the target person 50 within the ground surface 60 in the captured image 500, and performs a projective transformation of the ground surface area in the captured image 500 onto the captured image 600. The information processing device 300 determines a rectangular area indicating the range of the target person 50 in the captured image 600 based on the ground surface area projectively transformed onto the captured image 600, and assigns metadata representing the target person 50 to the determined rectangular area. Even if the rectangular area of the target person 50 within the captured image 600 cannot be directly identified from the captured image 600, it can be indirectly identified by using the captured image 500, and an annotated image can be generated for the captured image 600 in which the target person 50 is captured from directly above, with metadata of the target person 50 added to appropriate areas. The information processing device 300 may be any device that can execute the above-described processing. The information processing device 300 may be, for example, a PC (Personal Computer), a server device, a smartphone, etc.

Figure 2 is an explanatory diagram illustrating how the information processing device 300 identifies the ground surface range 518. The captured image 500 is captured by the imaging unit 100, and includes the target person 50 and the ground surface 60 within the captured range.

A plurality of reference points are arranged on the ground surface 60. In the example shown in FIG. 2, reference points 71, 72, 73, and 74 are arranged on the ground surface 60. The plurality of reference points may be arranged in any manner so long as their positions can be identified in the captured image. For example, the plurality of reference points may be indicated by tape, such as a marking. The shapes of the plurality of reference points may be the same or different from each other.

In the example shown in FIG. 2, the target person 50 included in the captured image 500 is seen from behind with his left foot forward while walking. The information processing device 300 analyzes the captured image 500 and identifies a rectangular bounding box 510 that indicates the extent of the target person 50 in the captured image 500. The bounding box 510 may be an example of a first rectangular area.

In this embodiment, a rectangle may be a shape with four right-angled vertices, and may include a rectangle and a square. In this embodiment, a bounding box may be a partial area that surrounds an object in an image or video. The information processing device 300 may use well-known technology to identify a bounding box 510 that surrounds the target person 50 in the captured image 500.

The information processing device 300 may use the bounding box 510 to identify the contact surface range 518. For example, the information processing device 300 identifies points 511 and 512, which are the two lower vertices of the bounding box 510. Point 511 may be an example of a first point. Point 512 may be an example of a second point. The information processing device 300 may identify point 513 within the bounding box 510, which is the center of the horizontal length of the bounding box 510 and is located a predetermined percentage of the vertical length of the bounding box 510 from the top of the bounding box 510. Point 513 may be an example of a third point. The predetermined percentage may be set in advance and may be changeable. The example shown in FIG. 2 illustrates a case where the predetermined percentage is set to 0.95.

The information processing device 300 may identify point 514, which is a position symmetrical to point 511 with point 513 as the center. The information processing device 300 may identify point 515, which is a position symmetrical to point 512 with point 513 as the center. Point 514 may be an example of a fourth point. Point 515 may be an example of a fifth point. The information processing device 300 may identify points 514 and 515 by vector calculation. The information processing device 300 may identify a quadrangular contact surface range 518 formed by points 511, 512, 514, and 515.

Figure 3 is an explanatory diagram illustrating the projective transformation of the ground plane range 518 performed by the information processing device 300. In the example shown in Figure 3, for the sake of explanation, four reference points 71, 72, 73, and 74 are shown with different shapes.

Captured image 600 is captured by imaging unit 200, and includes target person 50 and ground surface 60 within the imaging range. Captured image 600 also includes reference points 71 to 74, which are also included within the imaging range of captured image 500, at positions captured from directly above.

The information processing device 300 uses reference points 71, 72, 73, and 74 in the captured image 500 and reference points 71, 72, 73, and 74 in the captured image 600 to project five points 511, 512, 513, 514, and 515 in the captured image 500 onto the captured image 600.

Figure 4 is an explanatory diagram illustrating how the information processing device 300 determines the size of a rectangular area. The information processing device 300 may determine the size of a rectangular area surrounding the target person 50 in the captured image 600 using points 511, 512, 514, and 515 that have been projectively transformed onto the captured image 600.

For example, the information processing device 300 determines the size of the rectangle information by taking the larger of the difference between the maximum and minimum values in the horizontal direction and the difference between the maximum and minimum values in the vertical direction of the coordinates of points 511, 512, 514, and 515 as the length of one side. In the example shown in Fig. 4, the information processing device 300 determines the difference 612 as the difference between the maximum and minimum values in the horizontal direction, which is the length obtained by subtracting the x-coordinate (x _min ) of point 515, which is the minimum value of the x-coordinate, from the x-coordinate (x _max ) of point 512, which is the maximum value of the x-coordinate. The information processing device 300 may generate rectangle information 622 with the difference 612 as one side. The information processing device 300 may determine, as the difference between the maximum and minimum values in the vertical direction, the length obtained by subtracting the y coordinate (y _min ) of point 514, which is the minimum value of the y coordinate, from the y coordinate (y _max ) of point 511, which is the maximum value of the y coordinate, as difference 614. The information processing device 300 may generate rectangle information 624, one side of which is equal to the difference 614. The information processing device 300 determines the size of the rectangle information by taking the larger of the difference 612 and the difference 614 as the length of one side. In the example shown in FIG. 4 , the information processing device 300 determines the size of the rectangle information 622 as the size of a rectangular area surrounding the target person 50 in the captured image 600.

The information processing device 300 may place the rectangular information 622 within the captured image 600. For example, the information processing device 300 may estimate the posture of the target person 50 in the captured image 500 and determine the center of gravity of the target person 50 on the ground surface 60 based on the estimated posture of the target person 50. For example, the information processing device 300 may determine the midpoint between the ground contact point of the target person 50's right foot and the ground contact point of the target person 50's left foot as the center of gravity of the target person 50. The method for determining the center of gravity will be described later. The information processing device 300 may determine a rectangular range indicating the range of the target person 50 in the captured image 600 by projectively transforming the determined center of gravity onto the captured image 600 and placing the rectangular information 622 so that the center of gravity is aligned with the center of gravity. The information processing device 300 may determine a rectangular range indicating the range of the target person 50 in the captured image 600 by positioning the rectangular information 622 so that the center point of the rectangular information 622 coincides with the point 513 projected onto the captured image 600.

In Figures 1 to 4, the image capture unit 200 is mainly installed to capture an image of the target person 50 from directly above, but as described above, the image capture unit 200 may also be installed to capture an image of the person from an oblique angle substantially equivalent to directly above. In this case, an annotated image can be generated using a method similar to that described in Figures 1 to 4.

The imaging unit 200 may also be installed so as to capture an image of a person from directly below. In this case, the imaging unit 100 and the imaging unit 200 may be placed under a transparent floor. As a specific example, the imaging unit 100 and the imaging unit 200 are placed under a glass or acrylic floor. Multiple markers are placed on the transparent floor.

The imaging unit 200 is placed in a position where it can image the person from directly below, and captures images from below and upward to capture the target person 50 and the ground surface 60 from directly below. The imaging unit 100 is placed in a position where it can image the target person 50 from a position other than directly below, and captures images of the target person 50 and the ground surface 60. The imaging unit 100 is placed, for example, in a position where it can image the target person 50 from diagonally below, and captures images of the target person 50 and the ground surface 60 from diagonally below.

The information processing device 300 acquires a captured image 500 from the imaging unit 100, in which the imaging unit 100 captures an image of the target person 50 and the ground surface 60. The information processing device 300 acquires a captured image 600 from the imaging unit 200, in which the imaging unit 200 captures an image of the target person 50 and the ground surface 60. The information processing device 300, for example, analyzes the captured image 500 to identify a rectangular area indicating the range of the target person 50 in the captured image 500, identifies a ground surface area from the rectangular area indicating the range of the target person 50 within the ground surface 60 in the captured image 500, and performs a projective transformation of the ground surface area in the captured image 500 onto the captured image 600. The information processing device 300 determines a rectangular area indicating the range of the target person 50 in the captured image 600 based on the ground surface area projectively transformed onto the captured image 600, and assigns metadata representing the target person 50 to the determined rectangular area. The information processing device 300 may identify the ground contact surface range 518, perform projective transformation of the ground contact surface range 518, and determine the size of the rectangular range using methods similar to those described with reference to Figures 2 to 4.

As mentioned above, the imaging unit 200 may be installed so as to capture an image of a person from a diagonal angle substantially equivalent to directly below. In this case, a method similar to that described above can be used.

Figure 5 shows an example of the functional configuration of the information processing device 300. The information processing device 300 includes a memory unit 302, an image acquisition unit 304, a rectangular area identification unit 306, a ground surface area identification unit 308, a projection transformation unit 310, a rectangular area determination unit 312, an annotation unit 314, and a learning execution unit 316. Note that it is not essential for the information processing device 300 to include all of these components.

The image acquisition unit 304 may acquire a captured image 500, which is an image of an object and a ground surface on which the object is grounded, captured by the imaging unit 100. The image acquisition unit 304 stores the acquired captured image 500 in the storage unit 302. The image acquisition unit 304 may acquire a captured image 600, which is an image of an object and a ground surface on which the object is grounded, captured by the imaging unit 200. The image acquisition unit 304 stores the acquired captured image 600 in the storage unit 302. The object may be a person. The object is not limited to a person, and may be any object that is difficult to automatically recognize from directly above but can be automatically recognized from other than directly above.

When the target is a person, the image acquisition unit 304 may acquire a captured image 500 in which the imaging unit 100 captures the person in a manner in which the person's torso, limbs, and head are separated. The image acquisition unit 304 stores the acquired captured image 500 in the storage unit 302. The image acquisition unit 304 may acquire a captured image 600 in which the imaging unit 200 captures the person in a manner in which the person's torso, limbs, and head are not separated. The image acquisition unit 304 stores the acquired captured image 600 in the storage unit 302.

The rectangular area identification unit 306 analyzes the captured image 500 and identifies a rectangular area (sometimes referred to as a first rectangular area) that indicates the range of the object in the captured image 500. The rectangular area identification unit 306 may identify a bounding box 510 that indicates the range of the object in the captured image 500.

The rectangular area identification unit 306 may further identify the center of gravity of the target's ground contact surface in the captured image 500. For example, if the target is a person, the rectangular area identification unit 306 estimates the target's posture in the captured image 500, and based on the estimated target posture, identifies the midpoint between the target's right foot and left foot as the target's center of gravity.

The ground plane range determination unit 308 determines the ground plane range 518, which indicates the range of the object within the ground plane in the captured image 500, based on the bounding box 510. For example, the ground plane range determination unit 308 determines the ground plane range 518, which indicates the range of the object within the ground plane in the captured image 500, using points 511 and 512, which are the two lower vertices of the bounding box 510. For example, the ground plane range determination unit 308 determines point 513 within the bounding box 510, which is the horizontal center of the bounding box 510 and is a position a predetermined percentage of the vertical length of the bounding box 510 from above the bounding box 510; point 514, which is a point-symmetrical position of point 511 with point 513 at its center; and point 515, which is a point-symmetrical position of point 512 with point 513 at its center, thereby determining the quadrangular ground plane range 518 formed by points 511, 512, 514, and 515.

The projection transformation unit 310 uses a plurality of predetermined reference points on the ground surface in the captured image 500 and a plurality of reference points on the ground surface in the captured image 600 to projectively transform the ground surface range 518 in the captured image 500 onto the captured image 600. For example, the projection transformation unit 310 uses a plurality of reference points on the ground surface in the captured image 500 and a plurality of reference points on the ground surface in the captured image 600 to projectively transform points 511, 512, 513, 514, and 515 in the captured image 500 onto the captured image 600.

The projection transformation unit 310 may further projectively transform the center of gravity of the object identified by the rectangular area identification unit 306 into the captured image 600. The projection transformation unit 310 may identify multiple reference points in the captured image 500 and multiple reference points in the captured image 600 by detecting multiple markers placed on the ground surface. As described above, the multiple markers may be tape, such as a tape strip. The multiple markers may be of any type, as long as they are placed in the captured image so that their positions can be identified. It is desirable that the multiple markers be on the same plane as much as possible, and that this is preferably the ground surface. For example, a mark such as a sticker may be placed on the ground surface to serve as a marker. The shapes of the multiple markers may be the same or different from each other.

The rectangular area determination unit 312 determines a rectangular area (sometimes referred to as a second rectangular area) indicating the area of the object in the captured image 600 based on the ground plane area 518 projected into the captured image 600 by the projection transformation unit 310. The rectangular area determination unit 312 may determine the size of the second rectangular area from the ground plane area 518 projected into the captured image 600 by the projection transformation unit 310. For example, the rectangular area determination unit 312 determines the size of the second rectangular area by taking the larger of the difference between the maximum and minimum horizontal values of the coordinates of points 511, 512, 514, and 515 and the difference between the maximum and minimum vertical values as the length of one side. The rectangular area determination unit 312 may determine the center of gravity of the object projected into the captured image 600 by the projection transformation unit 310 as the center point of the second rectangular area. The rectangular area determination unit 312 may set the point 513 projected into the captured image 600 by the projective transformation unit 310 as the center point of the second rectangular area.

The annotation unit 314 assigns metadata describing the subject to the second rectangular area of the captured image 600 determined by the rectangular area determination unit 312.

The learning execution unit 316 performs machine learning using learning data including captured images 600 to which metadata has been added by the annotation unit 314. The learning execution unit 316 uses a plurality of pieces of learning data to generate a learning model that, using a captured image as input, can output that part of the captured image is a person when the captured image contains a person captured from directly above. The learning execution unit 316 stores the generated learning model in the memory unit 302. Note that the information processing device 300 does not necessarily have to be equipped with the learning execution unit 316.

Figure 6 is an explanatory diagram illustrating how the information processing device 300 identifies the center of gravity of a target. The information processing device 300 may estimate the posture of the target and identify the center of gravity of the target on the ground surface based on the estimated posture of the target. Here, the captured image 520, which is an example of the captured image 500, will be used as an example for explanation. The captured image 520 includes the target person 50.

The rectangular area identification unit 306 analyzes the captured image 520 to identify a bounding box 530 that indicates the extent of the target person 50 in the captured image 520. The rectangular area identification unit 306 performs posture estimation of the target person 50. By performing posture estimation, the rectangular area identification unit 306 may estimate the skeleton of the target person 50, as shown in FIG. 6. The rectangular area identification unit 306 may estimate the skeleton of the target person 50, including points for each joint of the target person 50. The rectangular area identification unit 306 may estimate the skeleton 540 of the target person 50, including point 542 indicating the right ankle, point 546 indicating the right knee, point 544 indicating the left ankle, and point 548 indicating the left knee of the target person 50. The rectangular area identification unit 306 may estimate the skeleton of the target person 50 using well-known techniques.

The rectangular area determination unit 306 may determine the adjustment point 554 by identifying the midpoint 552 between points 542 and 544 and adjusting the vertical position of the midpoint 552 based on the shin length 550 calculated from points 542 and 546 or points 544 and 548. For example, the rectangular area determination unit 306 may compare the distance between points 542 and 546 and the distance between points 544 and 548 in the captured image 520 and determine the longer distance as the shin length 550. In the example shown in FIG. 6 , the distance between points 542 and 546 may be determined as the shin length 550. The rectangular area determination unit 306 may determine the adjustment point 554 by, for example, moving the midpoint 552 downward by a predetermined percentage of the determined shin length 550. The predetermined percentage may be set in advance or may be changeable. An example of the predetermined percentage is 30%, but it is not limited to this. The rectangular area identification unit 306 may set the adjustment point 554 as the center of gravity of the target person 50's ground surface. Note that the rectangular area identification unit 306 may also set the midpoint 552 as the center of gravity of the target person 50's ground surface.

When the rectangular area determination unit 306 is unable to determine the ground contact points of both feet of the target person 50 but can determine the positions of both ankles through posture estimation of the target person 50, it may use the adjustment point 554 as the center of gravity of the target person 50's ground contact surface. Because the midpoint 552 is the midpoint of both ankles, it is located upwardly away from the ground contact surface, but adjustment brings the adjustment point 554 closer to the ground contact surface through adjustment. Because multiple reference points are located on the ground contact surface, using the adjustment point 554 provides higher accuracy than using the midpoint 552 when performing projective transformation using multiple reference points. Therefore, using the adjustment point 554 as the center of gravity can improve the accuracy of the projective transformation and increase the degree of agreement between the range of the target person 50 in the captured image 600 and the second rectangular area. If the ground contact points of the target person 50's right foot and left foot can be identified by estimating the target person's pose, the rectangular area identification unit 306 may use the midpoint 552 as the center of gravity of the target person's 50's ground contact surface.

When the rectangular area identification unit 306 identifies the center of gravity of the target person 50 using the method described above, accurate identification may be difficult if the distance between the right foot and left foot of the target person 50 in the captured image 520 is too close. In response to this, the image acquisition unit 304 may select the captured image to be processed from multiple captured images of the target taken continuously by the imaging unit 100. For example, the paired image acquisition unit 304 selects the captured image in which the distance between the right foot and left foot of the target is greatest. As a specific example, the image acquisition unit 304 selects the captured image in which the distance between the right foot and left foot of the target is greatest from the multiple captured images of the target taken continuously by the imaging unit 100. This improves the accuracy with which the rectangular area identification unit 306 identifies the center of gravity of the target person 50.

Figure 7 shows an example of the processing flow of the information processing device 300. Here, the processing flow from when the information processing device 300 acquires the captured image 500 and the captured image 600 to when the information processing device 300 performs annotation on the captured image 600 will be described. The information processing device 300 may perform the processing shown in Figure 7 each time it acquires the captured image 500 and the captured image 600.

In step (sometimes abbreviated as S) 102, the image acquisition unit 304 acquires the captured image 500 and the captured image 600. The image acquisition unit 304 may acquire the captured image 500 from the imaging unit 100 and the captured image 600 from the imaging unit 200.

In S104, the rectangular area identification unit 306 identifies a first rectangular area in the captured image 500 and identifies the center of gravity of the target person 50. In S106, the ground surface area identification unit 308 identifies the ground surface area using the first rectangular area identified in S104.

In S108, the projection transformation unit 310 projectively transforms the installation surface range identified in S106 and the center of gravity identified in S104 into the captured image 600. In S110, the rectangular area determination unit 312 determines a second rectangular area that indicates the range of the target person 50 in the captured image 600. The rectangular area determination unit 312 determines the size of the second rectangular area using the installation surface range projectively transformed onto the captured image 600, and sets the center of gravity projectively transformed onto the captured image 600 as the center point of the second rectangular area.

In S112, the annotation unit 314 assigns metadata indicating a person to the second rectangular area of the captured image 600 determined in S110. Then, the processing ends.

Figure 8 shows a schematic diagram of an example of the hardware configuration of a computer 1200 functioning as the information processing device 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to this embodiment, or to perform operations associated with the apparatus according to this embodiment or one or more "parts," and/or to perform a process according to this embodiment or steps of the process. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input/output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input/output units such as a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program and the like that is executed by computer 1200 upon activation, and/or programs that depend on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided on a computer-readable storage medium such as a DVD-ROM or IC card. The programs are read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, resulting in cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing the operation or processing of information in accordance with the use of computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. CPU 1212 may perform various types of processing on data read from RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the program's instruction sequence, and write the results back to RAM 1214. CPU 1212 may also search for information in files, databases, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The programs or software modules described above may be stored on computer-readable storage media on or near computer 1200. Recording media such as a hard disk or RAM provided within a server system connected to a dedicated communications network or the Internet can also be used as computer-readable storage media, thereby providing the programs to computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device. As a result, a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

The computer-readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or another programmable data processing device processor or programmable circuit, either locally or via a local area network (LAN) or a wide area network (WAN) such as the Internet, so that the processor or programmable circuit of the programmable data processing device, such as a computer, executes the computer-readable instructions to generate means for performing the operations specified in the flowchart or block diagram. Here, the computer may be a personal computer (PC), tablet computer, smartphone, workstation, server computer, general-purpose computer, special-purpose computer, etc., or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a broad definition of computer. In a distributed computing system, multiple computers collectively execute a program by each executing a portion of the program and passing data between computers as needed during program execution.

Examples of processors include computer processors, central processing units, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a portion of a program and passes data between processors as needed during program execution, allowing the multiple processors to collectively execute a program. For example, in multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks for each time slice. In this case, which portion of a program each processor executes changes dynamically. Which portion of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using terms such as "first," "next," etc. for convenience, this does not mean that the processes must be performed in that order.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using terms such as "first," "next," etc. for convenience, this does not mean that the processes must be performed in that order.

10 System, 20 Network, 50 Target Person, 60 Ground Surface, 71, 72, 73, 74 Reference Point, 100 Imaging Unit, 200 Imaging Unit, 300 Information Processing Device, 302 Memory Unit, 304 Image Acquisition Unit, 306 Rectangular Area Identification Unit, 308 Ground Surface Area Identification Unit, 310 Projection Transformation Unit, 312 Rectangular Area Determination Unit, 314 Annotation Unit, 316 Learning Execution Unit, 500 Captured Image, 510 Bounding Box, 511, 512, 513, 514, 515 Points, 518 Ground Surface Area, 520 Captured Image, 530 Bounding Box, 540 Skeleton, 542, 544, 546, 548 Points, 550 Shin Length, 552 Midpoint, 554 Adjustment Point, 600 Captured Image, 612, 614 Difference, 622, 624 Rectangle information, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input/output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input/output chip

Claims (13)

an image acquisition unit that acquires a first captured image of the object and the ground surface by a first imaging unit that images the object and the ground surface from a position other than directly above the object, and a second captured image of the object and the ground surface by a second imaging unit that images the object from directly above;
a rectangular area specifying unit that analyzes the first captured image and specifies a first rectangular area that indicates a range of the object in the first captured image;
a ground surface range specifying unit that specifies a ground surface range indicating a range of the object within the ground surface in the first captured image based on the first rectangular range;
a projection transformation unit that performs projection transformation of the ground contact area in the first captured image into the second captured image using a plurality of predetermined reference points of the ground contact area in the first captured image and the plurality of reference points of the ground contact area in the second captured image;
and a rectangular area determination unit that determines a second rectangular area indicating the area of the object in the second captured image based on the contact surface area projected into the second captured image by the projection transformation unit. 2. The information processing device of claim 1, wherein the image acquisition unit acquires the first captured image of the object and the ground surface by the first imaging unit, which images the object and the ground surface from diagonally above the object, and the second captured image of the object and the ground surface by the second imaging unit, which images the object from directly above the object. The information processing device of claim 2, wherein the target is a person. the ground contact area specification unit specifies a first point and a second point which are two vertices on the lower side of the first rectangular area; a third point within the first rectangular area which is the horizontal center of the first rectangular area specified by the rectangular area specification unit and is located at a length from the top of the first rectangular area that is a predetermined percentage of the vertical length of the first rectangular area; a fourth point which is located at a position that is point-symmetrical to the first point with the third point as the center; and a fifth point which is located at a position that is point-symmetrical to the second point with the third point as the center; and specifies the ground contact area of a rectangle formed by the first point, the second point, the fourth point, and the fifth point;
4. The information processing device of claim 3, wherein the projection transformation unit uses the plurality of reference points of the ground surface in the first captured image and the plurality of reference points in the second captured image to projectively transform the first point, the second point, the third point, the fourth point, and the fifth point in the first captured image into the second captured image. the rectangular area specification unit estimates a posture of the subject in the first captured image, and specifies, based on the estimated posture of the subject, a midpoint between a ground contact point of the subject's right foot and a ground contact point of the subject's left foot as a center of gravity of the subject;
the projection transformation unit further performs projection transformation of the center of gravity of the object into the second captured image;
5. The information processing device according to claim 4, wherein the rectangular range determination unit determines the size of the second rectangular range from the contact surface range projected into the second captured image by the projection transformation unit, and sets the center of gravity of the object projected into the second captured image by the projection transformation unit as the center point of the second rectangular range. The information processing device of claim 5, wherein the rectangular area determination unit determines the size of the second rectangular area by taking the greater of the difference between the maximum and minimum horizontal coordinate values of the first point, the second point, the fourth point, and the fifth point, and the difference between the maximum and minimum vertical coordinate values, as the length of one side. The information processing device of claim 5, wherein the rectangular area identification unit estimates the subject's skeleton including a point indicating the subject's right ankle, a point indicating the right knee, a point indicating the left ankle, and a point indicating the left knee, and identifies the subject's center of gravity by adjusting the vertical position of the midpoint between the point indicating the right ankle and the point indicating the left ankle based on the length of the shin calculated from the point indicating the right ankle and the point indicating the right knee or the point indicating the left ankle and the point indicating the left knee. An information processing device according to any one of claims 2 to 7, wherein the image acquisition unit selects, as the first captured image, an image in which the distance between the right foot and left foot of the subject is greater from among a plurality of captured images of the subject successively captured by the first imaging unit. The information processing device according to claim 1 , further comprising: an annotation unit that assigns metadata representing the target to the second rectangular area determined by the rectangular area determination unit. The information processing device described in any one of claims 1 to 7, wherein the projective transformation unit identifies the multiple reference points in the first captured image and the multiple reference points in the second captured image by detecting multiple markers arranged on the ground surface. A program for causing a computer to function as the information processing device according to any one of claims 1 to 7 when executed by the computer. An information processing device according to any one of claims 1 to 7 ;
the first imaging unit;
The second imaging unit. 1. A computer-implemented information processing method, comprising:
an image acquisition step in which a first image capturing unit captures an image of the object and the ground surface on which the object is in contact from a position other than directly above the object, and acquires a first image of the object and the ground surface, and a second image capturing unit captures an image of the object and the ground surface from directly above the object, and acquires a second image of the object and the ground surface;
a rectangular area specifying step of analyzing the first captured image and specifying a first rectangular area indicating an area of the object in the first captured image;
a ground surface range specifying step of specifying a ground surface range indicating a range of the object within the ground surface in the first captured image based on the first rectangular range;
a projection transformation step of projecting the ground surface range in the first captured image into the second captured image using a plurality of predetermined reference points of the ground surface in the first captured image and the plurality of reference points of the ground surface in the second captured image;
and a rectangular range determination step of determining a second rectangular range indicating the range of the object in the second captured image based on the contact surface range projected into the second captured image in the projective transformation step.