JP7825438B2 - Human movement separation device, human movement separation method and program - Google Patents

Description

This disclosure relates to a human motion separation device, a human motion separation method, and a program.

Patent Document 1 discloses a conventional image display device that has an image memory that receives and temporarily stores image data to be displayed, and a display unit that displays the image data. The image display device is equipped with an imaging unit that captures an image of an object that is in a fixed position relative to the viewer, and a movement vector calculation unit that calculates a movement vector that indicates the amount and direction of relative movement between the display unit and the viewer based on the captured image data, and reduces or eliminates relative movement between the display unit and the viewer based on the movement vector.

Patent Document 2 discloses a conventional camera vector calculation device. This camera vector calculation device automatically extracts feature points from multiple frame images, calculates camera vectors such as highly accurate camera position and camera rotation angle from the multiple frame images, and corrects irregular blurring caused by camera shake in images captured during video shooting based on the calculated camera vectors, thereby generating blur-free images from blurred images.

Japanese Patent Application Laid-Open No. 2004-343622 Japanese Patent Application Laid-Open No. 2005-295495

When performing condition detection processing using captured images of a person's movement within a moving vehicle, the accuracy of condition detection can deteriorate if the person sways due to the swaying of the moving vehicle. However, with the image display device of Patent Document 1, for example, when applied to a moving moving vehicle to capture images of a person riding on it, a fixed object must be present within the moving vehicle near the person. Furthermore, the image display device of Patent Document 1 has the problem of being unable to determine whether the person swayed due to the swaying of the moving vehicle or whether the person moved of their own volition.

Furthermore, when the camera vector calculation device of Patent Document 2 is applied to a moving vehicle to capture images of a person riding on it, for example, it is unable to accurately correct shaking if a moving object is a feature point. Furthermore, the camera vector calculation device of Patent Document 2 also has the problem of being unable to determine whether a person is shaking due to the shaking of the moving vehicle or whether the person is moving of their own volition.

The present disclosure therefore aims to provide a human movement separation device and the like that can accurately separate human movements from movements based on a person's intentions within a moving body.

A human movement separation device according to one aspect of the present disclosure includes an image acquisition unit that acquires an image of a person by capturing an image of a person present on the moving body using an imaging unit fixed to the moving body; a human movement detection unit that detects human movement from the image; a posture detection unit that detects the posture of the person; an environmental fluctuation acquisition unit that acquires environmental fluctuation information that is information indicating environmental fluctuation in the horizontal and vertical directions relative to the person; a human movement estimation unit that estimates a movement amount indicating environmental human movement based on the posture of the person detected by the posture detection unit and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit; and a human movement separation unit that separates the movement amount indicating environmental human movement estimated by the human movement estimation unit from the human movement detected by the human movement detection unit, and calculates human movement with the separated movement amount indicating environmental human movement removed.

A method for separating human movement according to one aspect of the present disclosure includes capturing an image of a person present on a moving body using an imaging unit fixed to the moving body, detecting the movement of the person from the image, detecting the posture of the person, acquiring environmental fluctuation information that is information indicating environmental fluctuations in the horizontal and vertical directions relative to the person, estimating a movement amount indicating environmental fluctuations of the person based on the detected posture of the person and the acquired environmental fluctuation information, separating the estimated movement amount indicating environmental fluctuations of the person from the detected human movement, and calculating the human movement from which the separated movement amount indicating environmental fluctuations of the person has been removed.

Furthermore, a program according to one aspect of the present disclosure is a program for causing a computer to execute a human movement separation method.

Note that some of these specific aspects may be realized using a system, method, integrated circuit, computer program, or computer-readable storage medium such as a CD-ROM, or may be realized using any combination of a system, method, integrated circuit, computer program, and storage medium.

The human movement separation device and the like disclosed herein can accurately separate human movements from movements due to a person's intentions within a moving body.

FIG. 1A (a) is a side view showing a person riding on a moving object and the moving object, and FIG. 1A (b) is a top view showing a person riding on the moving object and the moving object. FIG. 1B(a) is a diagram showing the relationship between the acceleration in the x direction of the moving body and time, and FIG. 1B(b) is a diagram showing the relationship between the acceleration in the y direction of the moving body and time. (a) to (c) of FIG. 1C are diagrams showing images of a person moving when a moving object shakes. FIG. 1D(a) is a diagram showing the amount of movement of a person's head in the x direction, and FIG. 1D(b) is a diagram showing the amount of movement of a person's head in the y direction relative to the center of the person's head. FIG. 1E shows diagrams in which (a) to (c) of FIG. 1E are diagrams showing changes in a person's posture, and (d) of FIG. 1E is a diagram showing angles of the changing posture of the person. FIG. 2 is a diagram showing the shaking of a moving body and the shaking of a person riding on the moving body. FIG. 3 is a diagram showing the relationship between the amount of movement of a moving body in the y direction and time. Figure 4(a) is a diagram showing an example of the movement of a person riding on a moving body, Figure 4(b) is a diagram showing an example of the movement of a person when the horizontal swaying component of the moving body is small, Figure 4(c) is a diagram showing an example of the movement of a person when the moving body sways in both the horizontal and vertical directions, and Figure 4(d) is a diagram showing an example of the movement of a person when the moving body sways while the person is in a more tilted position than in (a). FIG. 5 is a block diagram showing a human motion separation device according to an embodiment. FIG. 6A is a diagram showing the amount of movement of a person's head in the x direction, and FIG. 6B is a diagram showing the amount of movement of a person's head in the y direction. FIG. 7 is a block diagram showing a human movement estimation unit. Figure 8A shows a human movement separation device, where (a) in Figure 8A shows the amount of movement of a person in the x direction, (b) in Figure 8A shows the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of the moving body 3, and (c) in Figure 8A shows the movement of the person after separation. Figure 8B is an example of a case where a person moves, and in a human movement separation device, (a) of Figure 8B is a diagram showing the amount of movement of the person in the x direction, (b) of Figure 8B is a diagram showing the amount of movement in the x direction indicating the swaying of the person due to the environment when not resisting the swaying of the moving body 3, and (c) of Figure 8B is a diagram showing the movement of the person after separation. FIG. 9 is a flowchart showing the operation of the human motion separator. FIG. 10 is a block diagram showing a human movement separating device according to the first modification of the embodiment. FIG. 11 is a block diagram showing a human movement separating device according to a second modification of the embodiment. FIG. 12 is a block diagram showing a human movement separating device according to a third modification of the embodiment.

The embodiments and the like described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims are described as optional components. Furthermore, the contents of each of the embodiments can be combined.

The figures are schematic diagrams and are not necessarily drawn precisely. In each figure, the same components are designated by the same reference numerals. In the following embodiments, expressions such as "approximately identical" are used. For example, "approximately identical" does not only mean a perfect match, but also means a substantial match, i.e., including an error of a few percent. Furthermore, "approximately identical" means a match within the scope in which the effects of the present disclosure can be achieved. The same applies to other expressions using "approximately."

The human movement separation device, human movement separation method, and program disclosed herein are described in detail below with reference to the drawings.

(Embodiment 1)
<Overview>
First, an outline of the human movement separation device, human movement separation method, and program according to the present embodiment will be described.

By acquiring images of people as subjects, the human movement separation device can estimate a person's drowsiness and the subject's level of concentration from the person's facial expression and the amount of movement of the subject. When a moving object carrying a person is moving, the moving object may sway, causing the person to sway along with it, or the person may brace against the swaying of the moving object. Therefore, in order to estimate a person's drowsiness and the subject's level of concentration, it is necessary to accurately estimate the person's movements when the moving object sways, but which are not caused by the swaying of the moving object.

The relationship between the swaying of a moving object and human movement will be explained using Figures 1A to 1E.

(a) of Figure 1A is a side view showing a person riding on a moving object and the moving object, and (b) of Figure 1A is a top view showing the person riding on the moving object and the moving object. (a) of Figure 1B is a diagram showing the relationship between x-direction acceleration of the moving object and time, and (b) of Figure 1B is a diagram showing the relationship between y-direction acceleration of the moving object and time. (a) to (c) of Figure 1C are diagrams showing images of a person moving when shaking occurs in the moving object. (a) of Figure 1C illustrates the center of the person's head in the x and y directions, but this is also true for the other figures. (a) of Figure 1D is a diagram showing the amount of movement of the person's head in the x direction (hereinafter referred to as the amount of movement of the person in the x direction), and (b) of Figure 1D is a diagram showing the amount of movement of the person in the y direction relative to the center of the person's head (hereinafter referred to as the amount of movement of the person in the y direction). The amount of movement in the x direction relative to the center of the head is represented by the amount of movement of pixels in the image. Figure 1E shows (a) to (c) diagrams illustrating changes in a person's posture, and (d) diagrams illustrating the angle of the changing posture of the person. In this embodiment, the amount of movement of the person's head is defined as the amount of movement in the x direction and the y direction relative to the center of the person's head.

First, let's explain the swaying of moving objects and human movement.

In Figure 1A, an image acquisition unit fixed to the moving body captures an image of a person riding on the moving body. For example, when the moving body moves, it sways horizontally (left and right) and vertically (up and down). In Figure 1B (a), for example, if the moving body sways to the right and a force is applied to the left on the human motion separation device, the acceleration sensor of the human motion separation device detects acceleration in the horizontal direction. Also, as shown in Figure 1B (b), for example, if the moving body continues to move downward and a force continues to be applied to the human motion separation device in the upward direction, the acceleration sensor detects acceleration in the vertical direction.

When the moving object sways horizontally, as shown in Figure 1B (a), the person also moves horizontally. As shown in Figure 1C (a) to (c), the image acquisition unit continues to capture images of the person moving.

When a moving object sways horizontally and vertically, the amount of movement of a person in the x and y directions can be seen to change based on the captured images, for example, as shown in (a) and (b) in Figure 1D. In other words, as shown in Figure 1D, as the moving object sways as it moves, it can be seen that the person's head also sways in the x and y directions over time. Furthermore, as shown in (a) to (d) in Figure 1E, by detecting a person's posture from their movement based on the captured images, changes in the angle of their posture can be seen.

Next, we will explain how a moving object and a person sway using Figure 2.

Figure 2 is a diagram showing the shaking of a moving object and the shaking of a person riding on the moving object. Figure 2 illustrates an example of a moving object, a person riding on the moving object, and an imaging unit fixed to the moving object.

As shown in Figure 2, when the moving object is in its standard position, the person's position is upright, aligned with the vertical dashed line. Furthermore, if the moving object suddenly swings vertically upward and tilts suddenly to the left, the person's position will tilt to the right relative to the vertical dashed line. Furthermore, if the moving object suddenly swings vertically downward and tilts suddenly to the right, the person's position will tilt to the left relative to the vertical dashed line.

Next, we will use Figure 3 to explain the shaking of a moving object when it is moving.

Figure 3 shows the relationship between the amount of movement in the y direction (vertical direction) of a moving object and time. In Figure 3, the solid line represents the amount of movement in the y direction, and the dashed line is an approximation curve.

The swaying of a moving object is irregular, influenced by factors such as the road environment, track environment, and wind. If the moving object is a train, the train's swaying includes irregular components caused by bending of the rails, bending of the wheels, sand on the rails, etc., and regular components caused by rail joints.

When a moving object suddenly sways from side to side, as a regular component of shaking, a person's head moves horizontally, i.e., in the x direction, like a pendulum. At this time, a person's head also moves slightly vertically, i.e., in the y direction, like a pendulum. When a moving object suddenly sways from side to side, a person will brace themselves against the shaking of the object to avoid falling over, but it is difficult to accurately determine from the movement of the person riding on the moving object alone whether the shaking is due to the shaking of the object or whether the person's movement is due to their own will to resist the shaking of the object.

Next, we will explain the movement of people riding on a moving vehicle using Figure 4.

Figure 4(a) is a diagram showing an example of the movement of a person riding on a moving object, Figure 4(b) is a diagram showing an example of the movement of a person when the horizontal swaying component of the moving object is small, Figure 4(c) is a diagram showing an example of the movement of a person when the moving object sways in both the horizontal and vertical directions, and Figure 4(d) is a diagram showing an example of the movement of a person when the moving object sways while the person is in a more tilted position than in (a). Furthermore, the swaying of the moving object in Figure 4(c) is a different pattern from that in Figures 4(a) and (b).

_θ1 indicates the acute angle of the straight line V along the person's posture relative to the vertical direction (the angle of the person's posture), _mx indicates the force of shaking the moving body in the horizontal direction, _my indicates the force of shaking the moving body in the vertical direction, _mx1 indicates the component of _mx in the direction perpendicular to the straight line V, and _my1 indicates the component of _myy in the direction perpendicular to the straight line V. _mx1 and _my1 depend on the angle θ of the person's posture.

As shown in Figure 4(a), if a person is tilted to the left at an angle _θ1 relative to the vertical direction when the moving body is swaying, the combined component in the direction perpendicular to the line V of the horizontal swaying force _mx of the moving body and the vertical swaying force _my of the moving body is _mx1 + _myy1 .

As shown in Figure 4(b), when the shaking caused by the moving object is small, even if a person leans to the left by an angle _θ1 with respect to the vertical direction, the shaking force _mx is smaller than in the case of Figure 4(a), and therefore the combined component _mx1 + _myl in Figure 4(b) is smaller than the combined component in Figure 4(a).

As shown in Figure 4(c), when a moving body sways both horizontally and vertically, if a person intentionally tilts to the left at an angle θ ₁ from the vertical, the combined component becomes m _x1 - _{m y1} = 0, and the force applied to the person by the swaying of the moving body is canceled out.

As shown in (d) of Figure 4, when a person is tilted to the left due to movement, if the moving body sways and the person tilts to the left by an angle _θ2 with respect to the vertical direction, the composite component m _x1 +m _y1 in (d) of Figure 4 will be larger than in (a), (b), and (c) of Figure 4.

As mentioned above, human movement can be mediated by the environment, which is caused by the swaying of the moving body, and by the person's own will. In this embodiment, the amount of head movement is calculated after removing the swaying of the environment, which is caused by the swaying of the moving body, from the person's movement within the moving body, and only the person's own will. In other words, the amount of head movement is calculated after separating the swaying caused by the person's own will.

<Configuration and Functions>
The configurations and functions of the human movement separation device 1, the human movement separation method, and the program according to this embodiment will be described.

Figure 5 is a block diagram showing the human motion separation device 1 according to an embodiment.

As shown in FIG. 5, the human movement separation device 1 includes an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, an environmental fluctuation acquisition unit 24, a human movement estimation unit 23, and a human movement separation unit 26.

[Image acquisition unit 21]
The image acquisition unit 21 acquires an image of a person by capturing an image of a person present on the moving body 3 using the imaging unit 10 fixed to the moving body 3. The image acquisition unit 21 can acquire an image by having the imaging unit 10 fixed to the moving body 3 continuously capture an image of a predetermined location. The image acquisition unit 21 may be mounted on the imaging unit 10. Here, the moving body 3 is a vehicle such as an automobile or train, an airplane, a ship, etc. Furthermore, the image includes at least a person's head, a person's torso, etc., as shown in FIG. 1C , for example. The image acquisition unit 21 outputs the acquired image to the human movement detection unit 25 and the posture detection unit 22.

[Human movement detection unit 25]
The human movement detection unit 25 acquires images from the image acquisition unit 21 and detects human movement from the acquired images. For example, the human movement detection unit 25 extracts a person's head for each frame of the image acquired from the image acquisition unit 21 and calculates the amount of movement of the extracted person's head.

Figure 6(a) is a diagram showing the amount of movement of a person's head in the x direction, and Figure 6(b) is a diagram showing the amount of movement of a person's head in the y direction. The amount of movement of a person's head is calculated as the amount of change in pixel units in the image, as shown in Figures 6(a) and 6(b). As shown in Figure 5, the human movement detection unit 25 outputs the calculated amount of movement of the person's head, which is the movement of the person, to the human movement separation unit 26.

[Attitude detection unit 22]
The posture detection unit 22 detects a person's posture from the image. The person's posture is indicated by the angle at which the person tilts relative to the vertical direction over time. For example, the posture detection unit 22 extracts the person's head and torso from each frame of the image acquired by the image acquisition unit 21 and calculates the tilt of the extracted person, i.e., the angle of the person's posture. Specifically, as shown in (a) to (c) of FIG. 1E, the posture detection unit 22 calculates the angle θ (θ ₁ , θ ₂ , θ ₃ , ...) of the posture direction along the head and torso relative to the vertical direction. (a) to (c) of FIG. 1E show how the person's posture changes over time. As the person sways in this way, the angle θ of the person's posture changes as shown in (d) of FIG. 1E. In this way, as shown in FIG. 5, the posture detection unit 22 detects the person's tilt, i.e., the person's posture, from the image. The posture detection unit 22 outputs the detected angle, i.e., the person's posture, to the human sway estimation unit 23.

[Environmental fluctuation acquisition unit 24]
The environmental fluctuation acquisition unit 24 acquires environmental fluctuation information, which is information indicating fluctuations due to the environment including horizontal and vertical directions relative to the person. Specifically, the environmental fluctuation acquisition unit 24 is disposed in the moving body 3 together with the image acquisition unit 21, and acquires the environmental fluctuation information, which is information indicating the fluctuation of the moving body 3 in the horizontal direction and the fluctuation of the moving body 3 in the vertical direction, in accordance with the fluctuation of the moving body 3. The environmental fluctuation acquisition unit 24 is, for example, an acceleration sensor, a vibration meter, etc. The environmental fluctuation acquisition unit 24 outputs the acquired environmental fluctuation information to the human fluctuation estimation unit 23.

[Human sway estimation unit 23]
The human sway estimation unit 23 estimates a movement amount indicating the swaying of the person due to the environment based on the posture of the person detected by the posture detection unit 22 and the environmental sway information acquired by the environmental sway acquisition unit 24. That is, the human sway estimation unit 23 estimates a movement amount indicating the swaying of the person due to the environment, which is caused by the swaying of the moving body 3, based on the posture of the person and the swaying of the moving body 3 in the horizontal direction and the vertical direction indicated in the environmental sway information. Specifically, as shown in FIG. 4 , the human sway estimation unit 23 estimates a movement amount indicating the swaying of the person due to the environment in FIG. 5 by calculating, over time, a composite component obtained by converting the horizontal sway component of the moving body 3 based on the environmental sway information into a direction perpendicular to the posture direction of the person according to the angle, and a component obtained by converting the vertical sway component of the moving body 3 based on the environmental sway information into a direction perpendicular to the posture direction of the person according to the angle. The movement amount indicating the swaying of the person due to the environment is represented by the movement amount of the person in the x direction due to the environment and the movement amount of the person in the y direction due to the environment. The human movement estimation unit 23 outputs the amount of movement indicating the human movement due to the environment to the human movement separation unit 26 .

Here, the processing of the human movement estimation unit 23 will be explained in more detail using Figure 7.

Figure 7 is a block diagram showing the human movement estimation unit 23.

The human sway estimation unit 23 has an angle acquisition unit 33, an x-direction acceleration acquisition unit 31, a y-direction acceleration acquisition unit 32, an estimation unit 34, and an output unit 35.

First, the angle acquisition unit 33 acquires an angle indicating the person's posture from the posture detection unit 22. The angle acquisition unit 33 outputs the acquired angle of the person's posture to the estimation unit 34. Note that if the posture detection unit 22 does not calculate the angle of the person's posture, the angle acquisition unit 33 may calculate the angle of the person's posture based on the person's posture.

Furthermore, the x-direction acceleration acquisition unit 31 acquires the acceleration of the moving body 3 in the x-direction indicated in the environmental fluctuation information from the environmental fluctuation acquisition unit 24. For example, the acceleration of the moving body 3 in the x-direction is shown in (a) of Figure 1B. The x-direction acceleration acquisition unit 31 outputs the acquired acceleration of the moving body 3 in the x-direction to the estimation unit 34.

The y-direction acceleration acquisition unit 32 acquires the acceleration of the moving body 3 in the y direction indicated in the environmental fluctuation information from the environmental fluctuation acquisition unit 24. For example, the acceleration of the moving body 3 in the y direction is shown in (b) of Figure 1. The y-direction acceleration acquisition unit 32 outputs the acquired acceleration of the moving body 3 in the y direction to the estimation unit 34.

The estimation unit 34 acquires the angle of the person's posture from the angle acquisition unit 33, acquires the acceleration of the moving body 3 in the x direction from the x direction acceleration acquisition unit 31, and acquires the acceleration of the moving body 3 in the y direction from the y direction acceleration acquisition unit 32. Based on the acquired angle of the person's posture, the acceleration of the moving body 3 in the x direction, and the acceleration of the moving body 3 in the y direction, the estimation unit 34 estimates the amount of movement indicating the swaying of the person due to the environment using a pendulum damping formula.

The pendulum damping formula is shown in equations (1) to (3) below.

ax is the observed acceleration in the horizontal component.

ay is the observed acceleration in the vertical component.

θ' is the angle of inclination of the person in the previous frame in which acceleration in the horizontal or vertical component was observed.

θ(s) is the person's tilt angle in the previous frame.

A1 and A2 are coefficients determined by calibration. Calibration is performed before the estimation process is carried out. A1 and A2 are calculated from the results (horizontal or vertical component) of detecting human movement when strong shaking occurs and the person is not moving of their own volition.

t is the time elapsed since the horizontal or vertical acceleration was observed. When the horizontal or vertical acceleration is observed, t is reset to 0.

As a result, the estimation unit 34 estimates the force exerted on the person by the swaying of the moving body 3, taking into account the person's posture, and thereby estimates the amount of movement indicating the swaying of the person due to the current environment described above.

The output unit 35 outputs the amount of movement estimated by the estimation unit 34, which indicates the swaying of the person due to the environment, to the person sway separation unit 26.

The human movement estimation unit 23 may store the amount of movement indicating the human movement due to the environment in the storage unit. In this case, the amount of movement indicating the human movement due to the environment may be stored in association with environmental movement information. The human movement estimation unit 23 may estimate the amount of movement indicating the human movement due to the environment based on the past amount of movement indicating the human movement due to the environment, the human posture, and the environmental movement information.

[Person movement separation unit 26]
The human movement separation unit 26 separates the human movement due to the environment estimated by the human movement estimation unit 23 from the human movement detected by the human movement detection unit 25, and calculates the human movement with the separated human movement due to the environment removed. Specifically, the human movement separation unit 26 acquires the amount of movement of the person's head in Fig. 5, which is the human movement output by the human movement detection unit 25, and the amount of movement indicating the human movement due to the environment in Fig. 5, which is output by the human movement estimation unit 23. Then, the human movement separation unit 26 calculates the separated head movement amount, which is the amount of head movement due to the person's own will in Fig. 5, obtained by removing the amount of movement indicating the human movement due to the environment from the amount of head movement.

Here, we will explain how to calculate the amount of head movement after voluntary separation using Figures 8A and 8B.

Figure 8A shows the amount of movement of a person in the x direction in the human movement separation device 1, (a) of Figure 8A shows the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of the moving object 3, and (c) of Figure 8A shows the movement of the person after separation. Figure 8B shows an example of a case where a person moves, in the human movement separation device 1, (a) of Figure 8B shows the amount of movement in the x direction of a person, (b) of Figure 8B shows the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of the moving object 3, and (c) of Figure 8B shows the movement of the person after separation.

When the moving body 3 sways in the x direction, the actual amount of movement of the person in the x direction (amount of head movement) is as shown in Figure 8A (a). At this time, because the person riding the moving body 3 does not resist the swaying of the moving body 3, the amount of movement indicating the swaying of the person due to the environment (amount of head movement due to the environment) is as shown in Figure 8A (b). The person sway separation unit 26 separates the swaying of the person due to the environment shown in Figure 8A (b) from the amount of movement of the person in the x direction shown in Figure 8A (a), and calculates the movement of the person with the separated swaying of the person due to the environment removed. In other words, the difference between the amount of movement in Figure 8A (a) and the amount of movement in Figure 8A (b) is as shown in Figure 8A (c). In other words, because the person does not resist the swaying of the moving body 3, the amount of head movement after separation, which is the movement of the person, is 0, as shown in Figure 8A (c).

Next, if a person moves when the moving object 3 sways in the x direction, the person's actual movement amount in the x direction (head movement amount) will be as shown in Figure 8B (a). At this time, the movement amount (head movement amount due to the environment) indicating the person's movement due to the environment when not resisting the movement of the moving object 3 will be as shown in Figure 8B (b). The person movement separation unit 26 separates the person's movement due to the environment shown in Figure 8B (b) from the person's movement amount in the x direction shown in Figure 8B (a), and calculates the person's movement with the separated person movement due to the environment removed. In other words, due to the person's movement of their own volition, the difference between the movement amount in Figure 8B (a) and the movement amount in Figure 8B (b) will be as shown in Figure 8B (c). In other words, because the person moves of their own volition, the separated head movement amount, which is the person's movement, is calculated as shown in Figure 8B (c).

The human movement separation unit 26 outputs the calculated amount of head movement after separation due to the person's own will to the state detection unit 27.

[Status detection unit 27]
A state detection unit 27 is communicatively connected to the human movement separation device 1. When the state detection unit 27 acquires the amount of head movement after separation due to the person's will from the human movement separation unit 26, it detects the person's state, such as drowsiness and concentration, based on the amount of head movement after separation due to the person's will. The state detection unit 27 outputs the state detection result to an external device. Note that the state detection unit 27 may be included as a component of the human movement separation device 1.

<Processing Operation>
The processing operation of the human movement separation device 1 of this embodiment will be described.

Figure 9 is a flowchart showing the operation of the human movement separation device 1.

As shown in FIG. 9, first, the image acquisition unit 21 sequentially acquires images of a person captured by the imaging unit 10 at a predetermined number of frames per second (S11). The image acquisition unit 21 outputs the acquired images to the human movement detection unit 25 and the posture detection unit 22.

The environmental fluctuation acquisition unit 24 also acquires environmental fluctuation information, which is information indicating fluctuations caused by the environment in the horizontal and vertical directions relative to the person (S12). The environmental fluctuation acquisition unit 24 outputs the environmental fluctuation information to the human fluctuation estimation unit 23. The processing operations of steps S11 and S12 may be reversed or may be performed simultaneously.

Next, the human movement detection unit 25 detects the amount of movement of the person's head by calculating the amount of movement of the person in the x direction and the amount of movement of the person in the y direction from the acquired image (S13). The human movement detection unit 25 outputs the amount of movement of the person's head detected as human movement to the human movement separation unit 26.

Next, the posture detection unit 22 extracts the person's head and torso from the image acquired by the image acquisition unit 21, and calculates the angle of the posture of the extracted person. Specifically, as shown in (a) to (c) of FIG. 1E, the posture detection unit 22 calculates the tilt of the posture direction along the head and torso with respect to the vertical direction, that is, the angle θ of the posture direction with respect to the vertical direction. In other words, the posture detection unit 22 detects the posture of the person by calculating the angle of the person's posture (S14). The posture detection unit 22 outputs the detected posture of the person to the human movement estimation unit 23.

Next, the human movement estimation unit 23 estimates the amount of movement indicating the human movement due to the environment, which is caused by the movement of the moving body 3, based on the posture of the person and the movement of the moving body 3 in the horizontal direction and the vertical direction indicated in the environmental movement information (S15). The human movement estimation unit 23 outputs the amount of movement indicating the human movement due to the environment to the human movement separation unit 26.

Next, the human movement separation unit 26 acquires the amount of movement of the person's head, which is the human movement output by the human movement detection unit 25 in Figure 5, and the amount of movement indicating the human movement due to the environment, which is the human movement due to the environment output by the human movement estimation unit 23. The human movement separation unit 26 then calculates the amount of head movement after separation due to the person's own will, by subtracting the amount of movement indicating the human movement due to the environment from the amount of head movement (S16). The human movement separation unit 26 outputs the calculated amount of head movement after separation due to the person's own will to the state detection unit 27. The human movement separation device 1 then ends the processing operation of Figure 9.

<Action and effect>
The effects of the human movement separation device 1, the human movement separation method, and the program according to this embodiment will be described.

As described above, the human movement separation device 1 according to this embodiment includes an image acquisition unit 21 that acquires an image of a person by capturing an image of a person present on the moving body 3 using an imaging unit 10 fixed to the moving body 3; a human movement detection unit 25 that detects human movement from the image; a posture detection unit 22 that detects the posture of the person; an environmental fluctuation acquisition unit 24 that acquires environmental fluctuation information that is information indicating environmental fluctuation in the horizontal and vertical directions relative to the person; a human movement estimation unit 23 that estimates a movement amount indicating environmental human movement (amount of head movement) based on the posture of the person detected by the posture detection unit 22 and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit 24; and a human movement separation unit 26 that separates the movement amount indicating environmental human movement estimated by the human movement estimation unit 23 from the human movement detected by the human movement detection unit 25, and calculates human movement with the separated movement amount indicating environmental human movement removed.

This allows the human movement estimation unit 23 to estimate the amount of movement indicating the human's movement due to the environment based on the person's posture and environmental movement information. In other words, it is possible to estimate the amount of movement indicating the human's movement due to the environment by taking into account not only the environmental movement but also the person's posture. Furthermore, the human movement separation unit 26 can remove the amount of movement indicating the human's movement due to the environment from the human movement. This makes it possible to obtain the person's movement due to their own will, that is, the amount of head movement after separation.

Therefore, the human movement separation device 1 can accurately separate intentional human movements from actual human movements within the moving object 3.

The human movement separation method according to this embodiment also includes capturing an image of a person present on the moving body 3 using an imaging unit 10 fixed to the moving body 3, thereby obtaining an image of the person; detecting the movement of the person from the image; detecting the posture of the person; obtaining environmental fluctuation information that indicates environmental fluctuations in the horizontal and vertical directions relative to the person; estimating the amount of movement indicating the environmental fluctuation of the person based on the detected posture of the person and the obtained environmental fluctuation information; separating the estimated amount of movement indicating the environmental fluctuation of the person from the detected human movement; and calculating the human movement with the separated amount of movement indicating the environmental fluctuation of the person removed.

This method also achieves the same effects as described above.

Furthermore, the program according to this embodiment is a program for causing a computer to execute a human movement separation method.

This program also has the same effects as those described above.

Furthermore, in the human motion separation device 1 according to this embodiment, the posture detection unit 22 detects the posture of a person from an image.

This allows for accurate detection of a person's posture by using captured images of the person.

Furthermore, in the human motion separation device 1 according to this embodiment, the human posture is the angle at which the human leans relative to the vertical direction over time.

This allows a person's posture to be quantitatively calculated as an angle. This can therefore be used by the human movement estimation unit 23 to estimate the amount of movement that indicates the human movement due to the environment.

In addition, in the human movement separation device 1 according to this embodiment, the human movement estimation unit 23 estimates the amount of movement indicating the human movement due to the environment by calculating, over time, a composite component of a component obtained by converting the horizontal movement component of the moving body 3 based on the environmental movement information into a direction perpendicular to the direction of the person's posture according to the angle, and a component obtained by converting the vertical movement component of the moving body 3 based on the environmental movement information into a direction perpendicular to the direction of the person's posture according to the angle.

This makes it possible to estimate the amount of movement that indicates a person's sway due to the environment, taking into account not only the sway caused by the environment, but also the angle of the person's posture. This makes it possible to accurately estimate the amount of movement that indicates a person's sway due to the environment.

(First Modification of the Embodiment)
The human motion separation device 1a of this modified example differs from the human motion separation device of the embodiment in that the posture detection unit 22 can detect the posture of a person without acquiring an image from the image acquisition unit 21. In this modified example, the same components and functions as those of the human motion separation device of the embodiment are assigned the same reference numerals, and detailed descriptions of these components and functions will be omitted.

Figure 10 is a block diagram showing a human movement separation device 1a according to variant 1 of the embodiment.

As shown in Figure 10, in this modified example, the image acquisition unit 21 outputs the acquired image only to the human movement detection unit 25.

The posture detection unit 22 is a camera sensor that can detect a person's posture by capturing an image of the person. The posture detection unit 22 detects the posture of the person based on the captured image of the person, and outputs the detected posture of the person to the human sway estimation unit 23.

This modified example also achieves the same effects as the above-described embodiment.

(Modification 2 of the embodiment)
The human motion separation device 1b of this modified example differs from the human motion separation device of the embodiment in that it includes a first environmental fluctuation acquisition unit 24a and a second environmental fluctuation acquisition unit 24b. In this modified example, the same components and functions as those of the human motion separation device of the embodiment are designated by the same reference numerals, and detailed descriptions of these components and functions will be omitted.

Figure 11 is a block diagram showing a human movement separation device 1b according to a second variation of the embodiment.

As shown in FIG. 11, the human movement separation device 1b in this modified example includes an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, a human movement estimation unit 23, and a human movement separation unit 26, as well as multiple environmental fluctuation acquisition units. The multiple environmental fluctuation acquisition units of the human movement separation device 1b in this modified example include a first environmental fluctuation acquisition unit 24a and a second environmental fluctuation acquisition unit 24b.

The first environmental fluctuation acquisition unit 24a acquires first environmental fluctuation information, which is information indicating fluctuations due to the environment in the up-down direction (an example of the vertical direction) and the left-right direction (an example of the horizontal direction). The first environmental fluctuation acquisition unit 24a outputs the acquired first environmental fluctuation information to the human fluctuation estimation unit 23. The first environmental fluctuation acquisition unit 24a is an example of an environmental fluctuation acquisition unit. The first environmental fluctuation information is an example of environmental fluctuation information.

The second environmental fluctuation acquisition unit 24b acquires second environmental fluctuation information, which is information indicating fluctuations due to the environment in the forward/backward direction (an example of the horizontal direction) of the moving body 3. The second environmental fluctuation acquisition unit 24b outputs the acquired second environmental fluctuation information to the human fluctuation estimation unit 23. The second environmental fluctuation acquisition unit 24b is an example of an environmental fluctuation acquisition unit. The second environmental fluctuation information is an example of environmental fluctuation information.

In this case, the image acquired by the image acquisition unit 21 may be a three-dimensional image. Furthermore, the human movement detection unit 25 may be able to calculate the amount of movement of the person's head in the up-down, left-right, and front-to-back directions. Furthermore, the posture detection unit 22 may be able to detect the posture of the person in the left-right, and front-to-back directions relative to the up-down direction.

The first environmental fluctuation acquisition unit 24a and the second environmental fluctuation acquisition unit 24b are, for example, an acceleration sensor, a vibration meter, etc.

The human fluctuation estimation unit 23 estimates the amount of movement indicating the human's fluctuation due to the environment based on the human posture detected by the posture detection unit 22, the first environmental fluctuation information acquired by the first environmental fluctuation acquisition unit 24a, and the second environmental fluctuation information acquired by the second environmental fluctuation acquisition unit 24b.

The human movement separation device 1b according to this modified example is equipped with multiple environmental fluctuation acquisition units (first environmental fluctuation acquisition unit 24a and second environmental fluctuation acquisition unit 24b).

This allows the human movement estimation unit 23 to acquire not only environmentally induced movement in the up-down and left-right directions, but also environmentally induced movement in the forward-backward direction. This allows for more accurate estimation of the amount of movement indicating the human movement due to the environment.

This modified example also achieves the same effects as the above-described embodiment.

(Modification 3 of the embodiment)
The human motion separation device 1c of this modification differs from the human motion separation device of the embodiment in that it includes a posture information acquisition unit 24c. In this modification, the same components and functions as those of the human motion separation device of the embodiment are designated by the same reference numerals, and detailed descriptions of these components and functions will be omitted.

Figure 12 is a block diagram showing a human movement separation device 1c according to variant 3 of the embodiment.

As shown in FIG. 12, the human movement separation device 1c includes an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, a human movement estimation unit 23, an environmental movement acquisition unit 24, and a human movement separation unit 26, as well as a posture information acquisition unit 24c.

In this modified example, the posture information acquisition unit 24c is worn by a person and acquires posture information that indicates the person's posture. The posture information acquisition unit 24c is, for example, an IMU (Inertial Measurement Unit) sensor that includes an accelerometer, a gyroscope, a geomagnetometer, etc. The posture information acquisition unit 24c outputs the acquired posture information of the person to the posture detection unit 22.

When the posture detection unit 22 acquires posture information from the posture information acquisition unit 24c, it detects the posture of the person based on the posture information.

The human motion separation device 1c according to this modified example further includes a posture information acquisition unit 24c that acquires posture information indicating the posture of a person. The posture detection unit 22 then detects the posture of the person based on the posture information.

For example, if the posture information acquisition unit 24c is worn by a person, the person's posture can be detected easily and accurately.

This modified example also achieves the same effects as the above-described embodiment.

(Other Modifications)
The human motion separation device, human motion separation method, and program according to the present disclosure have been described above based on the above-mentioned embodiments, but the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications that come to mind by those skilled in the art may also be included in the scope of the present disclosure.

For example, in the human movement separation device, human movement separation method, and program according to the above-described embodiments, human movement is not limited to the amount of movement of the person's head. Human movement may also be the amount of movement of the eyes, nose, mouth, etc. on the person's head, or the amount of movement of the person's torso, neck, etc. other than the head.

Furthermore, each component of the human motion separation device, human motion separation method, and program according to the above embodiments may be configured with dedicated hardware, or may be realized by executing a software program appropriate for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program stored on a storage medium such as a hard disk or semiconductor memory.

Furthermore, all numbers used above are examples used to specifically explain this disclosure, and embodiments of this disclosure are not limited to the numbers shown as examples.

Furthermore, the division of functional blocks in the block diagram is one example; multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to other functional blocks. Furthermore, the functions of multiple functional blocks with similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

Furthermore, the order in which each step in the flowchart is performed is merely an example to specifically explain the present disclosure, and orders other than those described above may also be used. Furthermore, some of the steps may be performed simultaneously (in parallel) with other steps.

This disclosure also includes forms obtained by applying various modifications to the above-described embodiments that would occur to those skilled in the art, as well as forms realized by arbitrarily combining the components and functions of the embodiments within the scope of the spirit of this disclosure.

The human movement separation device disclosed herein can be used, for example, as a device for calculating the movement of a person riding on a moving object.

1, 1a, 1b, 1c Human motion separation device 3 Moving object 10 Imaging unit 21 Image acquisition unit 22 Posture detection unit 23 Human motion estimation unit 24 Environmental motion acquisition unit 24a First environmental motion acquisition unit (environmental motion acquisition unit)
24b Second environmental fluctuation acquisition unit (environmental fluctuation acquisition unit)
24c Posture information acquisition unit 25 Human movement detection unit 26 Human movement separation unit

Claims (8)

an image acquisition unit that acquires an image of a person by capturing an image of a person present on the moving body using an imaging unit fixed to the moving body;
a human movement detection unit that detects human movement from the image;
a posture detection unit that detects the posture of a person;
an environmental fluctuation acquisition unit that acquires environmental fluctuation information, which is information indicating fluctuations caused by the environment in the horizontal and vertical directions relative to a person;
a human fluctuation estimation unit that estimates a movement amount indicating the human fluctuation due to the environment based on the human posture detected by the posture detection unit and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit;
and a human movement separation unit that separates the amount of movement indicating human movement due to the environment estimated by the human movement estimation unit from the human movement detected by the human movement detection unit, and calculates the human movement from which the separated amount of movement indicating human movement due to the environment has been removed. The human motion separation device according to claim 1 , wherein the posture detection unit detects a posture of the person from the image. The human movement separating device according to claim 1 or 2, comprising a plurality of the environmental fluctuation acquisition units that acquire environmental fluctuation information in different directions . The human motion separation device according to any one of claims 1 to 3, further comprising a posture information acquisition unit that acquires posture information that is information indicating a posture of a person, and the posture detection unit detects the posture of the person based on the posture information. The human motion separation device according to any one of claims 1 to 4, wherein the human posture is an angle at which the human leans relative to the vertical direction over time. The human movement separation device according to claim 5, wherein the human movement estimation unit estimates a movement amount indicating the human movement due to the environment by calculating, over time, a composite component of a component obtained by converting a horizontal movement component of the moving body based on the environmental movement information into a direction perpendicular to the posture direction of the person according to the angle, and a component obtained by converting a vertical movement component of the moving body based on the environmental movement information into a direction perpendicular to the posture direction of the person according to the angle. capturing an image of a person present on the moving body by an imaging unit fixed to the moving body; and
detecting human movement from the image;
Detecting a person's posture;
Acquiring environmental vibration information, which is information indicating vibrations caused by the environment in the horizontal and vertical directions relative to the person;
estimating a movement amount indicating the shaking of the person due to the environment based on the detected posture of the person and the acquired environmental shaking information;
separating an amount of movement indicating the swaying of the person due to the environment estimated from the detected human movement, and calculating the human movement from which the separated amount of movement indicating the swaying of the person due to the environment has been removed. A program for causing a computer to execute the human movement segmentation method according to claim 7.