JP4886575B2 - Attitude detection apparatus, attitude detection method, program, and information storage medium - Google Patents

Description

  The present invention relates to a posture detection device, a posture detection method, a program, and an information storage medium that detect a posture of a user based on an image obtained by imaging a user.

  2. Description of the Related Art Conventionally, there is a motion capture technique for recording postures and movements of people and animals on a computer as digital data. In such a technique, for example, a reflective marker, an infrared marker, or the like is attached to a predetermined portion of a human body, and the person is imaged by an imaging means such as a video camera. Then, by analyzing the captured image and detecting the position of the marker in the image, the posture and motion of the person can be specified. There is also a technique for specifying the posture of a human or animal body by analyzing a captured image without using a marker, thereby specifying the posture of the human or animal body.

  In the conventional technique, a marker or a part of the body may be hidden depending on the orientation of a person or animal with respect to the camera. In such a case, the posture may not be specified with high accuracy.

  The present invention has been made in view of the above circumstances, and one of its purposes is an attitude detection device, an attitude detection method, a program, and an information storage capable of improving the accuracy of attitude detection when detecting the attitude of a user. To provide a medium.

  The posture detection apparatus according to the present invention includes an imaging unit that captures an image including a user, a contact detection member that detects that any part of the body of the user is in contact, and the user based on the captured image. A posture specifying unit that limits a range of the specified posture according to information indicating whether or not the user is in contact with the contact detection member at the time of capturing the image. It is characterized by including these.

  Further, in the posture detection device, there are a plurality of the contact detection members, and each of the contact detection members is arranged in a predetermined positional relationship, and the posture identification unit is configured to connect the user to each of the plurality of contact detection members when the image is captured. It is good also as restrict | limiting the range of the said specific attitude | position according to the information which shows whether it is contacting.

  Further, the plurality of contact detection members may be attached to a sheet member disposed on the floor and detect that each of the contact detection members is stepped on by the user.

  In addition, the posture detection device further includes a motion data generation unit that generates motion data indicating the user's motion based on information indicating the user's posture specified by the posture specifying unit for each of a plurality of time points. It may be included.

  The posture detection device further includes a game process execution unit that executes a predetermined game process using the detection result of the contact detection member as an operation input of the user, and the motion data generated by the motion data generation unit However, it may be used for the output of the game process.

  In addition, the posture detection method according to the present invention includes a step of acquiring an image including a user imaged by an imaging unit, and a detection result by a contact detection member that detects that any part of the user's body is in contact. A step of acquiring and a step of specifying the posture of the user based on the acquired image, according to information indicating whether or not the user is in contact with the contact detection member at the time of capturing the image. And limiting the range of the specified posture.

  Further, the program according to the present invention is a means for acquiring an image including a user imaged by an imaging means, and a means for acquiring a detection result by a contact detection member for detecting that any part of the user's body is in contact. And means for specifying the posture of the user based on the acquired image, according to information indicating whether the user is in contact with the contact detection member at the time of capturing the image, This is a program for causing a computer to function as posture identifying means for limiting the range of postures to be identified. This program may be stored in a computer-readable information storage medium.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic diagram showing an overall outline of a posture detection apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the posture detection apparatus 1 includes a computer 11, an operation device 12, an agent device 13, and a display device 14. The operation device 12 and the agent device 13 are connected to the computer 11 via a wired or wireless communication interface.

  The computer 11 is, for example, a home game machine or a personal computer, and includes a control unit 21, a storage unit 22, a display control unit 23, and an interface unit 24 as shown in the block diagram of FIG. Consists of including.

  The control unit 21 is a CPU or the like and operates according to a program stored in the storage unit 22. In the present embodiment, the control unit 21 specifies the posture of the user based on information input from the operation device 12 or the agent device 13. A specific example of the process executed by the control unit 21 will be described later.

  The storage unit 22 includes a memory element such as a RAM and a ROM, and a computer-readable information storage medium such as a disk device such as a hard disk. The storage unit 22 stores a program executed by the control unit 21. The storage unit 22 also operates as a work memory for the control unit 21.

  The display control unit 23 converts the image information output from the control unit 21 into a video signal and outputs the video signal to the display device 14. The display device 14 is a home television, for example, and displays a video on the screen based on a video signal output from the display control unit 23.

  The interface unit 24 is, for example, a wireless communication unit based on the Bluetooth (trademark) standard. The interface unit 24 relays data transmission / reception between the control unit 21 and the operation device 12 and the agent device 13.

  The operation device 12 receives an instruction operation from the user U using the posture detection apparatus 1 and outputs a control signal indicating the content to the computer 11. In the present embodiment, the operation device 12 includes a plurality of contact detection members 15. Each of these contact detection members 15 includes a sensor such as a pressure sensor that detects that any part of the body of the user U is in contact. Each contact detection member 15 outputs a signal corresponding to the detection result to the computer 11 as a user U operation input at regular intervals.

  As a specific example, the operation device 12 in the present embodiment is used for a game in which the user U moves and moves, and is configured by a sheet member to which a plurality of contact detection members 15 are attached. The operation device 12 is used by being placed on the floor. Here, each contact detection member 15 is disposed on the sheet member in a predetermined positional relationship, and detects that the user U has stepped on the contact detection member 15. The user U plays the game by stepping on each contact detection member 15 at a predetermined timing in accordance with a game instruction on the operation device 12. FIG. 3 is a diagram illustrating an example of such an operation device 12. In the example of FIG. 3, four contact detection members 15a, 15b, 15c and 15d are attached to the sheet member.

  The agent device 13 is a robot system that operates autonomously based on a program built in advance and information from the computer 11. For example, the agent device 13 has a shape imitating a human shape, and includes a movable part corresponding to a human hand or foot. Further, the agent device 13 includes an imaging unit 16.

  The imaging unit 16 is, for example, a small video camera or the like, and captures an image including the user U. The captured image is output to the control unit 21 of the computer 11 via the interface unit 24. The agent device 13 moves to a position where the user U can be imaged based on an instruction from the computer 11, and images the user U by the imaging unit 16.

  Hereinafter, functions realized by the posture detection device 1 will be described. The posture detection device 1 functionally includes a game process execution unit 31, a posture identification unit 32, and a motion data generation unit 33, as shown in FIG. These functions can be realized, for example, when the control unit 21 executes a program stored in the storage unit 22. This program may be provided by being stored in a computer-readable information storage medium such as a CD-ROM or a DVD-ROM, or may be provided via a communication network such as the Internet.

  The game process execution unit 31 executes a predetermined game process. In the present embodiment, the game process execution unit 31 executes a dance game as described below. That is, the game process execution unit 31 causes the display device 14 to display an image that guides the user U to the operation content. This image is an image that changes with time. The user U steps on each contact detection member 15 at a predetermined timing so as to dance on the operation device 12 according to the guidance of the image displayed on the display device 14. In this game process, a detection result indicating that each contact detection member 15 detects the contact of the foot of the user U is an operation input of the user U.

  The posture specifying unit 32 acquires a captured image including the user U imaged by the imaging unit 16, and specifies the posture of the user U based on the acquired captured image. In this case, the posture specifying unit 32 has a posture specified based on information indicating whether or not the user U is in contact with each contact detection member 15 at the time of capturing a captured image that is a posture specifying target. Limit the range. In the following description, a captured image including the user U whose posture is to be specified is referred to as a processing target image. FIG. 5 is a diagram illustrating an example of a processing target image captured by the imaging unit 16. Hereinafter, information indicating a detection result by each contact detection member 15 is referred to as contact detection information.

  Hereinafter, a specific example of the process executed by the posture specifying unit 32 will be described. First, the posture specifying unit 32 executes a preparation process described below prior to the game play by the user U. That is, the posture specifying unit 32 moves the agent device 13 to a position where the operation device 12 can be imaged. And the position of each contact detection member 15 in a picked-up image is specified by performing a predetermined pattern recognition process with respect to the picked-up image imaged by the image pick-up part 16 in the said position. In order to enable such processing, it is desirable that each contact detection member 15 has an identifiable appearance, for example, by being colored with a different color or formed with a different pattern.

  Note that if the imaging position, imaging direction, and zoom magnification of the imaging unit 16 do not change, this preparation process may be executed once before the game starts. This is because the position of each contact detection member 15 in the captured image is considered not to change thereafter. On the other hand, when the agent device 13 moves during the game, or when the imaging direction or zoom magnification of the imaging unit 16 changes, the posture specifying unit 32 executes this preparation process each time.

  Further, the posture identifying unit 32 is prepared based on the size of the operation device 12 in the captured image and the information on the actual size (actual size) of the operation device 12 stored in the storage unit 22 in advance. A ratio between the size of the object in the captured image and the actual size may be calculated. In this way, when each part of the body of the user U in the processing target image is specified later, it is possible to estimate the size of the user U in the image, and to improve processing efficiency. Can do.

  Next, the posture specifying unit 32 performs processing for specifying the posture of the user U based on the processing target image captured during the execution of the game. Specifically, the posture specifying unit 32 performs image analysis that specifies the position of a predetermined reference portion (a joint such as an elbow or a knee, a top of the head, a tip of a hand or a foot) in the processing target image. Process. Then, using the skeleton model data representing the predetermined skeleton model 41, a matching process for specifying the posture of the user U is performed.

  Here, the skeleton model 41 is a structure that imitates the skeleton of a human body, and includes a plurality of reference points corresponding to the reference portions described above. FIG. 6 is a diagram illustrating an example of the virtual three-dimensional space 42 in which the skeleton model 41 and the virtual operation device 12 are arranged. The positional relationship between the reference points constituting the skeleton model 41 is constrained by a predetermined condition corresponding to the movable range of the human body. For example, there is a limit to the range in which arms and legs can be bent by joints such as elbows and knees. The skeleton model data includes data that defines the constraint conditions for the values that the position coordinates of each reference point can take.

  In addition, as shown in FIG. 6, a viewpoint position 43 and a line-of-sight direction 44 are set in the virtual three-dimensional space 42. The viewpoint position 43 and the line-of-sight direction 44 are determined in advance so that each contact detection member 15 in the operation device 12 can be seen at the position specified by the above-described preparation process. The viewpoint position 43 and the line-of-sight direction 44 correspond to the position and the imaging direction of the imaging unit 16 in the real space.

  The matching process is a process of matching the skeleton model 41 arranged in the virtual three-dimensional space 42 with the position of each reference part in the processing target image obtained by the image analysis process. As a specific example, the posture specifying unit 32 includes the position of each reference region in the processing target image and the skeleton when the line-of-sight direction 44 is viewed from the viewpoint position 43 within the range of the constraint conditions included in the skeleton model data described above. The position coordinates of each reference point constituting the skeleton model 41 are determined so that the difference between the position of each reference point constituting the model 41 is minimized. The posture of the user U is specified by the position coordinates of each reference point determined in the matching process in the virtual three-dimensional space 42.

  In the matching process, the posture specifying unit 32 uses, as a constraint condition when determining the position coordinates of each reference point, the contact detection information of each contact detection member 15 at the time when the processing target image is captured in addition to the skeleton model data. Use. For example, in the example of the processing target image in FIG. 5, the user U steps on the contact detection members 15b and 15c. Therefore, the posture specifying unit 32 acquires contact detection information indicating that the contact detection members 15 b and 15 c have detected the contact of the user U from the operation device 12. Based on the contact detection information, the posture specifying unit 32 determines that a part corresponding to a predetermined part of the body of the user U (in this case, the tip part of both legs) among the reference points constituting the skeleton model 41 is virtual 3. A condition is set such that it exists at a position corresponding to the contact detection members 15 b and 15 c in the dimension space 42. Based on this condition and a predetermined constraint condition of the skeleton model data, the matching process is executed.

  In addition, when any one of the contact detection members 15 detects the contact of the user U, the posture specifying unit 32, for example, based on the analysis result of the processing target image, the user U who has touched the contact detection member 15 Identify where the body part is. In addition, when the process of specifying the posture has already been performed for the time points before and after the time when the processing target image is captured, each contact detection member 15 is based on the information indicating the posture of the user U at the time points before and after that time. A part of the body of the user U who has contacted the user may be specified.

  As described above, the posture specifying unit 32 uses the contact detection information as a constraint condition in the matching process, so that each reference part of the user U can be relatively accurately obtained from the information of the two-dimensional processing target image. The position can be estimated as position coordinates in the virtual three-dimensional space 42. In addition, even if a part of the processing target image, such as one of the legs of the user U, is hidden, the position of the hidden part can be estimated.

  Note that the posture specifying unit 32 includes not only information indicating that the user U is in contact with each contact detection member 15, but also information indicating that the user U is not in contact with each contact detection member 15. It may be used to specify the posture of For example, when information indicating that the user U is not in contact with any of the contact detection members 15 is obtained as the contact detection information, it is estimated that the user U is jumping at that time. In this case, the height from the floor surface of the user U is estimated according to the time during which all the contact detection members 15 have not detected the contact of the user U, and used as a constraint condition for the matching process. May be.

  Further, depending on the viewing angle of the imaging unit 16, when the user U jumps, the user U's head may protrude from the viewing angle, and the user U's whole body may not be included in the processing target image. In such a case, the body part assumed to exist in the image analysis process is not all included in the processing target image, which may deteriorate the accuracy of the image analysis process. In such a case, for example, the posture specifying unit 32 estimates that the user U is jumping from the contact detection information and assumes that the head of the user U is not included in the processing target image accordingly. Then, the processing target image is analyzed. In this way, the accuracy of analysis can be improved.

  Note that the posture detection apparatus 1 may include a plurality of agent devices 13 each including an imaging unit 16. In this case, the computer 11 can acquire a plurality of processing target images by simultaneously imaging the same user U from different angles by the plurality of imaging units 16. The posture specifying unit 32 is a virtual three-dimensional space of each reference point constituting the skeleton model 41 based on the information on the position and imaging direction of each imaging unit 16 and the image analysis result for each of the plurality of processing target images. The position coordinates in 42 are determined. Thereby, the posture specifying unit 32 can specify the posture of the user U with higher accuracy.

  The timing at which the posture specifying unit 32 specifies the posture of the user U is determined, for example, according to the progress of the game process by the game process execution unit 31 and the detection result of each contact detection member 15. As a specific example, the posture specifying unit 32 specifies the posture of the user U at the timing when the detection result by each contact detection member 15 changes. FIG. 7 is an explanatory view showing the change over time of the detection result by each contact detection member 15. In FIG. 7, for each contact detection member 15, the solid line portion indicates a state where the user U's contact is detected, and the broken line portion indicates a state where the contact is not detected. In the example of FIG. 7, the timing at which the detection result by each contact detection member 15 changes (that is, the timing at which each contact detection member 15 transitions from a state in which it does not detect the contact of the user U to a detected state, or the contact by the user U). The timing of transition from the detected state to the undetected state) is indicated by times t1, t2, t3, t4, t5 and t6. The posture specifying unit 32 executes processing for specifying the posture of the user U using the captured images at these times as processing target images. If it carries out like this, the user's U attitude | position can be specified at the timing estimated that the attitude | position of the user U changes a lot.

  In addition, the posture specifying unit 32 may acquire a processing target image every predetermined time and specify the posture of the user U at each time point. In this case, while the detection result by each contact detection member 15 does not change, the posture specifying unit 32 can analyze the processing target image based on the same constraint condition. For example, in the example of FIG. 7, at the time between time t2 and time t3, the posture of the user U is specified using a constraint condition that both feet of the user U are stepping on the contact detection members 15a and 15b. As described above, the posture detection apparatus 1 specifies the posture of the user U every predetermined time even when the detection result by the contact detection member 15 does not change, so that the user U can move the user U only. The change in posture can be identified.

  In addition, the posture specifying unit 32 may not perform the posture specifying process until any contact detection member 15 detects the contact of the user U after the execution of the game process is started. Thereby, it is possible to avoid performing unnecessary processing until the user U starts playing the game on the operation device 12. Moreover, it is good also as not performing an attitude | position specific process also when all the contact detection members 15 have not detected the contact of the user U for a predetermined time or more. It is because it is estimated that the user U has interrupted the game play in such a case.

  The motion data generation unit 33 generates motion data indicating the motion of the user U based on information indicating the posture of the user U specified by the posture specifying unit 32 for each of a plurality of time points. For example, the motion data is time-series data indicating the time change of the position coordinates of each reference point described above, and thereby represents the movement of the user U's body.

  As a specific example, the motion data generation unit 33 generates motion data by connecting information indicating the posture of the user U every predetermined time specified by the posture specifying unit 32. In addition, the motion data generation unit 33 performs interpolation processing using information indicating the posture of the user U at each of a plurality of time points, thereby obtaining data indicating the posture of the user U at a time point between the plurality of time points. The calculated data may be included in the operation data.

  The motion data generated by the motion data generation unit 33 may be used as an output of game processing by the game processing execution unit 31. For example, the game process execution unit 31 uses the operation data as replay data that reproduces the movement of the user U during game play. In this case, the game process execution unit 31 causes the display device 14 to display an image representing the movement of the user U generated based on the motion data. Thereby, the replay data including the movement of the upper body of the user U that cannot be understood only by the operation input of the user U with respect to each contact detection member 15 can be presented to the user.

  In the present embodiment, the agent device 13 includes a movable part corresponding to a human hand or foot as described above. Therefore, the posture detection device 1 controls the movable part of the agent device 13 based on the motion data generated by the motion data generation unit 33 to cause the agent device 13 to execute an operation to reproduce the movement of the user U. Good. In this case, the plurality of reference points constituting the skeleton model 41 used by the posture specifying unit 32 may be determined according to the movable unit of the agent device 13. In addition, each reference point in this case is constrained by a constraining condition corresponding to the movable range of the movable part of the agent device 13. By doing so, the posture detection apparatus 1 can generate motion data that is relatively close to the actual movement of the user U within a range in which the movement of the user U can be reproduced by the movable portion of the agent device 13.

  According to the present embodiment described above, the posture detection apparatus 1 uses the information indicating the detection result of each contact detection member 15 at the time when the processing target image is captured, so that the user U can accurately detect the processing target image. Can be identified.

  The embodiment of the present invention is not limited to the example described above. For example, some or all of the processes that are executed by the control unit 21 of the computer 11 in the above description may be executed by a computer installed in the agent device 13.

  In the above description, the operation device 12 is arranged on the floor, and each contact detection member 15 detects the contact of the foot of the user U. However, the present invention is not limited to this, and is arranged on a wall surface or the like. You may detect the contact of another site | part, such as U's hand. In the above description, a plurality of contact detection members 15 are attached to the operation device 12, and each contact detection member 15 simply outputs data indicating whether or not the user U is in contact. The member 15 is a member that occupies a certain size range, and may detect a position where a part of the body of the user U contacts in the range. In this way, the operation device 12 can accurately detect the position where the body part of the user U is in contact, and the accuracy of the specified posture of the user U can be improved.

  In the above-described example, the posture of one user is detected. However, the posture detection apparatus 1 may detect the postures of a plurality of users at the same time. In this case, the posture detection apparatus 1 may not only generate the motion data of a plurality of users individually, but also generate motion data that combines the motions of the plurality of users. For example, the posture detection device 1 generates motion data by combining the motion of the upper body of a certain user and the motion of the lower body of another user. Alternatively, motion data may be generated by combining the movement of the right half of one user and the movement of the left half of another user. Thereby, a plurality of users can collaborate to realize one operation or generate an unexpected movement, and the entertainment of the game can be enhanced.

  Moreover, the user who is the target of posture detection by the posture detection device 1 is not limited to a person, and may be an animal such as a dog or a cat. In this case, the posture detection apparatus 1 stores a skeleton model corresponding to the animal's body structure in the storage unit 22 in advance. By using such a skeleton model, motion data that reproduces the movement of a dog or cat can be generated by the same processing as the above-described processing. In this case, the posture detection apparatus 1 may include an agent device 13 that imitates the shape of an animal, and cause the agent device 13 to execute an operation for reproducing the movement of the animal. Further, the posture detection device 1 may generate one motion data by combining the movements of the person and the animal instead of the above-described combination of movements of the plurality of persons.

  Note that the posture detection device 1 specifies the user's movement using the skeleton model, and thus the user's momentum during the game play can be accurately compared with the case where the play time or the operation input to the operation device 12 is simply used. Can be estimated. Therefore, the energy consumption due to the user's exercise may be calculated using such movement of the skeleton model and presented to the user. Further, the posture detection apparatus 1 can accurately grasp the user's movement by generating operation data including the movement of the upper body as compared with the case where the operation input to the operation device 12 is simply used. In view of this, such data may be used to present information that advises the movement of the center of gravity, an ideal posture, and the like for a location where the user has failed while playing the game. Further, the posture detection device 1 stores in advance motion data representing an exemplary motion of the game play, and displays the exemplary motion to the user by displaying the motion data on the display device 14 or by causing the agent device 13 to operate. May be.

1 is a schematic diagram showing an overall outline of a posture detection apparatus according to an embodiment of the present invention. It is a block diagram which shows the structural example of the attitude | position detection apparatus which concerns on embodiment of this invention. It is a figure which shows an example of an operation device. It is a functional block diagram which shows the function example of the attitude | position detection apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the captured image imaged with the attitude | position detection apparatus which concerns on embodiment of this invention. It is explanatory drawing shown notionally about an example of a virtual three-dimensional space and a skeleton model. It is explanatory drawing which shows an example of the time change of the detection result by a contact detection member.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Posture detection apparatus, 11 Computer, 12 Operation device, 13 Agent device, 14 Display apparatus, 15 Contact detection member, 16 Imaging part, 21 Control part, 22 Storage part, 23 Display control part, 24 Interface part, 31 Game process execution Unit, 32 posture specifying unit, 33 motion data generating unit.

Claims (8)

An imaging means for imaging an image including a user;
A contact detection member that detects that any part of the user's body has contacted;
A means for identifying the posture of the user based on the captured image, wherein the identification is performed according to information indicating whether or not the user is in contact with the contact detection member at the time of capturing the image. Posture identifying means for limiting the range of posture;
A posture detection apparatus comprising: In the posture detection device according to claim 1,
There are a plurality of the contact detection members, each arranged in a predetermined positional relationship,
The posture specifying means limits the range of the specified posture according to information indicating whether or not the user is in contact with each of the plurality of contact detection members at the time of capturing the image. An attitude detection device. In the posture detection device according to claim 2,
The plurality of contact detection members are attached to a sheet member arranged on a floor, and detect that each of the contact detection members is stepped on by the user. In the posture detection device according to any one of claims 1 to 3,
Attitude detection further comprising action data generating means for generating action data indicating the user's action based on information indicating the user's attitude specified by the attitude specifying means for each of a plurality of time points. apparatus. In the posture detection apparatus according to claim 4,
A game process execution means for executing a predetermined game process using the detection result of the contact detection member as an operation input of the user;
The posture detection apparatus characterized in that the motion data generated by the motion data generation means is used for output of the game process. Obtaining an image including a user imaged by an imaging means;
Obtaining a detection result by a contact detection member that detects that any part of the user's body has contacted;
The step of specifying the posture of the user based on the acquired image, wherein the specifying is performed according to information indicating whether or not the user is in contact with the contact detection member at the time of capturing the image. Limiting the range of posture;
A posture detection method comprising: Means for acquiring an image including a user imaged by the imaging means;
Means for acquiring a detection result by a contact detection member that detects that any part of the body of the user has contacted; and means for specifying the posture of the user based on the acquired image, wherein the image is Posture specifying means for limiting the range of the posture to be specified according to information indicating whether or not the user is in contact with the contact detection member at the time of imaging;
As a program to make the computer function as.   A computer-readable information storage medium storing the program according to claim 7.

Images