JP7817014B2 - Information processing device and program - Google Patents

Description

The present invention relates to an information processing device and a program.

Technology is known that estimates skeletal information of a person contained in an image captured by a camera (see, for example, Patent Document 1). Also known is technology that remotely controls a robot to perform specific operations based on the movement of a person (see, for example, Patent Document 2).

Japanese Patent Application Laid-Open No. 2021-189946 Japanese Unexamined Patent Publication No. 7-160310

Robots such as those described above are controlled to move in accordance with a person's physical movements or operations. With this type of movement control, the robot's movements are not only expected in advance by the person making the physical movements, including the operations, but also depend solely on that person. Depending on the robot's intended use, it may be desirable to increase the diversity of the robot's movements, taking into consideration convenience for people and the impression it leaves on people. This also applies when moving an image such as an avatar in response to a person's movements.

The present invention aims to provide an information processing device and program that can improve the diversity of movements of at least one of an avatar and a robot based on human movements.

An information processing device according to one aspect of the present disclosure includes image data acquisition means for acquiring image data obtained by capturing an image of a detection target; skeletal information generation means for generating skeletal information based on the detection target image represented by the image data, the skeletal information including positional information of joints in the detection target image and relationship information indicating relationships between joints; analysis means for analyzing the motion details in the detection target image based on the skeletal information; text information generation means for generating text information based on the analysis results of the motion details; and motion control means for operating at least one of a specified avatar to be displayed and a robot, which is a physical machine, as the motion target based on the text information.

A program according to one aspect of the present disclosure causes an information processing device to generate skeletal information including positional information of joints in the detection target image and relationship information indicating the relationships between joints, based on a detection target image represented by image data obtained by capturing an image of an assumed detection target; analyze the motion details in the detection target image based on the skeletal information; generate text information based on the analysis results of the motion details; and, based on the text information, execute a process of operating at least one of a specified avatar to be displayed and a robot, which is a physical machine, as the motion target.

This invention can improve the diversity of movements of at least one of an avatar and a robot based on human movements.

1 is a diagram illustrating an example of a service providing system constructed by applying the present invention and the contents of services provided by the system. 10A and 10B are diagrams illustrating an example of a motion including a posture for making a motion target perform another motion. 1 is a diagram illustrating an example of a network environment to which an AP server according to an embodiment of the information processing apparatus of the present invention is connected; FIG. 1 is a diagram illustrating an example of a place for games and the like. FIG. 2 is a block diagram showing an example of a hardware configuration of an AP server according to an embodiment of the information processing apparatus of the present invention. FIG. 2 is a functional block diagram showing an example of a functional configuration realized on an AP server according to an embodiment of the information processing device of the present invention. 10 is a flowchart illustrating an example of a video display process executed by a CPU installed in an AP server that is an information processing apparatus according to the present embodiment.

The following describes embodiments of the present invention with reference to the drawings. Please note that the described embodiments are merely examples, and the technical scope of the present invention is not limited to these. The technical scope of the present invention also includes various modifications.

Figure 1 is a diagram illustrating an example of a service provision system constructed by applying the present invention, and the content of the services provided by that system. Here, the services provided by this service provision system SY will be referred to as "this service."

In the example shown in Figure 1, it is assumed that the camera C captures images (video recording) of animals, primarily humans. In this example, a human is the detection target KT that is the subject of skeletal detection, and it is assumed that information to be communicated is transmitted to the detection target KT in a timely manner by audio output depending on the movement of the detection target KT.

Skeletal information of the detection target KT is generated to confirm the movement of the detection target KT. This skeletal information includes position information of the person's joints and relationship information that indicates the relationships between the joints. The relationship information is, for example, information that indicates the distance and direction between adjacent joints on the skeleton. Such skeletal information is generated by analysis using the target image ST1 captured by camera C. In this way, skeletal information of the detection target KT is generated from the target image ST1. The skeleton image SG shown in Figure 1 represents an example of the positions of the joints and the joints for which relationship information is generated. The generation of skeletal information itself is performed using well-known technology.

It should be noted that multiple cameras C may be used for imaging. When multiple cameras C are installed in different locations, the target image ST1 obtained from each camera C makes it possible to check parts of the detection target KT that are blind spots for one or more cameras C. This allows for more accurate generation of skeletal information. For example, this is particularly useful when there are two or more detection targets KT. For the sake of convenience, the following explanation will be given assuming that only one camera C is used.

As shown in FIG. 1, the service providing system SY includes a skeleton information generating system H1, a posture evaluation system H2, and a posture management system H3.
1, each of systems H1 to H3 is shown separately, but these systems (functions thereof) may be realized on the same information processing device. Two of them may be realized on the same information processing device. Here, as shown in FIG. 1, it is assumed that each of systems H1 to H3 is realized on a different information processing device, that is, the service providing system SY is constructed using three or more information processing devices.

The skeletal information generation system H1 is implemented on an information processing device equipped with an OS (Operating System) H11. The OSH11 is equipped with a camera device image acquisition and management switch (SW) H111. This camera device image acquisition and management switch H111 has the function of switching the destination of various information, including the target image ST1 received by the skeletal information generation system H1. As a result, the target image ST1 received by the skeletal information generation system H1 is passed to the processing engine H112 as the target image ST2 by the camera device image acquisition and management SW111. The processing engine H112 and the skeletal detection unit H113 are functions implemented, for example, by an application program (hereinafter abbreviated as "application") running on the OSH11.

The processing engine H112 is a function that performs overall control for providing this service. The processing engine H112 causes the skeleton detection unit H113 to execute the skeleton detection process by calling the skeleton detection process ST3, and generates skeleton information using the target video ST2. To achieve this, the processing engine H112 passes, for example, the target video ST2 to the skeleton detection unit H113 by calling the skeleton detection process ST3. What the processing engine H112 passes to the skeleton detection unit H113 may not be the target video ST2, but rather video information containing only the portion of the image to be detected. Here, it is assumed that the target video ST2 is passed from the processing engine H112 to the skeleton detection unit H113.

The skeleton detection unit H113 executes a skeleton detection process to generate skeleton information, and returns the processing result ST4 of the skeleton detection process to the processing engine H112. This processing result ST4 is the skeleton information generated in the form of a data string of skeleton representation.

The processing engine H112 can output the skeletal information, which is the processing result ST4 returned from the skeleton detection unit H113, as a skeletal representation data string ST5 to the posture evaluation system H2 and the posture management system H3. A detection target image KTG, which is an image of the detection target KT, is also output to the posture evaluation system H2 as an image ST6 of the detection target.

The posture evaluation system H2 can output the image ST6 of the detection target input from the skeletal information generation system H1, or the image of a character generated using the skeletal representation data string ST5, to the image display system GH as part of the image ST8 to be displayed. The image ST8 is intended to display, for example, one screen's worth of data, and the character image is positioned and displayed within that screen. The character image is assumed to be an alter ego of the detection target KT.

Even when other character images are displayed, the characters may include alter egos of other detection targets KT, or alter egos of virtual personalities. Virtual personalities are specifically envisioned for the provision of this service. Hereinafter, characters that can be considered as such alter egos will be referred to as "avatars." Unless otherwise specified, the term "characters" will be used to refer to characters other than avatars.

The image display system GH displays a screen on the display D using the image ST8 input from the posture evaluation system H2. Therefore, the posture evaluation system H2 can arbitrarily change the content of the screen displayed on the display D via the image ST8 output to the image display system GH. Figure 1 shows a state in which the detection target image KTG represented by the image ST8 is displayed on the display D. Instead of the detection target image KTG, an avatar image generated from skeletal information may be displayed.

The display D is, for example, a large display device or a screen. When the display D is a screen, the image display system GH also includes a projector capable of projecting onto the screen.
On the other hand, the posture evaluation system H2 generates text information to be output as voice by analysis using the data string ST5 of the skeletal representation. To this end, the posture evaluation system H2 performs an AI (Artificial Intelligence) analysis SH2 of the difference from the model.
As will be described in detail later, this text information also functions as control information for controlling various actions of the subject, which may be an avatar image or a robot. The various actions include not only actions that visually change the subject, but also displays that do not cause visual changes, or speech actions that are made by emitting sound.

The AI analysis SH2 of the difference from the model is an analysis performed on the premise that the posture (pose) that the detection target KT should assume at a certain timing is known in advance. For this analysis, model data TD representing the model posture is prepared. Examples of movements where the posture to be assumed is known in advance include dance, games that require movements that imitate the movements of a model or movements that require movements in one or more predetermined postures, and exercises (e.g., sports) in which ideal movements are conceivable. Hereinafter, such requests, or a series of movements resulting from exercise, etc., will be collectively referred to as "posture-requiring movements."

This model data TD includes skeletal information in addition to the model image OG, which is an image of the model posture. As a result, the AI difference analysis SH2 from the model performs an AI analysis using the difference, including calculating the difference between the skeletal representation data string ST5 and the skeletal information included in the model data TD. This analysis evaluates the posture represented by the detection target image KTG, and generates text information according to the evaluation results. In the example shown in FIG. 1, the difference correction instruction ST9 output from the posture evaluation system H2 to the audio output system TO instructs the output of the text information as audio. Specifically, this difference correction instruction ST9 is either an audio signal generated for audio output, or a command containing information necessary for generating the audio signal.

To perform analysis using differences, the AI performs deep learning using training data that represents the relationship between the differences and the text information to be generated. Through this deep learning, the AI analysis of differences from the model SH2 can generate appropriate text information by generating (calculating) the differences. Because the ratio between the size of the detection target image KTG and the size of the model image OG represented by the model data TD is not necessarily within an appropriate range, the differences are generated, for example, after performing an enlargement or reduction operation on one of the two pieces of skeletal information.

FIG. 1 shows that the display D displays a model image OG in addition to the detection target image KTG.
The right hand is the most significantly different part of the detection target image KTG from the model image OG. The left hand is also clearly different, but the difference is smaller than that of the right hand. For this reason, in this case, text information representing a string such as "Move your right hand a little higher" is generated by the AI analysis SH2 of the difference from the model. As a result, this string is output as sound from the audio output system TO, that is, emitted as sound. This sound output is performed by assuming the detection target image KTG or the model image OG as an avatar image.

The audio output by the audio output system TO is performed with the detection target KT in mind. Therefore, the detection target KT can obtain useful information in a timely manner from the detection target image KTG or model image OG displayed as an avatar image on the display D to help it adopt the appropriate posture it should take. As a result, the displayed detection target image KTG or model image OG is treated as the action target, and by generating text information from the detection target image KTG, a speech action is virtually performed in which the action target speaks.

The action of the action target does not have to be a speaking action, that is, an audio output. For example, the action target may be made to perform the movement necessary to bring the detection target KT closer to the posture of the model image OG. Alternatively, a string of characters may be displayed as the speech content of the avatar image. Audio output (output of a message, etc.) is performed only when the posture of the detection target image KTG is evaluated as inappropriate, and when the posture is evaluated as appropriate, a sound effect or a dramatic sound may be emitted so that the detection target KT can recognize this. For this reason, the text information may represent the emission of a sound effect or a dramatic sound. The speaking action may be represented by either sound emission or display.

As described above, the posture evaluation system H2 uses the skeletal representation data sequence ST5 input from the skeletal information generation system H1 to evaluate the posture of the detection target KT, generating text information and realizing the movement of the avatar image according to that text information. In this way, the movement target performs a different movement than the movement of the detection target KT, and useful information is provided to the detection target KT through this different movement. This not only improves the diversity of the movement of the movement target, but also improves convenience for the detection target KT. The detection target KT can more easily perform movements requiring a more appropriate posture.

The detection target KT is required to recognize the audio output in order to move their body appropriately. This requires a higher level of concentration. Because they must move in this state, the detection target KT can more easily achieve a greater sense of immersion. As a result, the detection target KT can be given a better impression or a greater sense of satisfaction.

On the other hand, the posture management system H3 uses the skeletal representation data string ST5 input from the skeletal information generation system H1 to control the movement of at least one of the physical machine, a robot, and an avatar image displayed on the screen. As a result, the moving object moves in accordance with the movement of the detection object image KTG. Therefore, the detection object KT can control the movement of the moving object through its own movement. This provides the detection object KT with an environment in which it can enjoy moving the moving object as entertainment, or check its own posture or movement. A physical machine here is one equipped with one or more power sources such as motors. A robot, which is a physical machine, is capable of changing its overall appearance using the power transmitted from the power source.

The robot's movement is controlled, for example, by having the posture management system H3 send control information for the next posture to be taken as a movement/display instruction ST7. This control information is mainly information for operating the power source. The movement of the avatar image is controlled, for example, by having the posture management system H3 send image information representing the screen to be displayed as a movement/display instruction ST7 to an image/audio generation system capable of displaying a screen including the avatar image as a moving image.

The overall proportions of each part of both the robot and the avatar image that are the target of movement typically differ from those in the detection target image KTG. For example, if the detection target image KTG has eight heads, the target of movement is likely to have less than eight heads, and the proportions will usually be different. For this reason, the posture management system H3 performs a scale conversion process SH3 to match the input skeletal information to the proportions of the target of movement, and manipulates the skeletal information. This manipulation is performed by referencing parameters P, which are prepared depending on the difference in proportions. By generating and transmitting movement/display instructions ST7 using the skeletal information after such manipulation, the target of movement will move in accordance with the movement represented by the detection target image KTG without giving an unnatural impression.

The posture management system H3 may also prepare model data TD, generate text information, and output the text information. This may allow, for example, information on the quality of a key posture or the next posture to be taken to be transmitted to the detection target KT in a timely manner. When transmitting information on the next posture to be taken, the model data TD may be used to determine whether the detection target image KTG has reached a posture for which text information should be generated.
The above-described different motion of the motion subject is performed by comparing with one posture represented by one example data TD. However, the different motion may be performed by comparing with a motion including a posture.

2 is a diagram illustrating an example of a motion including a posture for causing a motion target to perform another motion. Each circle in FIG. 2 represents a different joint.
The example shown in Figure 2 is an example of a ninja throwing a shuriken. The action of throwing the shuriken is when it is thrown with the left hand. The diagram on the left side of Figure 2 shows an example of the posture taken when throwing a shuriken. In this posture, the right elbow is bent, the right hand holding the shuriken is positioned near the waist at waist height, and the left hand grasps the shuriken. Hereafter, for convenience, this posture will be referred to as the "initial assumed posture."

On the other hand, the diagram on the right side of Figure 2 shows an example of the posture after throwing the shuriken. This posture is an example of the posture after throwing the shuriken by moving the left hand to the right from the initial assumed posture shown in the diagram on the left side of Figure 2. In order to throw the shuriken, the body is rotated counterclockwise when viewed from above.

To throw a shuriken held in the left hand, as shown in Figure 2, the left hand must be moved relatively widely. Therefore, by starting from the initial assumed posture and checking the movement of the hand from the starting point of the hand assumed to be holding the shuriken, it is possible to determine whether the detection target KT has performed the action of throwing the shuriken. Because such a determination is possible, if it is determined that the action of throwing the shuriken has been performed, it is possible to generate text information, such as the sound of the wind cutting through the thrown shuriken, or a special sound. The emission of such sounds makes it possible to provide an environment with a higher sense of realism, particularly in games that require physical movement. The emission of such sounds can be considered a separate action from the action of throwing the shuriken, which was part of the detection target image KTG. In addition to determining whether the above-mentioned action of throwing the shuriken has been performed, it is also possible to further determine the position at which the shuriken was released and the direction in which it was thrown.

When it is possible to detect movements such as those shown in the example of Figure 2, the model data TD may include, for example, the following data. In other words, model data TD may be prepared that includes data such as a first condition under which an initial assumed posture should be considered, a second condition under which a specified movement is considered to have been performed from that initial assumed posture, and the time range from when the first condition is no longer satisfied until the second condition is satisfied.

When making it possible to detect the action of throwing a shuriken in a game or the like, the subject of the action could be an avatar image of an opponent or the like. When the action of throwing a shuriken is confirmed, such a subject of the action could be made to make a statement in response to the shuriken being thrown and perform an action in response to the shuriken. In other words, the generation of one piece of text information could be made to generate one or more pieces of text information.

For this reason, various possibilities are conceivable for the movement of the detection target image KTG that generates text information, the content of the text information generated by that movement, and the movement control after the generated text information is output. For example, specifications could be considered that focus on movements that anticipate the use of weapons other than shuriken. However, regardless of the specifications, not only will the range of expression through the movements of the movement target be wider, but the amount of information provided to the detection target KT will also be greater. As a result, the satisfaction of the detection target KT can be increased.

Hereinafter, a specific method for providing the present service illustrated in FIG. 1 will be described in detail with reference to FIGS.
FIG. 3 is a diagram illustrating an example of a network environment to which an AP (Application) server is connected according to an embodiment of the information processing apparatus of the present invention.

The AP server 1 is an information processing device installed by a service provider SK that provides this service. The service provider SK enters into a contract with an organization, such as a company that provides entertainment such as games, and by providing this service to the contracted organization, helps to increase the satisfaction of visitors who use the organization.
3, the AP server 1 is shown as being installed within the service provider company SK, but it may be installed elsewhere, for example, it may be provided by a cloud service.

AP server 1 is actually connected to network N via another information processing device, such as a proxy. However, for the sake of convenience, information processing devices intervening between AP server 1 and network N will be ignored here. In other words, it is assumed that AP server 1 is directly connected to network N. Network N includes, for example, the Internet.

Customer company KK is an organization that has entered into a contract with service provider SK. This customer company KK provides a space where users can play physical games and other activities. This service makes it possible to provide a more satisfying environment for users who use the space. Hereinafter, customer company KK will be used as a general term for the contracting organization. There will be multiple customer companies KK.

In addition to the camera C, client company KK is also equipped with a server 2, a projector 3, a sound system 4, and a microphone (abbreviated as "mic" in Figure 3; this abbreviation will be used hereafter) 5. All except server 2 are installed in a single location where games and other such activities can be played. For this reason, it is common for there to be multiple copies of all components except server 2.

FIG. 4 is a diagram illustrating an example of a place for games and the like.
A place for games and the like (hereinafter referred to as a "game center") is a space where a screen SC is installed or can be installed, as shown in Fig. 4. In the space, a projector 3 capable of projecting onto the screen SC, a sound system 4 capable of emitting various sounds, a camera C, and a microphone 5 are installed. The camera C is installed so that it can capture images of users from the screen SC side.

The server 2 is connected to a camera C and a microphone 5. The projector 3 and sound system 4 are connected to the AP server 1 via a network N.
The sound system 4 is a system including a speaker, which is a sound emitting device, and also includes an information processing device that functions as a terminal capable of communication via the network N. As a result, the sound system 4 emits sound from the speaker in response to an audio signal transmitted from the AP server 1.

The projector 3 is an information processing device equipped with a communication function via the network N, and can generate a screen to be projected using a video signal transmitted from the AP server 1, and project the generated screen onto the screen SC.
Camera C is used to capture video of users at the game center. The captured video is sent as image data (video information) from server 2 to AP server 1. All or at least one of these users is a detection target KT. Camera C is installed on the screen SC side because it is assumed that users who are detection targets KT will face the screen SC and move their bodies while playing the game.

Microphone 5 is installed to pick up sounds made by game center users. The audio signal output from microphone 5 is converted into audio information by server 2 and transmitted to AP server 1.

Figure 5 is a block diagram showing an example of the hardware configuration of the AP server 1 according to an embodiment of the information processing device of the present invention. Next, with reference to Figure 5, an example of the hardware configuration of the AP server 1 will be described in detail. Note that this example configuration is merely an example, and the hardware configuration of the AP server 1 is not limited to this.

As shown in FIG. 5, the AP server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input/output interface 15, an output unit 16, an input unit 17, a memory unit 18, a communication unit 19, and a drive 20.

The CPU 11 executes various processes in accordance with programs stored in the ROM 12 and programs loaded from the storage unit 18 to the RAM 13. The programs loaded from the storage unit 18 to the RAM 13 include, for example, an OS and various application programs that run on the OS. The various application programs include one or more programs developed for the purpose of providing this service.

The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes, including various programs executed by the CPU 11.
The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input/output interface 15 is also connected to the bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input/output interface 15.

The output unit 16 includes a display such as an LCD. The output unit 16 displays various images or screens under the control of the CPU 11. The output unit 16 may be built into the AP server 1, or may be connected as needed. In other words, the output unit 16 is not a required component.

The input unit 17 is configured to include various hardware buttons, such as a keyboard. It may also include one or more pointing devices, such as a mouse. The operator can input various information via the input unit 17. This input unit 17 may also be built into the AP server 1, but may also be connected as needed. In other words, the input unit 17 is not an essential component.

The storage unit 18 is, for example, a hard disk drive or an auxiliary storage device such as an SSD (Solid State Drive). Large amounts of data are stored in the storage unit 18.
The communication unit 19 enables communication with other information processing devices via the network N. The server 2, the projector 3, and the information processing devices that make up the sound system shown in FIG. 3 all correspond to other information processing devices.

The drive 20 is a device into which removable media 25, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory card, can be attached or detached. The drive 20 can, for example, read information from and write information to the attached removable media 25. As a result, programs recorded on the removable media 25 can be stored in the memory unit 18 via the drive 20. In addition, the removable media 25 attached to the drive 20 can be used as a copy or transfer destination for various data stored in the memory unit 18.

Application programs developed for this service may be recorded on removable media 25 and distributed. They may also be available for distribution via a network N, etc. For this reason, the recording medium on which the application program is recorded may be one that is installed or attached to an information processing device that is directly or indirectly connected to the network N, or one that is installed or attached to an externally accessible device.

The hardware resources of AP Server 1 are controlled by various programs, including application programs. As a result, AP Server 1 is able to provide this service to client company KK.

6 is a functional block diagram showing an example of a functional configuration realized on the AP server 1 according to an embodiment of the information processing device of the present invention. Next, with reference to FIG. 6, an example of the functional configuration realized on the AP server 1 will be described in detail.
To avoid confusion, we will assume that there is only one game arena. We will also assume that the game is played in the game arena. We will assume that both the avatar image and the robot, which are the objects of movement, are controlled using skeletal information generated from an image of the detection object (hereinafter also referred to as the detection object image KTG).

As shown in FIG. 6, the CPU 11 of the AP server 1 has a functional configuration including a setting unit 111, an image extraction unit 112, a skeleton detection unit 113, a posture evaluation unit 114, a text conversion unit 115, a video generation unit 116, a posture control unit 117, a voice recognition unit 118, a conversation processing unit 119, an image recognition unit 120, and a face recognition unit 121.

These functions are realized by the CPU 11 executing various programs, including application programs developed for providing this service. As a result, or in order to provide this service, the memory unit 18 is allocated with a reference image storage unit 181, a character image storage unit 182, a background image storage unit 183, a dictionary storage unit 184, and a control information storage unit 185.

The setting unit 111 has functions for setting up games played at the game center, starting or ending the set game, and various settings required for playing the set game. These various requests for setting up the game are transmitted, for example, from the server 2.

3 and 4 omit the device that enables such settings. This device is connected to the server 2, which processes requests from this device and generates and transmits requests to be sent to the AP server 1.
For example, from before the game starts until the game ends, the server 2 causes the camera C to take moving images and continues to transmit image data (moving image information) obtained by taking moving images to the AP server 1. During this time, the server 2 also transmits audio information obtained by converting an audio signal from the microphone 5 to the AP server 1.

The image extraction unit 112 identifies the range of the detection target image KTG in the image represented by the image data received from the server 2, and extracts the identified range. This range is extracted, for example, by generating still image data from the image data at predetermined time intervals, and targeting the generated still image data. Image data extracted in this way will hereinafter be referred to as "detection target image data."

The skeleton detection unit 113 generates skeleton information from the detection target image KTG represented by each detection target image data, for example (see, for example, Patent Document 1).
The posture evaluation unit 114 evaluates the posture of the detection target image KTG or the movement including the posture using the skeleton information generated by the skeleton detection unit 113. This evaluation is basically the same as that based on the difference AI analysis SH2 from the model in FIG. 1, for example, or is based on the initial assumed posture and subsequent movement as shown in FIG. 2. To perform this evaluation, reference image information corresponding to the model data TD is stored in the reference image storage unit 181. As a result, the corresponding reference image information is also used for the evaluation.

The text conversion unit 115 generates text information by converting the evaluation results from the posture evaluation unit 114. As explained with reference to Figures 1 and 2, the generated text information depends on the reference image information actually used for the evaluation and the difference between the action content represented by the reference image information and the action content represented by the detection target image KTG. The dictionary storage unit 184 stores a dictionary for generating text information, etc., tailored to the intended game. As a result, for example, a sentence or phrase or other character string registered in the dictionary, or a combination of two or more character strings, is generated as text information.

The video generation unit 116 generates an image (screen) to be projected by the projector 3, and causes the communication unit 19 to transmit the generated image as image data.
The video generation unit 116 also generates, as necessary, information for emitting music, sound effects, dramatic sounds, and voices that are intended to be heard by the detection target KT, and transmits the generated information to the communication unit 19. This provides an environment in which the detection target KT can play games, etc., while various sounds are being emitted. Hereinafter, information for emitting various sounds will be collectively referred to as "audio information."

The character image storage unit 182 stores various image information for generating various character images, including avatar images. This image information is, for example, image information representing three-dimensional representations of each part that makes up an avatar image. The background image storage unit 183 stores image information for generating images other than the various character images as background image information. As a result, the video generation unit 116 generates a background image from the various image information stored in the background image storage unit 183, and generates each avatar image using skeletal information and the various image information stored in the character image storage unit 182. After generating these, the video generation unit 116 generates one screen's worth of images by placing each avatar image on the background image.

The posture control unit 117 generates posture control information for operating the robot using the skeleton information. This posture control unit 117 can operate the robot in accordance with the movement of the detection target image KTG, instead of or in addition to the avatar image. The posture control information is generated by referencing other control information stored in the control information storage unit 185. This control information is used to generate the posture control information, and the parameter P shown in Figure 1 is stored in the control information storage unit 185 as one piece of control information.

The speech recognition unit 118 performs speech recognition by natural language processing using speech information received from the server 2, for example. Through this speech recognition, the speech recognition unit 118 generates a character string, which is text information representing the content of the speech of the detection target KT. The character string is generated by identifying each character represented by the speech information through pattern matching. This character string corresponds to a first character string in this embodiment.
The conversation processing unit 119 regards the generated character string as the content of a statement made by the detection target KT, and generates a character string to be returned in response to the content of the statement as a response sentence. To this end, the conversation processing unit 119 sequentially performs morphological analysis, syntactic analysis, semantic analysis, contextual analysis, and anaphora analysis on the generated character string to identify the content represented by the character string. Using the identification result, the conversation processing unit 119 generates, for example, a character string associated with the identification result as a response sentence. The character string generated as a response sentence corresponds to the second character string in this embodiment.

The response sentences generated by the conversation processing unit 119 and the audio information for emitting the text information generated by the text conversion unit 115 are generated, for example, by the video generation unit 116 and transmitted from the communication unit 19. As a result, the detection target KT can not only see the image (video) projected onto the screen SC by the projector 3, but also hear the audio for conversation, etc., emitting from the sound system 4.
The sounds that are emitted include not only voice but also music, sound effects, sound effects, etc. Here, unless otherwise specified, audio information is used as a general term for information that causes these sounds and various sounds including voice to be emitted.
If the operation target is a robot, the generation of audio information is performed by the posture control unit 117. As a result, even when the robot is operated, the detection target KT can hear audio such as conversation.

The voice recognition unit 118 and the conversation processing unit 119 enable conversation between the detection target KT and a character whose image is projected on the screen SC. This allows the detection target KT to progress through the game while conversing with the character. Through this conversation, the detection target KT can, for example, make actions or choose a way to progress through the game that they had not considered. This not only allows the detection target KT to feel more immersed in the game, but also broadens the range of ways to enjoy it and achieve greater satisfaction. Examples of actions that the detection target KT had not considered include the appearance (and actions after appearance) of another avatar image or character image due to the detection target KT's own utterances, actions or changes in the detection target KT's own avatar image in response to the content of the utterances, changes in the background image, etc.
It should be noted that well-known technologies are used for both the voice recognition unit 118 and the conversation processing unit 119 .

The image recognition unit 120 recognizes an object that is included in the detection target image KTG or exists as an image nearby. This object should be considered separate from the detection target KT. The object is mainly an object worn by the detection target KT or held in the hand. After recognizing this object image, the image recognition unit 120 identifies the action content of the object image by focusing on changes in the position, posture, size, etc. of the object image. Identification of the action content can also be performed by employing well-known technology in the image recognition unit 120.
By specifying the action content, it is possible to change the object image depending on how the object of the detection target KT is handled, for example. Specifically, if the object is a sword-like object that is regarded as a sword, it is possible to change the representation of the object image depending on the positional relationship with the detection target KT, or to add effects to the object image depending on the speed at which the sword-like object is swung.

The face recognition unit 121 extracts a face portion from the detection target image KTG and recognizes the gender, age, etc. of the detection target KT. Well-known technology can also be used for this face recognition unit 121.
The object may be a dedicated object equipped with an acceleration sensor, a communication device, etc. If the detection results of such a sensor are used for image recognition by the image recognition unit 120, the movement control of the object image can be performed with higher precision. By transmitting information indicating the type of object from the communication device via, for example, the server 2, the AP server 1 can recognize the type of object that the detection target KT is holding or wearing.

The object recognition results by the image recognition unit 120 can be used to generate an avatar image holding the object. For example, if the detection target KT is holding a sword-shaped object, an avatar image holding the sword may be displayed. Alternatively, if the detection target KT has thrown an object, the movement of the thrown object may be depicted in a video. The image representing the object may be stored, for example, in the character image storage unit 182, and selected from the object recognition results or the movement content of the detection target image KTG identified by skeletal information. This selection may be performed, for example, by the image recognition unit 120. Whether the detection target KT is holding an object can be determined, for example, by whether there is a location where the detection target image KTG and the object image are considered to overlap, and whether the degree of change in their sizes exceeds a preset value.
For example, when throwing a shuriken as shown in Figure 2, the direction in which the left hand is moved can be used to estimate the throwing direction of the shuriken and even the position of the target. Such estimation results can be reflected in the movement of the shuriken.

This type of screen control can be achieved by having the video generation unit 116 perform processing in accordance with the response sentence generated by the conversation processing unit 119. The robot's posture control can be achieved by having the posture control unit 117 perform processing in accordance with the response sentence. What moves in accordance with the response sentence is basically an avatar image or robot that is expected to utter the response sentence.

The results of face recognition by the face recognition unit 121 may be used to determine the type (including gender) of the avatar to be displayed as an image, as well as the facial expression, etc. In this case, it becomes possible to select an avatar that is more desirable for the detection target KT, and it also becomes possible to change the facial expression of the avatar depending on the facial expression of the detection target KT.
All of the above-mentioned controls act to further enhance the sense of immersion in the game for the detection target KT, and also to provide a higher level of satisfaction.

In the above functional configuration, the skeleton detection unit 113 corresponds to the skeleton information generation means in this embodiment. Similarly, the posture evaluation unit 114 corresponds to the analysis means, the text conversion unit corresponds to the text information generation means, the video generation unit 116 and posture control unit 117 correspond to the movement control means, the voice recognition unit 118 corresponds to the voice recognition means, and the conversation processing unit 119 corresponds to the conversation processing means. The CPU 11 itself corresponds to the image data acquisition means and the voice information acquisition means.

Figure 7 is a flowchart showing an example of video display processing executed by the CPU installed in the AP server 1, which is an information processing device according to this embodiment. This video display processing is executed, for example, when a request to start a game is made by the server 2 or the like, in order to have the detection target KT play the game. This processing is realized by the CPU 11 executing an application program developed for providing this service. Finally, this video display processing will be described in detail with reference to Figure 7.

To avoid confusion, the video display process shown in the example flowchart in Figure 7 is based on the assumption of a single game center like the one shown in Figure 4. The explanation will be based on this assumption. Therefore, the explanation will be based on the assumption that video information (image data) captured by camera C and audio information from microphone 5 are sent from server 2 to AP server 1. The avatar image is assumed to move in accordance with the movement of the detection target KT. The CPU 11 is assumed to be the entity that executes the process.

First, in step S1, the CPU 11 determines whether or not video information has been received from the server 2. If video information has been received, the determination in step S1 is YES, and the process proceeds to step S2. If video information has not been received, the determination in step S1 is NO, and the process proceeds to step S6.
In step S2, the CPU 11 generates a still image from, for example, the received video information and extracts a detection target image KTG from the still image. In the following step S3, the CPU 11 executes a skeleton detection process to generate skeleton information from the extracted detection target image KTG. In the next step S4, the CPU 11 evaluates the posture of the detection target image KTG from the generated skeleton information. This posture evaluation may also use previously generated skeleton information, if necessary.

In step S5 following step S4, the CPU 11 converts the evaluation result into text by generating text information according to the evaluation result of the posture. After converting the evaluation result into text, the process proceeds to step S6.
In step S6, the CPU 11 determines whether or not voice information has been received from the server 2. If voice information has been received, the determination in step S6 is YES, and the process proceeds to step S7. If voice information has not been received, the determination in step S6 is NO, and the process proceeds to step S10.

In step S7, the CPU 11 executes a voice recognition process using the received voice information and previously received voice information. The voice recognition process here includes not only generating a character string from the voice information but also identifying the content of the character string. In the following step S8, the CPU 11 determines whether or not the content of one conversation made by the detection target KT has been identified. If the content of one conversation, that is, the content of the speech from the start to the end of the speech, has been identified, the determination in step S8 is YES and the process proceeds to step S9. If the content of one conversation has not been identified, the determination in step S8 is NO and the process proceeds to step S10.
The determination of the end of speech is made, for example, when the state in which the voice of the detection target KT cannot be confirmed continues for a set time. As a result, if the determination in step S8 is YES, the content last identified in step S7 becomes the content of speech by the detection target KT.
In step S9, the CPU 11 (conversation processing unit 119) generates a character string to be returned to the detection target KT as a response sentence from the identified utterance content. After the generation, the process proceeds to step S10.

In step S10, the CPU 11 determines whether the game has ended. If an end event occurs, such as an instruction to end the game from the detection target KT or the like, or an event that should end the game has occurred within the game, the determination in step S10 is YES and the process proceeds to step S11. If an end event has not occurred, the determination in step S10 is NO and the process proceeds to step S12.

In step S11, the CPU 11 transmits an end screen for notifying the detection target KT of the end of the game to the server 2. After transmitting this end screen, the video display process ends.
In step S12, the CPU 11 executes other processes necessary for the progress of the game.
Some games present the user with options such as items during the game and allow the user to select one of the options. The presentation of options and the selection from among the options are realized by executing other processing in step S12.

In the following step S13, the CPU 11 generates moving image information and audio information, and transmits the generated moving image information and audio information to the projector 3 and sound system 4, respectively. Then, the process returns to step S1.
If an avatar image is present on the screen represented by the video information, the avatar image is generated using the skeleton information generated by executing the skeleton detection process in step S3. An image for one screen is generated by generating each avatar image, character image, and background image, and arranging each avatar image and character image on the background image. The text information generated in step S5 or the response sentence generated in step S9 is reflected in the generation of any avatar image or any character image, as necessary. This allows the movement of the avatar image to be changed by at least one of the generated text information and the response sentence. The video information generated in step S13, where such screen generation is performed, is information generated, for example, by compressing the generated screen.

If text information is generated in step S5, or if a response sentence is generated in step S9, the above-mentioned voice information is generated in step S13.
The audio information to be transmitted is, for example, a digital audio signal. When a digital audio signal is transmitted as an audio signal, the audio signal is generated, for example, by an application that performs text-to-audio conversion (hereinafter referred to as "audio conversion software"). In this case, in step S13, the audio conversion software is called by specifying the text information or response sentence to be converted, and the audio signal is acquired. The acquired audio signal is continuously sent as audio information until there are no more audio signals to be transmitted. When sound is emitted by the sound system 4 via the server 2 equipped with this audio conversion software, the text information and the response sentence can be transmitted as audio information. For this reason, the audio information is not particularly limited.

If the object to be operated is a robot, posture control processing is executed instead of video display processing. The flow of this posture control processing is basically the same as the video display processing described above. The difference from this video display processing is that posture control information is generated instead of video information. For this reason, a detailed explanation of the posture control processing will be omitted.

In this embodiment, the explanation has been given with a focus on games played by moving the body, but as described above, the present invention can be widely applied to any game in which the body is made to perform a posture-requiring action in which one or more postures to be taken are known in advance.
For example, sign language performed by moving the fingers is intended to express the content to be communicated as speech using a predetermined posture and predetermined movements from that posture. Therefore, the content expressed in sign language may be converted into text information using skeletal information, and the text information may be output. Specifically, for example, the sign language movements may be reproduced by an object of movement, and the content of the sign language reproduced by the object of movement may be output as at least one of a display and a sound. Considering use by both hearing-impaired and non-hearing people, it is desirable to output text information for both.

In sports, posture-requiring movements include various types of running (sprints, long-distance running, hurdle races, etc.) and golf. In such sports, for example, while the movement target performs the movement represented by the skeletal information, text information can be generated to inform the detection target KT of areas for improvement or good points by comparing it with a model image OG, and the text information can be output as at least one of a display and a sound. Since it is difficult to convey text information in a timely manner to a running detection target KT, it is desirable to save video information representing the movement of the movement target along with the text information and make it playable.

Furthermore, by using skeletal information, it is possible to identify the state of the detection target KT in a timely manner. This can be used to estimate the future movements of the detection target KT, and generate and output text information based on the estimation results. This can be used to warn the detection target KT when they try to touch an exhibit that they should not touch, or when they move to a place that they should not go to, for example. It is also conceivable that customers visiting a store can be used to prevent criminal acts such as shoplifting by the detection target KT, or to apprehend detection target KT who have committed a criminal act. In any case, more information can be provided, and convenience for users will be increased.

When estimating movement, the movement speed and direction of the detection target image KTG may be estimated from skeletal information, and the estimation results may be reflected in the movement of the moving target. By making such estimations (predictions), it is expected that response will be improved. When applied to detecting risky behavior of the detection target KT, this detection can be performed more quickly.

This movement estimation may be used to control the movements of an avatar, such as an opponent in a fighting game or a game such as an RPG (Role-Playing Game), or a character that is confronted. It is conceivable that the opponent will perform defensive movements in accordance with the movements represented by the skeletal information. However, by estimating movement, it is possible to make the defensive movements more natural. By having the opposing character perform movements in accordance with the movements of the detection target KT, the detection target KT will be given the impression that they are confronting the character. When the movement estimation results are used to control the character's movement, the character's movements can be made more natural, and the detection target KT will be given a stronger impression that they are actually confronting an opponent. In either case, it is desirable to generate text information by posture evaluation using skeletal information, and to have the character perform movements and converse naturally in accordance with the generated text information.

The present invention can also be applied to the control of the movement of avatars or robots in unmanned stores or unmanned reception desks. Both the avatars and robots are made to perform different actions depending on the movement of the detection target KT, and are used to provide the service desired by the detection target KT.
As an application thereof, for example, it may be combined with image recognition technology to recognize the presence or absence of products or luggage, etc., and different responses may be taken depending on the recognition results. More specifically, for example, if the detection target KT makes a movement as if showing a product, the type of product may be identified, and the identification result may be reflected in the subsequent conversation. This is because the detection target KT who makes a movement as if showing a product is likely to have the intention of checking other products of different colors, other products in the same category, or inventory, etc.

As mentioned above, movement targets can be broadly divided into two types: those that perform movements in accordance with the movements of the detection target KT (hereinafter referred to as "first movement targets"), and those that perform movements different from the movements of the detection target KT (hereinafter referred to as "second movement targets"). Therefore, the generated text information can also be broadly divided into two types: that which assumes the first movement target, and that which assumes the second movement. The same is true for the model data TD (reference image information).

The first and second movement targets may both be subjected to movement control. When controlling the movement of both targets simultaneously, text information may be generated assuming its use in controlling the movement of both targets. For example, when the detection target KT takes a transformation pose represented by the model data TD, text information may be generated to output a voice message saying "Transform" to the first movement target, and the second movement target may stop moving or move to a set pose. When using voice recognition technology and natural conversation (language) technology, the detection target KT may utter "Transform" to evaluate whether or not the detection target KT has taken a transformation pose, and generate text information. In this case, the text information may be used to emit a sound effect for the first movement target, or to perform similar movement control for the second movement target. For this reason, various modifications are possible for the conditions for generating text information, the content of the generated text information, and the movement control of the movement targets using the text information.

1 AP server, 2 Server, 3 Projector, 4 Sound system, 5 Microphone, 11 CPU, 18 Memory unit, 19 Communication unit, 111 Setting unit, 112 Image extraction unit, 113 Skeleton detection unit, 114 Posture evaluation unit, 115 Text conversion unit, 116 Video generation unit, 117 Posture control unit, 118 Voice recognition unit, 119 Conversation processing unit, 120 Image recognition unit, 121 Face recognition unit, 181 Reference image storage unit, 182 Character image storage unit, 183 Background image storage unit, 184 Dictionary storage unit, 185 Control information storage unit, C Camera, H1 Skeleton information generation system, H2 Posture evaluation system, H3 Posture management system, KK Client company, KT Detection target, KTG Detection target image, OG Model image, SK Service provider, SY Service provision system

Claims (4)

image data acquisition means for acquiring image data obtained by capturing an image of the detection target;
a skeleton information generating means for generating skeleton information including position information of joints in the detection target image and relationship information indicating relationships between joints, based on the detection target image represented by the image data;
an analysis means for analyzing the details of the motion in the detection target image based on the skeleton information;
a text information generating means for generating text information based on the analysis result of the operation content;
an action control means for controlling at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, as an action target based on the text information;
and
the analyzing means analyzes the motion details using a difference between the skeletal information and information indicating a posture that the detection target should take;
the text information generating means generates the text information in accordance with the difference;
The operation control means
performing a scale conversion process on the skeleton information to convert it in accordance with the ratio of each part of the moving object, and causing the moving object to reproduce the movement represented by the detection object using the skeleton information after the scale conversion process;
Based on the text information, the action target is made to perform an action different from the action represented by the image of the detection target by a speech action of the action target by at least one of displaying a message according to whether the posture of the detection target is appropriate or emitting a sound.
Information processing device. a voice information acquisition means for acquiring voice information representing a voice emitted by the detection target;
a speech recognition unit that performs speech recognition using the speech information and generates a first character string from the speech information;
a conversation processing means for generating a second character string to be a response to the first character string,
the action control means outputs the second character string as a speech content in the speech action of the action target;
The information processing device according to claim 1 . the text information generation means generates the text information representing the content of the sign language when the finger movement in the detection target image is identified as a movement for sign language as a result of the analysis of the content of the action,
the motion control means causes the motion target to perform a finger motion in accordance with the finger motion using the skeletal information, and causes the motion control means to output the text information.
3. The information processing device according to claim 1 or 2 . In the information processing device,
a skeletal information generating step of generating skeletal information including position information of joints in a detection target image and relationship information indicating relationships between joints, based on the detection target image represented by image data obtained by capturing an image of an assumed detection target;
an analyzing step of analyzing the details of the motion in the detection target image based on the skeleton information;
a text information generating step of generating text information based on the analysis result of the operation content;
a motion control step of controlling at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, as a motion target based on the text information;
Including,
In the analyzing step, the motion content is analyzed using a difference between the skeletal information and information indicating a posture that the detection target should take;
In the text information generating step, the text information is generated in accordance with the difference;
In the operation control step,
performing a scale conversion process on the skeleton information to convert it in accordance with the ratio of each part of the moving object, and causing the moving object to reproduce the movement represented by the detection object using the skeleton information after the scale conversion process;
Based on the text information, the action target is made to perform an action different from the action represented by the image of the detection target by a speech action of the action target by at least one of displaying a message according to whether the posture of the detection target is appropriate or emitting a sound.
A program that executes a process.