JP7627488B2 - Pseudo-touch presentation system, control program, and control method - Google Patents

Description

This invention relates to a pseudo-touch presentation system, a control program, and a control method, and in particular to a pseudo-touch presentation system, a control program, and a control method that use, for example, an acting body that presents a touch stimulus to a user and a head-mounted display that presents a visual stimulus to the user.

One example of this type of conventional pseudo-touch presentation system is disclosed in Patent Document 1. According to the pseudo-touch presentation system disclosed in Patent Document 1, a communication robot presents a touch stimulus to a user by, for example, hugging the user with arms long enough to wrap around the user's back. A virtual agent whose state changes substantially in sync with the hugging action of the communication robot is displayed on an HMD (Head Mounted Display) worn on the user's head. The communication robot hugs the user back, providing the user with a touch stimulus, and furthermore the virtual agent on the HMD provides the user with a visual stimulus.

JP 2019-45928 A

In the pseudo-contact presentation system disclosed in the above-mentioned Patent Document 1, one virtual agent is assigned to one communication robot, and the movement of the virtual agent is synchronized with the movement of the communication robot. In other words, the configuration for presenting a stimulus to the user through contact using a communication robot and a virtual agent is simple. Patent Document 1 does not take into consideration presenting a stimulus to the user through contact with multiple virtual agents.

Therefore, the main object of this invention is to provide a novel pseudo-tactile presentation system, control program, and control method.

Another object of the present invention is to provide a pseudo-touch presentation system, a control program, and a control method that can provide complex touch stimuli.

The first invention is a pseudo-touch presentation system comprising a contact stimulation presentation device having a plurality of stimulation presentation units that act on a user to provide the user with a contact stimulation, an allocation means for assigning a virtual agent to each of two or more of the plurality of stimulation presentation units, and a head-mounted display device that is worn on the user's head, displays the plurality of virtual agents assigned by the allocation means, and operates some or all of the plurality of virtual agents to present a visual stimulation to the user, wherein the contact stimulation presentation device selectively executes a first process in synchronization with one virtual agent that performs a first predetermined action toward the user, causing the stimulation presentation unit to which the virtual agent is assigned to perform an action corresponding to the first predetermined action, or a second process in which two or more of the plurality of virtual agents simultaneously perform a second predetermined action toward the user, and each of the stimulation presentation units to which two or more of the plurality of virtual agents are assigned performs an action corresponding to the second predetermined action .

A second invention is dependent on the first invention, and the head-mounted display device displays an animation of some or all of the multiple virtual agents showing their actions toward the user, the stimulation presentation unit includes a movable part, and the contact stimulation presentation device presents a contact stimulation to the user by operating the movable part such that movement of the movable part changes in relation to the animation of the actions of some or all of the multiple virtual agents.

A third invention is a control program executed by a computer of a pseudo-touch presentation system including a contact stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide the user with contact stimuli, and a head-mounted display device that is worn on the user's head, the sensory stimulus presentation program causing the computer to function as: a stimulus presentation unit control means for controlling the plurality of stimulus presentation units of the contact stimulus presentation device to present contact stimuli to the user; an allocation means for assigning a virtual agent to each of two or more of the plurality of stimulus presentation units ; a display control means for displaying some or all of the plurality of virtual agents assigned by the allocation means on the head-mounted display device ; and an execution means for selectively executing a first process for causing the stimulus presentation unit to which the virtual agent is assigned to perform an operation corresponding to the first predetermined operation in synchronization with one virtual agent that performs a first predetermined action toward the user, or a second process for causing two or more of the plurality of virtual agents to simultaneously perform a second predetermined action toward the user and for each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned to perform an operation corresponding to the second predetermined action .

A fourth invention is a control method for a pseudo-touch presentation system including a contact stimulus presentation device having a plurality of stimulus presentation units that act on a user to present a contact stimulus to the user, and a head-mounted display device that is worn on the user's head, the control method including the steps of controlling the plurality of stimulus presentation units of the contact stimulus presentation device to present a contact stimulus to the user, assigning a virtual agent to each of two or more of the plurality of stimulus presentation units , displaying some or all of the assigned plurality of virtual agents on the head-mounted display device , and selectively executing a first process of causing the stimulus presentation unit to which the virtual agent is assigned to perform an action corresponding to the first predetermined action in synchronization with one virtual agent that performs a first predetermined action toward the user, or a second process of causing two or more of the plurality of virtual agents to simultaneously perform a second predetermined action toward the user, and causing each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned to perform an action corresponding to the second predetermined action .

According to this invention, some or all of a plurality of virtual agents displayed on a head-mounted display device worn on the user's head are operated, and in association with the operation, a stimulus presentation unit corresponding to some or all of the plurality of virtual agents is operated, so that visual stimuli due to the operation of the virtual agents are presented, and in association with this, a tactile stimulus is presented by the stimulus presentation unit. In other words, it is possible to obtain a bodily sensation as if one were touching the virtual agent. Thus, a new pseudo-touch presentation system, control program, and control method for presenting pseudo-touch can be presented.

The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

FIG. 1 is a block diagram showing the electrical configuration of a pseudo-tactile presentation system according to an embodiment of the present invention. FIG. 2 is a front view of the external configuration of the robot shown in FIG. FIG. 3 is a block diagram showing the electrical configuration of the robot shown in FIG. FIG. 4 is a side view of an example of a state in which the pseudo tactile presentation system shown in FIG. 1 is used. FIG. 5 is a diagram showing an example of two virtual agents displayed on the HMD shown in FIG. FIG. 6 is a diagram showing an example of three virtual agents displayed on the HMD shown in FIG. FIG. 7 is a diagram showing another example of two virtual agents displayed on the HMD shown in FIG. FIG. 8 is a diagram showing another example of three virtual agents displayed on the HMD shown in FIG. FIG. 9 is a diagram showing an example of a memory map of a RAM built in the server shown in FIG. FIG. 10 is a flowchart showing a part of an example of the first pseudo-contact process of the CPU built in the server shown in FIG. FIG. 11 is a flowchart showing another part of the first pseudo-contact processing by the CPU built in the server shown in FIG. 1, which follows FIG. FIG. 12 is a flowchart showing a part of an example of the second pseudo-touch process of the CPU built in the server shown in FIG. FIG. 13 is a flowchart showing another part of the second pseudo-contact processing by the CPU built in the server shown in FIG. 1, which follows FIG.

Referring to FIG. 1, the pseudo-touch presentation system (hereinafter referred to as the "system") 10 of this embodiment includes a robot 12 for presenting a physical or bodily stimulus to a user (person or participant), that is, a stimulus given to the user's sense of touch or a stimulus given to the user by contact (hereinafter referred to as the "touch stimulus"). In this embodiment, the robot 12 is a communication robot that includes an action of presenting a touch stimulus and generates sound as necessary.

Figure 2 shows an example of the external appearance of the robot 12 as a tactile stimulus presentation device used in the embodiment of Figure 1, viewed from the front. However, any tactile stimulus presentation device having a different appearance and structure can be used.

The robot 12 includes three rod-shaped frames 120a, 120b, and 120c. The frames 120a, 120b, and 120c may be made of a metal such as aluminum, or a resin such as fiber-reinforced plastic or engineering plastic, and are basically designed with a cross-sectional shape and thickness that ensures safe strength. For example, a rectangular column or rectangular tube frame may be used.

Frame 120a constitutes the bottom of robot 12, extends horizontally, and is attached to the floor or floor surface. Frame 120b constitutes the torso of robot 12, extends vertically, and has its lower end fixed to the center or nearly the center of frame 120a. Frame 120c constitutes the shoulder of robot 12, extends horizontally, and is fixed to the upper part of frame 120b.

The frame 120c has robot arms (hereinafter simply referred to as "arms") 122, 124, and 126 extending from both ends. Each of the arms 122, 124, and 126 has a structure that mimics a human arm, and each has joints that correspond to an elbow joint and a shoulder joint. Therefore, in the following description, for convenience, names of parts of a human arm may be used.

Arms 122 and 124 are provided at the left end of frame 120c when robot 12 is viewed from the front, and arm 126 is provided at the right end of frame 120c when robot 12 is viewed from the front. Hereinafter, when describing robot 12 or parts thereof using directions, up, down, left and right when robot 12 is viewed from the front will be used.

The arm 122 includes a frame 122a corresponding to the forearm and a frame 122b corresponding to the upper arm, and further includes a joint (corresponding to an elbow joint) 122c that connects frames 122a and 122b, and a joint (corresponding to a shoulder joint) 122d that connects frames 122b and 120c.

The arm 124 includes a frame 124a corresponding to the forearm and a frame 124b corresponding to the upper arm, and further includes a joint 124c (corresponding to an elbow joint) that connects frames 124a and 124b, and a joint 124d (corresponding to a shoulder joint) that connects frames 124b and 120c.

However, as can be seen from FIG. 2, arm 122 is connected to the underside of the left end of frame 120c, and arm 124 is connected to the upper side of the left end of frame 120c.

The arm 126 includes a frame 126a corresponding to the forearm and a frame 126b corresponding to the upper arm, and further includes a joint (corresponding to an elbow joint) 126c that connects the frames 126a and 126b, and a joint (corresponding to a shoulder joint) 126d that connects the frames 126b and 120c. This arm 126 is connected to the underside of the right end of the frame 120c.

As an example, joints 122c, 124c, and 126c can each open the forearm to a maximum of 90° relative to the upper arm. Thus, by controlling the angles of joints 122c, 124c, and 126c, arms 122, 124, and 126 can be bent and extended, respectively.

As another example, the joints 122d, 124d, and 126d can open the upper arm up to 150° with respect to the plane where the joints 122d, 124d, and 126d are connected to the frame 120c. However, the angle when the frames 122b, 124b, and 126b that make up the upper arm are perpendicular to the plane where the joints 122d, 124d, and 126d are connected to the frame 120c is set to 0°. Therefore, by controlling the angles of the joints 122d, 124d, and 126d, the arms 122, 124, and 126 can be raised and lowered, or extended forward and backward.

Furthermore, in this embodiment, actuators are incorporated in joints 122c, 122d, 124c, 124d, 126c and 126d so that robot 12 can shake hands with user 60 (see FIG. 4) with arms 122, 124 and 126, respectively.

The contact stimulation presentation device, i.e., the robot 12, is connected to a server 16 via a network 14 such as the Internet or a telephone communication line. The server 16 is a general-purpose computer such as a PC, a PDA, a smartphone, or a tablet terminal.

Although not shown, the server 16 includes an input device such as a touch panel and/or a keyboard, and an operator can operate such an input device. Programs and data for controlling the operation of the server 16 are stored in a memory (e.g., HDD and RAM) 16b built into the server 16, and the overall operation of the server 16 is controlled by a CPU 16a also built into the server 16. This CPU 16a controls the movement and speech of the robot 12 through a CPU 36 (described below) of the robot 12 in accordance with the above-mentioned operation of the operator, and further controls the display of a virtual agent by the HMD 20 described below. For this reason, the server 16 is provided with a communication module 16c, and the CPU 16a is connected to the network 14, i.e., the robot 12 and the HMD 20, via the communication module 16c, in a wired or wireless manner so as to be able to communicate with them.

The system 10 in FIG. 1 also includes an HMD 20, which is a head-mounted display device, and this HMD 20 is worn on the head 64 of a user 60 who receives contact stimuli presented by the robot 12 (see FIG. 4). In the embodiment, an Oculus Rift (product name) manufactured by Oculus Inc. is used as this HMD 20, and a position detection device 18a such as an Oculus sensor (product name) is used to accurately detect the position of this HMD 20, i.e., the head position of the user 60.

In other words, the server 16 is provided with the output of the position detection device 18a. The position detection device 18a functions as a head position measurement unit, and the Oculus sensor used as this position detection device 18a combines a gyro sensor and acceleration to track the position of the HMD 20 (Oculus Rift), thereby enabling it to accurately detect the position of the user 60's head 64.

As described above, the robot 12 is fixed to the floor, and the position of the HMD 20 in real space, i.e., the head position of the user 60, can be detected by setting or registering the initial position of the HMD 20 relative to the robot 12. Therefore, in this embodiment, the CPU 16a controls the positions of the virtual agent and the viewpoint (virtual camera) in the virtual space using the positional relationship between the head positions of the robot 12 and the user 60 in real space. Since the HMD 20 displays a so-called first-person perspective image (in this embodiment, the virtual agent), the position of the virtual camera corresponds to the head position of the user 60 in the virtual space. In addition, the gaze point of the virtual camera is set to a predetermined position of the robot 12 in the virtual space. For example, the predetermined position is set to the center of the robot 12 in the virtual space. This is just one example, and other positions may be adopted as long as virtual agents of various sizes fit into the image displayed on the HMD 20.

However, the robot 12 itself is not rendered in the virtual space, and the position of the robot 12 and the like are only used when placing a virtual agent in the virtual space and when determining the position of a virtual camera in the virtual space. For example, the position of the robot 12 (the position of the center of frame 120a) is set to the origin in the three-dimensional virtual space, the direction parallel to frames 120a and 120c of the robot 12 is set to the X-axis direction, the direction parallel to frame 120b is set to the Z-axis direction, and the direction perpendicular to both the X-axis and Z-axis is set to the Y-axis direction.

Furthermore, a microphone 18b that collects environmental sounds and the voice of the user 60, and a camera 18c that captures environmental images and images of the user 60 are connected to the server 16. Voice data collected by the microphone 18b is input to the server 16, and the server 16 can use the voice data to perform voice recognition. Image data from the camera 18c is input to the server 16, and the server 16 can process the image data to perform, for example, personal identification.

In this embodiment, the robot 12 is capable of interacting with the user 60. To this end, the topics necessary for such an interaction with the user 60 and scenarios corresponding to those topics are preset in the memory 16b of the server 16. However, as explained above, a remote operator can also interact with the user 60 through the server 16.

The above-mentioned HMD 20 can display high-resolution stereoscopic images with a 110° field of view at a refresh rate of, for example, 90 Hz.

The HMD 20 displays virtual agents 70 (70a, 70b, 70c) as shown in Figures 5-8, which will be described later, based on image data transmitted from the server 16. This provides visual and/or auditory stimuli (visual stimuli) to the user 60.

For this reason, image data of multiple virtual agents 70 of different genders and ages is preset in the memory 16b of the server 16. The image data includes not only still image data of the virtual agent 70 standing upright, but also animation data of the virtual agent 70 performing specific actions (in this embodiment, greeting, shaking hands, and walking).

Then, the server 16 identifies the user 60 who will cooperate (interact) with the robot 12, and then selects a virtual agent 70 of the gender and age that matches the identified user 60, and sends the image data to the HMD 20 via the network 14. This allows the HMD 20 to display the virtual agent 70 that matches the user 60. However, the virtual agent 70 may also be registered by the user 60.

In this embodiment, multiple virtual agents 70 can be assigned to the robot 12. In this embodiment, the number of virtual agents 70 assigned to the robot 12 is equal to or less than the number of arms of the robot 12. Thus, one, two or three virtual agents 70 are assigned to correspond to the three arms 122, 124 and 126 of the robot 12.

The user 60 can decide how many virtual agents 70 to assign to the robot 12. However, when the system 10 is applied to any application such as a game, the number can be determined either fixedly or dynamically depending on the scenario of the application.

When one virtual agent 70 is assigned to the robot 12, one virtual agent 70 is assigned to the arms 122 and 126. When two virtual agents 70 are assigned to the robot 12, one of the two virtual agents 70 is assigned to the arm 122, and the other of the two virtual agents 70 is assigned to the arm 126. When three virtual agents 70 are assigned to the robot 12, one of the three virtual agents 70 is assigned to the arm 122, one of the remaining two virtual agents 70 is assigned to the arm 124, and the other of the remaining two virtual agents 70 is assigned to the arm 126.

However, this is just an example and need not be limiting. In addition, the multiple assigned virtual agents 70 are displayed on the HMD 20 so as not to overlap each other.

Note that a detailed explanation of the case where one virtual agent 70 is assigned to a robot 12 is omitted since this is not essential to the present invention and overlaps with the contents disclosed in JP 2019-45928, which was previously filed by the applicant and has already been published.

Although not shown in FIG. 2, a speaker and a microphone are attached to the upper part of the robot 12 (for example, the upper end of the frame 120b), and the robot 12 can speak to the user 60 and listen to what the user 60 says. XIMERA (product name: ATR Speech and Language Research Laboratories) was used as the speech synthesis software for speaking.

Figure 3 is a block diagram showing an example of the electrical configuration of the robot 12. As shown in Figure 3, the robot 12 is equipped with a CPU 36 that is responsible for the overall control of the robot 12. The CPU 36 is connected to a communication module 40 via a bus 38, and therefore the CPU 36 is communicatively connected to the network 14, i.e., the server 16, via the communication module 40 in a wired or wireless manner.

The CPU 36 can also access the memory 42 through the bus 38, and according to the programs and data set in the memory 42, it issues command values to the actuator control circuit 44 through the bus 38 to control the operation of each actuator A1-An. The actuator control circuit 44 generates a number of pulse powers according to the command values provided by the CPU 36, and provides them to the corresponding stepping motors to drive each actuator A1-An.

However, in addition to actuators that use stepping motors, any actuator can be used, such as actuators that use servo motors or fluid actuators.

Here, actuators A1-An are actuators (stepping motors) for joints 122c, 122d, 124c, 124d, 126c, and 126d. Therefore, the actuator control circuit 44 also functions as a joint control unit.

The sensor I/F (interface) 46 is connected to the CPU 36 via the bus 38 and receives the outputs from the tactile sensor 48 and the camera 50.

The tactile sensor 48 is, for example, a touch sensor, and constitutes part of the sense of touch of the robot 12. In other words, the tactile sensor 48 detects whether or not a human or other object has touched the robot 12. For example, the tactile sensor 48 is provided on each of the forearms (frames 122a, 124a, and 126a) of the arms 122, 124, and 126 of the robot 12, and can detect whether or not the hand of the user 60 has touched the arms 122, 124, and 126 of the robot 12.

The tactile sensor 48 also functions as a pressure sensor. As an example, such a tactile sensor 48 is formed, for example, from a polymer piezoelectric sensor sheet, and multiple sensor sheets are appropriately distributed and provided, for example, on the inner surface of the arm 28. Such a tactile sensor 48 can detect the pressure applied to the forearm (frames 122a, 124a, 126a) of the robot 12 when shaking hands with the user 60.

The output (detection data) from the tactile sensor 48 is provided to the CPU 36 via the sensor I/F 46. Therefore, the CPU 36 can detect when a human or other object has touched the robot 12 (and the strength or pressure of the touch) and when the human or other object has left the robot 12.

Furthermore, the pressure data detected by the tactile sensor 48 is output from the CPU 36 and provided to the server 16 via the communication module 40 and the network 14.

The camera 50 is an image sensor and constitutes part of the robot 12's vision. In other words, the camera 50 is used to detect an image or pictures seen by the eyes of the robot 12. In this embodiment, data (image data) corresponding to the image (moving image or still image) captured by the camera 50 is provided to the CPU 36 via the sensor I/F 46. The CPU 36 not only detects changes in the captured image, but also transmits the image data to the server 16 via the communication module 40 and the network 14 (Figure 1). The server 16 then outputs the received image data to a monitor (not shown). Thus, the image captured by the camera 50 is displayed on the monitor.

The speaker 54 and microphone 56 are also connected to the input/output I/F 52. The speaker 54 outputs sound when the robot 12 speaks. When the operator of the server 16 (hereinafter sometimes referred to as the "remote operator") speaks directly, the sound is output via the network 14, the communication module 40, and the input/output I/F 52. Specifically, when the remote operator speaks through the microphone 56, corresponding voice data is provided from the server 16 to the CPU 36 via the network 14. The CPU 36 then outputs the voice data from the speaker 54 via the input/output I/F 52.

The microphone 56 is a sound sensor that constitutes part of the robot 12's hearing. The microphone 56 is directional and is primarily used to detect the voice of the user 60 (person) who is interacting (communicating) with the robot 12.

The memory 42 of the robot 12 is, for example, a RAM and a HDD, and is preset with a voice recognition program and a voice synthesis program such as the above-mentioned XIMERA. The voice recognition program is a program that allows the CPU 36 to recognize the contents of speech from a human being to the robot 12, which is input through the microphone 56, and the CPU 36 recognizes the contents of the human speech by, for example, DP matching or the Hidden Markov Model (HMM).

The CPU 36 recognizes the speech input by the remote operator through the microphone 56 according to the speech recognition program. For example, the speech can be detected using an acoustic model such as the above-mentioned HMM method or a deep neural network (DNN).

In addition to directly outputting the voice of the remote operator from the speaker 54, the robot 12 can also speak using a synthetic voice. When an instruction to speak using a synthetic voice from the speaker 54 is input from the server 16, the CPU 36 outputs the synthetic voice to the speaker 54 according to the synthetic voice data provided by the server 16.

In this embodiment, basically, all of the movements of the robot 12 are given by the server 16 operated by an operator (not shown). To this end, the CPU 36 gives sensor data acquired through the sensor I/F 46 and audio data and image data acquired through the input/output I/F 52 to the CPU 16a of the server 16 via the communication module 40 and the network 14. The server 16 can obtain this data and know the state of the robot 12.

However, the server 16 can also autonomously control the behavior of both the virtual agent 70 and the robot 12 using position information and pressure information from the position detection device 18a.

Figure 4 shows an example of a state in which a user 60 uses the system 10, viewed from the side. As shown in Figure 4, the user 60 wears the HMD 20 and stands directly in front of the robot 12. In other words, the robot 12 and the user 60 face each other. The robot 12 and the user 60 can converse or talk with each other. Also, when the user 60 approaches the robot 12 and the distance d between the robot 12 and the user 60 becomes equal to or smaller than a predetermined distance df, the robot 12 can touch the user 60 with the arms 122, 124, and 126. In this case, the user 60 can touch at least one of the arms 122, 124, and 126 of the robot 12. As an example, the predetermined distance df is 70 cm-1 m, and is determined by the length of the arms 122, 124, and 126.

As described above, multiple virtual agents 70 can be assigned to the robot 12. When two virtual agents 70 are assigned to the robot 12, one virtual agent 70 is placed at a position overlapping the arm 122 of the robot 12, and the other virtual agent 70 is placed at a position overlapping the arm 126 of the robot 12. In the example shown in FIG. 5, the virtual agents 70a and 70b are placed in the virtual space such that the left arm of the virtual agent 70a overlaps the arm 122, and the right arm of the virtual agent 70b overlaps the arm 126. In other words, when two virtual agents 70 are assigned to the robot 12, the arm 122 of the robot 12 corresponds to the left arm of the virtual agent 70a, and the arm 126 of the robot 12 corresponds to the right arm of the virtual agent 70b.

In addition, when three virtual agents 70 are assigned to the robot 12, one of the three virtual agents 70 is arranged at a position overlapping the arm 122 of the robot 12, one of the remaining two virtual agents 70b is arranged at a position overlapping the arm 126 of the robot 12, and the other of the remaining two virtual agents 70c is arranged at a position overlapping the arm 124 of the robot 12. In the example shown in FIG. 6, the right arm of the virtual agent 70a is arranged at a position overlapping the arm 122 of the robot 12, the right arm of the virtual agent 70b is arranged at a position overlapping the arm 126 of the robot 12, and the left arm of the virtual agent 70c is arranged at a position overlapping the arm 124 of the robot 12. In other words, when three virtual agents 70 are assigned to the robot 12, the arm 122 of the robot 12 corresponds to the right arm of the virtual agent 70a, the arm 124 of the robot 12 corresponds to the left arm of the virtual agent 70c, and the arm 126 of the robot 12 corresponds to the right arm of the virtual agent 70b.

Therefore, the CPU 16a of the server 16 can execute a process (hereinafter referred to as "first pseudo-contact process") in which each of the multiple virtual agents 70 individually performs a first predetermined action toward a user 60 (person or participant) 60 approaching the robot 12, and also operates the arm 122, 124 or 126 to which the virtual agent 70 is assigned in synchronization with the virtual agent 70 performing the first predetermined action toward the user 60. However, the action of the arm 122, 124 or 126 is an action corresponding to the first predetermined action.

The CPU 16a of the server 16 can also execute a process (hereinafter referred to as "second pseudo-contact process") in which multiple virtual agents 70 are brought close to the user 60 in the virtual space, and the multiple virtual agents 70 simultaneously perform a second predetermined action toward the user 60, and the arms 122 and 124, or the arms 122, 124 and 126 to which the multiple virtual agents 70 are assigned are operated. However, the actions of the arms 122, 124 and 126 correspond to the second predetermined action.

In this embodiment, the first and second predetermined actions are handshakes. As an example, the first pseudo-contact process is executed in a situation where the user 60 individually shakes hands with multiple virtual agents 70 who are idols at a handshake event venue set in a virtual space. On the other hand, the second pseudo-contact process is executed in a situation where the user 60 is an idol, is surrounded by multiple virtual agents 70 who are fans, and is asked to shake hands by each virtual agent 70.

Therefore, which of the first and second pseudo-contact processes is to be executed is determined by the operator of the server 16 or a remote operator, or by the scenario (context) of the application to which the system 10 is applied. However, the application scenario may be variably determined depending on which option the user 60 selects.

For example, in the first pseudo-contact process, the user 60 interacts with one, two or three virtual agents 70 assigned to the robot 12, and when the user 60 approaches the robot 12 to a predetermined distance df at which the user 60 can shake hands with the robot 12, the CPU 16a of the server 16 transmits a signal to both the virtual agent 70 and the robot 12, causes the HMD 20 to display an animation of the virtual agent 70 shaking hands with the robot 12, and causes the robot 12 to start shaking hands. That is, a contact action toward the user 60 is executed. Therefore, the user 60 can grasp the arm 122, 124 or 126 of the robot 12 to shake hands, and gets the experience of shaking hands with the virtual agent 70 assigned to the grasped arm 122, 124 or 126. That is, a pseudo-contact is presented to the user 60. However, among the multiple virtual agents 70, only the virtual agent 70 assigned to the arm 122, 124 or 126 that is closest to the user 60 displays the animation of the handshake.

For example, in a case where two virtual agents 70 are assigned to the robot 12, when the user 60 approaches the virtual agent 70a, as shown in FIG. 7, the virtual agent 70a shakes hands, and in relation to this (or in synchronization), the arm 122 corresponding to the virtual agent 70a is operated. That is, the robot 12 uses the arm 122 to shake hands. Therefore, the user 60 grasps the arm 122 of the robot 12 and shakes hands while watching the virtual agent 70a shaking hands on the HMD 20. That is, the user 60 is presented with a visual stimulus in the form of an animation of the virtual agent 70 displayed on the HMD 20, and is given a tactile stimulus due to the movement of the robot 12.

Although not shown in the figure, by making it possible to detect the user's hand movements from images from camera 18c, etc., an image of the user's hand can also be displayed on HMD 20 when the user shakes hands. However, even if the user's hand movements are not detected, it is possible to know from the output of tactile sensor 48 that the user's hand, etc. has touched the arm of robot 12, so an image of the user's hand may be displayed on HMD 20 when it is detected that the user's hand, etc. has touched the arm of robot 12. The same applies below when users shake hands.

Although not shown in the figures, in the first pseudo-contact process, when three virtual agents 70 are assigned to the robot 12, the virtual agent 70 that is closest to the user 60 (viewpoint) in the virtual space performs a handshake, and the robot 12 performs the handshake action using the arm 122, 124, or 126 that corresponds to that virtual agent 70.

In the first pseudo-contact process, when one virtual agent 70 is assigned to the robot 12, when the user 60 (viewpoint) approaches the virtual agent 70 in the virtual space, the virtual agent 70 performs a handshake, and the robot 12 performs the handshake action using the arm 122 or 124 that corresponds to the virtual agent 70.

In this embodiment, the intimacy between the user 60 and each virtual agent 70 is set, and in the first pseudo-contact process, when the user 60 shakes hands with a virtual agent 70, the intimacy of the virtual agent 70 with respect to the user 60 is improved. For example, the intimacy of a virtual agent 70 with which the user has never shook hands is set to 0, and the intimacy is increased by 1 each time the user shakes hands. For example, the intimacy can be used as a condition for selecting a virtual agent 70, and can also be used to determine the content of speech when greeting. When selecting a virtual agent 70, a virtual agent 70 with a high or low intimacy can be selected. Also, a user 60 with an intimacy of 0 can be greeted with "Nice to meet you," and a user 60 with an intimacy of 1 or more can be greeted with "Hello" or "It's been a while." These are just examples, and the intimacy may be used for other purposes.

In the first pseudo-contact process, at least one of the virtual agents 70 greets the user 60, and the virtual agents 70 converse with the user 60 individually. When the virtual agent 70 greets the user 60, the user 60 is guided to approach the robot 12. In addition, when the user 60 converses with the virtual agents 70 individually, as an example, the virtual agents 70 speak to the user 60 in turn, and when each of the virtual agents 70 finishes speaking, the user 60 speaks next.

Note that since the purpose of this case is to allow the user 60 to virtually experience the movements of the virtual agent 70 through the movements of the robot 12, i.e., to present a simulated contact, there is no need for interaction between the user 60 and the virtual agent 70.

The first pseudo-contact process is terminated when the user 60 performs a specific action such as moving away from the robot 12, waving at the robot 12, or removing the HMD 20, or when the user 60 utters a specific message such as "goodbye" or "bye bye." The same applies to the termination of the second pseudo-contact process. Note that it is possible to detect that the user 60 has removed the HMD 20 based on the image from the camera 18c.

In the second pseudo-contact process, one, two, or three virtual agents 70 assigned to the robot 12 approach the user 60, and the user 60 shakes hands with the one, two, or three virtual agents 70 assigned to the robot 12. When the user 60 approaches the robot 12 to a predetermined distance df at which the user 60 can shake hands, the CPU 16a of the server 16 starts the second pseudo-contact process. This is because, in the second pseudo-contact process, the virtual agent 70 is brought close to the user 60 (viewpoint) in the virtual space to shake hands, so the user 60 needs to be close enough to the robot 12 to shake hands.

In addition, in the second pseudo-contact process, the virtual agent 70 is moved closer to the user (the viewpoint), so in this embodiment, when the server 16 draws one or more virtual agents 70 at the start of the second pseudo-contact process, the server 16 places (draws) the one or more virtual agents 70 in a position farther away from the viewpoint than the positional relationship between the robot 12 and the user 60 in the real space.

When the second pseudo-contact process is started, one, two or three virtual agents 70 assigned to the robot 12 are gradually brought closer to the user 60 (virtual camera). A signal is sent to each of the one, two or three virtual agents 70 assigned to the robot 12 and the robot 12, and an animation of all the virtual agents 70 assigned to the robot 12 shaking hands is displayed on the HMD 20, and the robot 12 starts shaking hands. That is, a contact action toward the user 60 is executed. However, in the second pseudo-contact process, all the virtual agents 70 assigned to the robot 12 shake hands at the same time, so that when the robot 12 shakes hands, the arms corresponding to all the virtual agents 70 assigned to the robot 12 are operated. That is, when one virtual agent 70 and two virtual agents 70 are assigned to the robot 12, the arms 122 and 126 are operated. Also, when three virtual agents 70 are assigned to the robot 12, the arms 122, 124 and 126 are operated. Therefore, the user 60 can grasp any one or two of the arms 122, 124, and 126 of the robot 12 to shake hands, and experience the sensation of shaking hands with one, two, or all three virtual agents 70 assigned to the robot 12. In other words, a pseudo-contact is presented to the user 60.

For example, when three virtual agents 70 are assigned to the robot 12, when the three virtual agents 70 approach the user 60 (viewpoint) in the virtual space, as shown in FIG. 8, the three virtual agents 70 (70a, 70b, 70c) shake hands, and in association with (or in synchronization with) this, the arms 122, 124, and 126 corresponding to each of the three virtual agents 70 are operated. That is, the robot 12 shakes hands using the arms 122, 124, and 126. Therefore, the user 60 grasps one or two of the arms 122, 124, and 126 of the robot 12 and shakes hands while watching the three virtual agents 70 shaking hands on the HMD 20. That is, a visual stimulus due to the animation of the virtual agents 70 displayed on the HMD 20 is presented to the user 60, and a tactile stimulus due to the movement of the robot 12 is given to the user 60.

Although not shown in the figure, in the second pseudo-contact process, when two virtual agents 70 are assigned to the robot 12, in the virtual space, when the two virtual agents 70 approach the user 60 (viewpoint), the two virtual agents 70 shake hands, and the robot 12 performs the handshake action using the arms 122 and 126 corresponding to the two virtual agents 70.

In the second pseudo-contact process, when one virtual agent 70 is assigned to the robot 12, when the one virtual agent 70 approaches the user 60 (viewpoint) in the virtual space, the one virtual agent 70 performs a handshake, and the robot 12 performs the handshake action using the arm 122 that corresponds to the one virtual agent 70.

Although detailed explanation is omitted, the robot 12 is equipped with a tactile sensor 48 on each of the frames 122a, 124a, and 126a of the arms 122, 124, and 126. Therefore, in the first pseudo-contact process and the second pseudo-contact process, when the robot 12 is shaking hands, upon detecting contact with the hand or the like of the user 60, the arms 122, 124, and 126 may be moved up and down repeatedly.

The movements of the arms 122, 124, and 126 change in relation to the movements of the virtual agent 70, and the server 16 is programmed with a program that can synchronize the movements (state transitions) of the robot 12 with the movements (state transitions) of the virtual agent 70 on the HMD 20 to enhance such correlation.

In this program, the joint angles of the robot 12's joints 122c, 122d, 124c, 124d, 126c, and 126d (detected by the actuators A1-An that control each joint and fed back to the motor control) are controlled to match the angles of the virtual agent 70's arm and shoulder joints.

However, the movements of the robot 12 and the virtual agent 70 do not necessarily need to be synchronized. It has been confirmed that there is no problem if the movements of the robot 12 and the virtual agent 70 displayed on the HMD 20 are slightly out of sync in time, so long as there is a certain degree of lag.

When synchronizing the movements of the robot 12 and the virtual agent 70, the delay time of the movements of the robot 12 is detected in advance, and the animation of the virtual agent 70 is played back taking that delay time into account.

Figure 9 shows an example of a memory map 300 of memory (here, RAM) 16b built into server 16. As shown in Figure 9, memory 16b includes a program memory area 302 and a data memory area 304. The control program of this embodiment is stored in program memory area 302.

The control programs include a communication program 302a, an assignment program 302b, a sensor value detection program 302c, an image generation program 302d, and a robot control program 302e.

The communication program 302a is a program for communicating with external devices, in this embodiment, the robot 12 and the HMD 20, either wired or wirelessly.

The allocation program 302b is a program for allocating one or more virtual agents 70 to the robot 12 according to the number of virtual agents 70 set according to user operations, etc.

The sensor value detection program 302c is a program for detecting the output from various sensors (in this embodiment, the tactile sensor 48, the camera 50, and the microphone 56).

The image generation program 302d is a program for generating image data to be displayed on the HMD 20. In this embodiment, image data is generated for still images and videos (animations) of one or more virtual agents 70 assigned to the robot 12.

The robot control program 302e is a program for transmitting to the robot 12, when one or more virtual agents 70 assigned to the robot 12 perform a predetermined first motion and a second motion (in this embodiment, the first motion and the second motion are shaking hands), motion data for moving the arms (122 and 124, or 122, 124 and 126) corresponding to each virtual agent 70 in synchronization with the predetermined first motion and second motion, and, if necessary, voice data corresponding to the speech content of the robot 12.

Although not shown in the figure, the program memory area 302 also stores other programs for controlling the robot 12 and the virtual agent 70.

The data memory area 304 also stores sensor value data 304a, arm control data 304b, speech data 304c, and image data 304d.

The sensor value data 304a is data output from various sensors and is data detected according to the sensor value detection program 302c. The arm control data 304b is data for controlling the arms 122, 124, and 126 of the robot 12, and more specifically, data for driving the actuators.

The speech data 304c is synthetic voice data of the contents spoken by the virtual agent 70 assigned to the robot 12. However, the speech (dialogue) content can use content previously set in a database (for example, memory 16b of the server 16) or can use an API (Application Programming Interface) on the web, etc.

The image data 304d is image data for displaying the virtual agent 70 on the HMD 20, and is generated according to the image generation program 302d. However, the image data 304d includes still image data of the virtual agent 70 and animation data of the movements of the virtual agent 70.

Although not shown in the figure, the data memory area 304 stores other data necessary for controlling the robot 12 and the virtual agent 70, and is provided with timers (counters) and flags.

Figures 10 and 11 are flow diagrams of the first pseudo-contact process of the CPU 16a built into the server 16 shown in Figure 1. Figures 12 and 13 are flow diagrams of the second pseudo-contact process of the CPU 16a built into the server 16 shown in Figure 1. Below, the first pseudo-contact process and the second pseudo-contact process will be explained, but the same processes as those already explained will be explained briefly. In this embodiment, the operations of the first pseudo-contact and the second pseudo-contact are executed by the CPU 16a of the server 16 according to a control program set in the memory 16b. In other words, the CPU 16a of the server 16 sends a command signal to the CPU 36 (Figure 3) of the robot 12, so that the server 16 indirectly controls the operation of the robot 12.

As shown in FIG. 10, when the CPU 16a starts the first pseudo-contact process, in step S1, it detects image data from the camera 18c, and in step S3, it identifies who the user 60 is based on the image data detected in step S1.

However, instead of image data from the camera 18c, other means such as an RFID (radio frequency identifier) may be used to identify the user 60. In this case, the CPU 16a detects the RFID in step S1.

If the user 60 has not been registered in advance and the user's identity cannot be identified in step S1, the process can be continued by temporarily assigning an anonymous ID.

Then, in step S5, the number of virtual agents 70 is set. As described above, the number of virtual agents 70 can be set according to the user's operation within the number of arms equipped to the robot 12, and in this embodiment, it is set to any one of 1 to 3.

In the following step S7, it is determined whether the number of virtual agents 70 is 1. If the answer is "YES" in step S7, that is, if the number of virtual agents 70 is 1, one virtual agent 70 is displayed on the HMD 20 in step S9, and the process proceeds to step S17 shown in FIG. 11.

However, in step S9, image data of one virtual agent 70 selected according to the user 60 identified in step S3, drawn at a position where its right arm overlaps with arm 122 and its left arm overlaps with arm 124, is transmitted to the HMD 20.

In this case, the server 16 may randomly select from candidates for a virtual agent 70 of the same or opposite sex as the user 60, or the user 60 may manually select a virtual agent 70 that has been registered for himself or herself in advance.

However, for a user 60 with the above-mentioned anonymous ID, a specific virtual agent 70 may be manually selected.

When manually selecting a virtual agent 70 in this manner, for example, a selection switch (not shown) may be provided on the HMD 20, and in response to the operation of the selection switch, the server 16 may sequentially display images of candidate virtual agents 70 on the HMD 20, and when the desired virtual agent 70 is displayed, the operation of the selection switch may be stopped.

The distance d between the robot 12 and the user 60 is calculated based on the predetermined position of the robot 12 and the position information from the position detection device 18a, and in the virtual space, the distance between the viewpoint (virtual camera) and the robot 12 is set to a distance equivalent to the distance d in real space, and the virtual agent 70 is rendered. As described above, the gaze point of the virtual camera is the center of the robot 12 in the virtual space. The same applies below to the case of rendering the virtual agent 70 and generating image data.

On the other hand, if step S7 is "NO," that is, if the number of virtual agents 70 is not 1, then in step S11 it is determined whether the number of virtual agents 70 is 2.

If the answer is "YES" in step S11, that is, if the number of virtual agents 70 is two, then in step S13, two virtual agents 70 are displayed on the HMD 20, and the process proceeds to step S17.

However, in step S13, two virtual agents 70 selected according to the user 60 identified in step S3 are drawn side-by-side with the left arm of one of them overlapping with the arm 122 and the right arm of the other one overlapping with the arm 124, and image data is transmitted to the HMD 20.

On the other hand, if step S11 is "NO", that is, if the number of virtual agents 70 is three, three virtual agents 70 are displayed on the HMD 20 in step S15, and the process proceeds to step S17.

However, in step S15, the image data of the three virtual agents 70 selected according to the user 60 identified in step S3, drawn side-by-side with the right arm of one of them overlapping with the arm 122, the right arm of one of the remaining two overlapping with the arm 124, and the left arm of the other of the remaining two overlapping with the arm 126, is transmitted to the HMD 20.

As shown in FIG. 11, in step S17, a topic is selected and a dialogue is started. At this time, the user 60 may select the theme of the dialogue, or a specific theme may be automatically selected based on the dialogue history of each user 60 stored in the server 16. In addition, in step S17, when starting the dialogue, synthetic voice data of the speech content of a greeting is transmitted to the robot 12 and the HMD 20. However, instead of a greeting, the speech content may be for bringing the user 60 closer to the robot 12 or for inducing a dialogue. Furthermore, when multiple virtual agents 70 are assigned to the robot 12, synthetic voice data of the speech content for each of the multiple virtual agents 70 is transmitted to the robot 12 and the HMD 20 in a predetermined order. However, this is just one example, and synthetic voice data of the speech content of only one virtual agent 70 may be transmitted. However, the pitch (frequency) of the synthetic voice is changed depending on the type of virtual agent 70.

Next, in step S19, all arms 122, 124, and 126 of the robot 12 are returned to their initial positions. In other words, the CPU 16a transmits control data to the robot 12 to return the arms 122, 124, and 126 to their initial positions.

Then, in step S21, the robot waits for a certain period of time (for example, 2 to 3 seconds). Here, the movement and operation of the robot 12 and all virtual agents 70 are stopped for a certain period of time. Then, in step S23, it is determined whether the user 60 (viewpoint) has come close enough to shake hands with the virtual agents 70. However, if multiple virtual agents 70 are assigned to the robot 12, it is determined whether the user has come close enough to shake hands with any one of the virtual agents 70. In other words, it is determined whether the distance between the user 60 and the arms 122, 124, and 126 of the robot 12 to which the virtual agent 70 is assigned in real space is equal to or less than a predetermined distance df. Specifically, the CPU 16a determines whether the user 60 has come close enough to shake hands with at least one of the arms 122, 124, and 126 based on image data from the camera 50 received from the robot 12 via the network 14.

However, the image data from the camera 50 may be viewed on a monitor (not shown) and the operator may make the decision.

In the embodiment, the image data from the camera 50 is analyzed to make the judgment in step S23, but it is also possible to use a camera 18c installed in the environment, such as the floor or ceiling on which the robot 12 is placed, the laser range finder (Oculus Sensors) described above, or a user 60 position detection system such as a mat sensor.

If step S23 is "NO", that is, if the user 60 has not approached close enough to shake hands with the virtual agent 70, the process returns to step S21. On the other hand, if step S23 is "YES", that is, if the user 60 has approached close enough to shake hands with the virtual agent 70, then in step S25, the arm of the virtual agent 70 closest to the user 60 is brought closer to the user 60 (the viewpoint) to shake hands. Here, the CPU 16a plays an animation in which the arm of the virtual agent 70 closest to the user 60 is brought closer to the direction of the viewpoint in the virtual space to shake hands.

In the next step S27, the arm 122, 124 or 126 of the robot 12 corresponding to the virtual agent 70 closest to the user 60 is brought close to the user 60 to shake hands. In step S27, the CPU 16a controls the actuators of the joints (122c and 122d, 124c and 124d, or 126c and 126d) through the actuator control circuit 44 to slightly raise (or move forward) the arm 122, 124 or 126 of the robot 12 corresponding to the virtual agent 70 closest to the user 60 in the direction of the user 60 with the angle of the forearm (frame 122a) relative to the upper arm (frame 122b), the angle of the forearm (frame 124a) relative to the upper arm (frame 124b), or the angle of the forearm (frame 126a) relative to the upper arm (frame 126b) set to 70°.

Next, in step S29, the intimacy level of the virtual agent 70 is improved, and in step S31, a dialogue according to the topic is executed. Here, the CPU 16a executes a dialogue according to the topic selected in step S17. However, by storing a dialogue history, if a dialogue has been held in the past about the selected topic, non-overlapping utterance content is selected.

In addition, when the virtual agent 70 performs an action in conjunction with a speech, the arm 122, 124, or 126 to which the virtual agent 70 is assigned also operates in synchronization with the action of the virtual agent 70.

Then, in step S33, it is determined whether the user 60 wishes to end the dialogue. Here, the CPU 16a determines that the user 60 wishes to end the dialogue when it detects a specific action of the user 60 as described above, or detects an utterance by the user 60 about a specific content as described above. The same applies to the processing of step S81 described below.

If step S33 is "NO", that is, if the user 60 does not wish to end the dialogue, the process returns to step S31 to continue the dialogue. On the other hand, if step S33 is "YES", that is, if the user 60 wishes to end the dialogue, the first pseudo-contact process is terminated.

Also, as shown in FIG. 12, when the CPU 16a starts the second pseudo-contact process, in step S51, it detects image data from the camera 18c, and in step S53, it identifies who the user 60 is based on the image data detected in step S51.

However, in the second pseudo-contact process, an image is displayed on the HMD 20 to make the virtual agent 70 approach the user 60. As described above, the second pseudo-contact process is started on the premise that the user 60 has approached a position where they can shake hands with the robot 12.

In the following step S55, the number of virtual agents 70 is set, and in step S57, it is determined whether the number of virtual agents 70 is 1. If the answer is "YES" in step S57, in step S59, one virtual agent 70 is displayed on the HMD 20, and the process proceeds to step S67 shown in FIG. 13.

On the other hand, if the result of step S57 is "NO", then in step S61 it is determined whether the number of virtual agents 70 is two. If the result of step S61 is "YES", then in step S63 two virtual agents 70 are displayed on the HMD 20, and the process proceeds to step S67. On the other hand, if the result of step S61 is "NO", then in step S65 three virtual agents 70 are displayed on the HMD 20, and the process proceeds to step S67.

In the second pseudo-contact process, as described above, in order to move the virtual agent 70 closer to the user (point of view), when drawing one or more virtual agents 70 in steps S59, S63, and S65, the CPU 16a places (draws) the one or more virtual agents 70 in a position farther away from the point of view than the positional relationship between the robot 12 and the user 60 in the real space.

As shown in FIG. 13, in step S67, all arms 122, 124, and 126 of the robot 12 are returned to their initial positions. Then, in step S69, the process waits for a certain period of time. In the next step S71, all virtual agents 70 are moved closer to the user 60. For example, the CPU 16a generates image data of the case where all virtual agents 70 are moved a predetermined distance (e.g., 70 cm) toward the viewpoint in the virtual space, and transmits the image data to the HMD 20.

Then, in step S73, it is determined whether all virtual agents 70 have come close enough to shake hands with the user 60. Here, it is determined whether the distance between each of the virtual agents 70 and the viewpoint in the virtual space has become equal to or less than a predetermined distance df.

If step S73 is "NO", that is, if all virtual agents 70 have not come close enough to shake hands with the user 60, the process returns to step S69. On the other hand, if step S73 is "YES", that is, if all virtual agents 70 have come close enough to shake hands, in step S75, all virtual agents 70 bring their arms closer to the user 60 and shake hands. Here, the CPU 16a plays an animation in which all virtual agents 70 bring their arms closer to the direction of the viewpoint in the virtual space and shake hands. However, if there is only one virtual agent 70, the animation of that virtual agent 70 is played. If there are two virtual agents 70, the animation of each of the two virtual agents 70 is played. If there are three virtual agents 70, the animation of each of the three virtual agents 70 is played.

In the next step S77, the arms of the robots 12 corresponding to all the virtual agents 70 are brought close to the user 60 to shake hands. In step S77, the CPU 16a raises (or moves forward) the arms of the robots 12 corresponding to all the virtual agents 70 slightly in the direction of the user 60, with the angle of the forearm relative to the upper arm set to 70°. However, if there is one virtual agent 70, the movement of the arm 122 is controlled. If there are two virtual agents 70, the movements of the arms 122 and 124 are controlled. If there are three virtual agents 70, the movements of the arms 122, 124 and 126 are controlled.

Next, in step S79, the process waits for a certain period of time (for example, 2 to 3 seconds), and in step S81, it is determined whether the user 60 wishes to end the handshake. If the answer is "NO" in step S81, that is, if the user 60 does not wish to end the handshake, the process returns to step S79. On the other hand, if the answer is "YES" in step S81, that is, if the user 60 wishes to end the handshake, the second pseudo-contact process is terminated.

According to this embodiment, an arbitrary visual stimulus is presented using multiple virtual agents using an HMD, while tactile stimuli are presented from multiple arms of a robot to which multiple virtual agents are assigned, allowing the user to experience the sensation of touching the virtual agents. In other words, a new pseudo-tactile presentation system can be provided.

In addition, according to this embodiment, the visual stimulation presentation device, i.e., the HMD, displays the movements of multiple virtual agents, and in association with these movements, the movable parts of the tactile stimulation presentation device, i.e., the multiple arms of the robot, are moved to present tactile stimulation to the user, thereby enhancing the sense of unity between the visual stimulation and the tactile stimulation.

In the second pseudo-contact process of this embodiment, when three virtual agents are assigned to a robot, the three virtual agents are brought closer to the user (point of view) to perform an action, but two of the three virtual agents may be brought closer to the user to perform an action. In this case, the two of the three virtual agents are selected according to a predetermined rule such as random, or by the user.

In this embodiment, the robot arm is configured to have joints equivalent to an elbow joint and a shoulder joint, but it may also be configured to have only an elbow joint or a shoulder joint. If it has only an elbow joint, it is possible to shake hands by moving the upper arm, and if it has only a shoulder joint, it is possible to shake hands by moving the entire arm.

In addition, in this embodiment, the robot has three arms, but it is also possible to provide four or more arms, as long as the number of arms is two or more. When providing four or more arms, it is possible to display even more virtual agents on the HMD and to interact with each virtual agent individually or simultaneously.

Furthermore, in this embodiment, a case has been described in which the user shakes hands with the virtual agent, but the predetermined action does not have to be limited to a handshake and may be a touch.

In the above embodiment, a single server controls each element of the system, but the robot, server, and HMD may each be controlled by a separate computer, and the system may be managed and controlled by any number of computers.

The virtual agent displayed on the HMD shown in the above embodiment may be any character that is a 3D computer graphics agent. Therefore, the appearance, gender, clothing, species (human, animal, anthropomorphized character, etc.), and realism (animation style, live action style, etc.) may be changed depending on the user.

Note that the specific values of angles and other values given in the above examples are merely examples and can be changed as necessary.

10 ... Pseudo-touch presentation system 12 ... Communication robot 16 ... Server 20 ... HMD
60 ... User 70 ... Virtual agent 122, 124, 126 ... Robot arm

Claims (4)

A contact stimulus presentation device including a plurality of stimulus presentation units that act on a user to provide the user with a contact stimulus;
an allocation means for allocating virtual agents to each of two or more of the plurality of stimulus presentation units; and a head-mounted display device that is worn on the head of the user, displays the plurality of virtual agents allocated by the allocation means, and operates some or all of the plurality of virtual agents to present a visual stimulus to the user,
The contact stimulation presentation device selectively executes a first process in which, in synchronization with one of the virtual agents performing a first predetermined action toward the user, the stimulation presentation unit to which the virtual agent is assigned executes an action corresponding to the first predetermined action, or a second process in which two or more of the plurality of virtual agents simultaneously perform a second predetermined action toward the user and each of the stimulation presentation units to which two or more of the plurality of virtual agents are assigned executes an action corresponding to the second predetermined action. This is a pseudo-contact presentation system. the head-mounted display displays some or all of the plurality of virtual agents as animations showing actions for the user;
The pseudo-touch presentation system of claim 1, wherein the stimulus presentation unit includes a movable part, and the tactile stimulus presentation device presents the tactile stimulus to the user by operating the movable part so that the movement of the movable part changes in association with animation of actions for some or all of the plurality of virtual agents. A control program executed by a computer of a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, the control program comprising:
a stimulus presentation unit control means for controlling the plurality of stimulus presentation units of the tactile stimulus presentation device to present a tactile stimulus to the user;
an allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units;
a display control means for displaying , on the head-mounted display device, a part or all of the plurality of virtual agents assigned by the assignment means; and
a sensory stimulus presentation program that functions as an execution means for selectively executing a first process of causing the stimulus presentation unit to which the virtual agent is assigned to perform an action corresponding to the first predetermined action in synchronization with one of the virtual agents performing a first predetermined action toward the user, or a second process of causing two or more of the plurality of virtual agents to simultaneously perform a second predetermined action toward the user and causing each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned to perform an action corresponding to the second predetermined action . A method for controlling a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, comprising:
A step of controlling the plurality of stimulus presentation units of the contact stimulus presentation device to present a contact stimulus to the user;
assigning a virtual agent to each of two or more of the stimulus presentation units;
A control method comprising: a step of displaying some or all of the multiple assigned virtual agents on the head-mounted display device; and selectively executing a first process in which, in synchronization with one of the virtual agents performing a first predetermined action toward the user, the stimulus presentation unit to which the virtual agent is assigned executes an action corresponding to the first predetermined action, or a second process in which two or more of the multiple virtual agents simultaneously perform a second predetermined action toward the user and each of the stimulus presentation units to which two or more of the multiple virtual agents are assigned executes an action corresponding to the second predetermined action.

Images