JP5055486B2 - Remote operation android utterance control system - Google Patents

Description

  The present invention relates to a remote operation android utterance operation control system, and more particularly to a system for controlling the lip movement of an android utterance based on an utterance voice of an operator, for example.

In general, a method of outputting the lip shape using phoneme information used for speech synthesis is effective. However, when there is no phoneme information, it is necessary to estimate the lip shape from the acoustic features of the uttered speech. For example, Non-Patent Documents 1-4 propose a technique for reproducing the movement of a person's lips based only on uttered speech. The method for estimating the shape of the lips from only the speech is divided into two steps: acoustic feature extraction and mapping from acoustic features to the lip shape. As acoustic features extracted from a speech signal, there are LPC (linear predictive coding) -Cepstral coefficient, MFCC (Mel-Frequency Cepstral Coefficients) coefficient, LSP (Line Spectrum Pair) coefficient, and the like. Examples of mapping methods from acoustic features to lip shapes include linear regression analysis, neural networks, HMM (Hidden Markov Model), and KNN (K-Nearest Neighbor). In the above method, one-to-one mapping between voice and lip-shaped image information is performed for each regular frame.
Lavagetto, F., “Converting speech into lip movements: A multimedia telephone for hard of hearing people”, IEEE Trans. On Rehabilitation Engineering, Vol. 3, No. 1, pp. 90-102, March 1995. Yehia, H., Rubin, P., Vatikiotis-Bateson, E., “Quantitative association of vocal-tract and facial behavior”, Speech Communication, vol. 26, no. 1-2, pp. 23-43, 1998. Hong, P., Wen, Z., Huang, T., “Real-time speech-driven face animation with expressions using neural networks”, IEEE Trans. On Neural Networks, Vol.13, No.4, pp.916- 927, July 2002. Gutierrez-Osuna, R., Kakumanu, P., et al., “Speech-driven facial animation with realistic dynamics”, IEEE Trans. On Multimedia, Vol. 7, No. 1, pp. 33-41, Feb. 2005 .

  However, since there is a non-linear relationship between the acoustic features and the shape of the lips, there is still nothing that works properly. In the control of a real robot or an android (a robot having a form resembling that of a human and performing a movement resembling a human), the response of the actuator varies depending on the operation. For example, the opening operation is slower than the closing operation. Therefore, it is difficult to perform one-to-one mapping between the acoustic features and the target lip shape for each frame as in the related art.

  There are also the following problems. In utterance, a sound is produced by the passage of air after the shape of the lips, tongue, etc. is determined. In the case of silent plosives (/ p /, / t /, / k /), the shape of the lips and tongue is determined prior to the sound. Therefore, moving the lips based on the sound results in an unnatural operation. In other words, the process of controlling the facial motion mainly based on the lips based on the sound inevitably involves a delay.

  Also, the face of an Android moves differently than a human face. This is because it is impossible with the current technology to incorporate an actuator in the same way as a human facial muscle. In addition, since the reaction speed and the like are different, it is impossible to make the person move exactly the same. In general, there are many movements that are possible with humans but not with Android.

  Therefore, a main object of the present invention is to provide a speech operation control system for a remotely operated android that can perform a lip motion adapted to the voice of the operator in the android.

The invention of claim 1 is a system for controlling an utterance operation of an Android remotely operated by an operator according to an utterance voice of the operator, and a voice acquisition means for acquiring voice data of a voice uttered by the operator Lip shape estimating means for estimating the lip shape for generating the sound data from the acoustic features of the acquired sound data using a non-linear model; after issuing an operation command for forming the lip shape, the lip shape is An operation delay estimating means for estimating an operation delay indicating time information required for formation; a value larger than a value of the maximum operation delay estimated by the operation delay estimating means after the sound data is obtained by reproducing the sound data; audio reproducing means for starting after a lapse of the reproduction delay time, based on the reproduction start timing of the audio reproducing means, and operation slow earlier than the playback start timing Comprising an operation instruction issuing means for issuing each operation command in accordance with issuance timing set by the operation command setting means, and the operation command setting means for setting the timing of issuing an operation command based on a system.

  In the first aspect of the invention, the system (10: reference numeral corresponding to an embodiment to be described later, the same applies hereinafter) is a speech operation of an Android (12) that is remotely operated, that is, a facial operation mainly using a lip during speech. Is controlled according to the voice of the operator. The system includes, for example, a remote control terminal (20) and an android control device (14), and includes the following means. The voice uttered by the operator is acquired by the voice acquisition means (24, 70, 74, 78, S3). The sound reproduction means (70, 80, 48, S7, S9) reproduces the acquired sound with a certain time delay. The lip shape estimation means (70, S5, S21-S27) estimates the lip shape from the acquired acoustic features using a nonlinear model. For example, the MFCC coefficient is applied as the acoustic feature, and the lip shape is estimated using the acoustic feature for a predetermined time before and after the time point when the acoustic feature exhibits a high fluctuation amount. Further, since there is a non-linear relationship between the audio signal and the lip shape, a non-linear model such as a neural network is used. The operation delay estimation means (70, S31) takes time from when the operation command for forming the lip shape is issued until the android actuator (40) is actually driven to form the lip shape on the face. Estimate informational operational delay. The operation command setting means (70, S37) sets the issuance timing of the operation command based on the estimated operation delay with reference to the reproduction start timing of the uttered voice reproduced with a certain delay. Specifically, the operation command issuance timing for the lip shape formed when a certain sound is produced is earlier than the timing at which the sound corresponding to the shape is output in consideration of the operation delay. The operation command issuing means (70, S39) issues each operation command in accordance with the set issue timing.

  Therefore, according to the first aspect of the present invention, after the android actuator is driven and a lip shape is actually formed on the face, sound corresponding to the lip shape is output. As described above, in the remotely operated android, it is possible to realize the lip movement adapted to the voice of the operator.

The invention of claim 2 is characterized in that the lip shape estimation means determines whether or not the variation amount of the acoustic feature exceeds the threshold value, and before and after the time point when the determination means determines that the variation amount of the acoustic feature exceeds the threshold value. 2. The system according to claim 1, further comprising an extraction unit configured to extract the voice data of the lip, and estimating a lip shape based on the extracted voice data.
The invention of claim 3 further comprises optimization means for performing optimization including simplification of operation through the section based on the time series of the lip shape of the section of the predetermined time estimated by the lip shape estimation means. Item 3. The system according to item 1 or 2.
The invention according to claim 4 is the system according to claim 3, further comprising re-estimation means for re-estimating the operation delay based on the optimized operation after the optimization by the optimization means.

In the invention of claim 3 , the optimizing means (70, S35) performs the reconstruction of the operation in the section based on the time series of the lip shape of the predetermined section, and optimizes the operation through the section. Done. The predetermined section is set to a time that includes a plurality of phonemes and word units, for example. Specifically, conversion such as simplification of operation, presentation using only important phonemes, relativity of operation amounts, conversion of operation speed, and the like is attempted through the section. Then, the issuance timing of each operation command included in the optimized lip motion is set, and the optimized lip motion is presented. Accordingly, it becomes possible to make the lip movement appear more natural and to perform the movement more quickly.

  According to the present invention, the actuator operation command is issued in consideration of the operation delay from the operation command issuance until the lip shape is formed on the basis of the reproduction start timing of the utterance voice. It is possible to perform the lip movement suitable for the uttered voice.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, an Android control system (hereinafter simply referred to as “system”) 10 of this embodiment includes an Android 12. The android 12 is a humanoid robot having a form (appearance or the like) that closely resembles a human, and performs an action (shake, behave, or speak) that resembles a human. The android 12 is connected to a control device 14, and an environmental sensor 16 is connected to the control device 14.

  The control device 14 is connected to the remote operation terminal 20 via a network 18 such as the Internet or a telephone communication line. The remote operation terminal 20 is a general-purpose computer such as a PC or PDA. A speaker 22, a microphone 24 and a monitor 26 are connected to the remote operation terminal 20. Although illustration is omitted, as a matter of course, the remote operation terminal 20 includes an input device such as a keyboard and a computer mouse. A program and data for controlling the operation of the remote operation terminal 20 are stored in a memory (not shown), and the overall operation of the remote operation terminal 20 is controlled by a CPU (not shown).

  FIG. 2 is a block diagram showing an electrical configuration of the android 12, the control device 14, and the environment sensor 16. As shown in FIG. Referring to FIG. 2, the android 12 includes a plurality of actuators (for example, air actuators) 40, and each actuator 40 is connected to a control board (actuator control board) 98 of the control device 14 (14b). The actuator 40 is provided for presenting the body movement of the android 12. Some actuators 40 are used to perform facial movements mainly on the lips of Android 12. For example, actuators 40 for moving the upper jaw, the lower jaw, the upper lip, the lower lip, and the left and right side surfaces (cheeks) of the mouth are provided. The other actuators 40 are used to move other parts or portions of the android 12 such as the joints, eyes, and abdomen. However, for the sake of simplicity, the compressor is omitted in this embodiment.

  The android 12 includes a tactile sensor 42, an eye camera 44, a collision sensor 46, a speaker 48, and a microphone 50. The tactile sensor 42, the eye camera 44, and the collision sensor 46 are connected to the sensor input / output board 100 of the control device 14 (14b).

  The tactile sensor 42 or the skin sensor is a touch sensor, for example, and constitutes a part of the tactile sense of the Android 12. That is, the tactile sensor 42 is used to detect whether or not a human or another object touches the Android 12. Output (detection data) from the touch sensor 42 is given to the MPU 90 via the sensor input / output board 100. The MPU 90 transmits detection data from the touch sensor 42 to the CPU 70 of the first computer 14a. Therefore, the CPU 70 can detect that a human or another object touches the Android 12. However, a pressure sensor can also be used as the tactile sensor 42. In such a case, it is possible to know not only whether or not a human or other object has touched the skin of Android 12, but also how to touch it (strength).

  The eye camera 44 is an image sensor and constitutes part of the vision of the Android 12. That is, the eye camera 44 is used to detect a video or an image viewed from the eyes of the Android 12. In this embodiment, data (image data) corresponding to a captured video (moving image or still image) of the eye camera 44 is given to the MPU 90 via the sensor input / output board 100. The MPU 90 transmits the image data to the CPU 70 of the first computer 14 a, and the CPU 70 not only detects the change in the captured video but also transmits the image data to the remote operation terminal 20 via the network 18. Then, the remote control terminal 20 outputs the received image data to the monitor 26. Therefore, the captured image of the eye camera 44 is displayed on the monitor 26.

  The collision sensor 46 determines whether a human or another object has collided with the android 12. The output (detection data) of the collision sensor 46 is given to the MPU 90 via the sensor input / output board 100. The MPU 90 transmits detection data from the collision sensor 46 to the CPU 70 of the first computer 14a. Therefore, the CPU 70 can detect that a human or another object has collided with the android 12.

  The speaker 48 and the microphone 50 are connected to the audio input / output board 80 of the control device 14 (14a). The speaker 48 outputs sound when the Android 12 speaks. When the operator or operator (hereinafter referred to as “remote operator”) of the remote operation terminal 20 directly speaks, the voice is output. Specifically, when the remote operator speaks through the microphone 24, the corresponding voice data is given from the remote operation terminal 20 to the control device 14 a (CPU 70) via the network 18. Then, the CPU 70 outputs the audio data from the speaker 48 via the audio input / output board 80. Note that when a predetermined operation programmed in advance is performed, synthesized speech is output from the speaker 48.

  The microphone 50 is a sound sensor and forms part of the hearing of the Android 12. The microphone 50 has directivity and is mainly used to detect the voice of a human (user) who interacts (communicates) with the Android 12.

  The control device 14 includes a first computer 14a and a second computer 14b. For example, the first computer 14a is a main computer, and the second computer 14b is a sub computer. In this embodiment, the control device 14 is constituted by two computers (14a, 14b). However, if the processing capability is high, it can also be constituted by one computer.

  The first computer 14 a includes a CPU 70, and a memory 74, a communication board 76, a LAN board 78, a voice input / output board 80 and a sensor input / output board 82 are connected to the CPU 70 via an internal bus 72. The memory 74 includes, for example, a main storage device such as a hard disk device (HDD), ROM, and RAM. Although detailed description is omitted, the memory 74 stores a program and data for controlling the entire operation of the control device 14, and particularly corresponds to a command name for the operation of the Android 12, for example. Control information for executing the operation indicated by the command name is stored.

  Here, the movement refers to a body movement such as swinging and behavior and a speech movement. Therefore, in this embodiment, the control information includes not only the control data for driving and controlling the actuator 40 but also the voice data of the synthesized voice regarding the utterance contents as necessary. However, the body movement includes a natural movement (unconscious movement). Representative examples of unconscious movements include blinking and breathing. In addition to such physiological movements, movements caused by human heels are also included in the unconscious movements. For example, the operation with a heel corresponds to, for example, an operation of touching hair, an operation of touching a face, or an operation of biting a nail.

  Such an operation is executed by the Android 12 in response to a stimulus from the outside (environment) or according to a command (remote operation command) from a remote operator.

  Returning to FIG. 2, the communication board 76 is an interface for data communication with another computer (in this embodiment, the second computer 14b). For example, the communication board 76 is connected to the communication board 96 of the second computer 14b described later using a cable (not shown) such as RS232C. The LAN board 78 is an interface for data communication with another computer (in this embodiment, the remote operation terminal 20) via the network 18. In this embodiment, the LAN board 78 is connected using a LAN cable (not shown).

  In this embodiment, each computer is described as configuring a wired network using a cable. However, the present invention is not limited to this, and a wireless network may be configured. And a wireless network may be configured.

  The voice input / output board 80 is an interface for inputting and outputting voice, and the speaker 48 and the microphone 50 of the Android 12 are connected as described above. The voice input / output board 80 converts the voice data given by the CPU 70 into a voice signal and outputs it to the speaker 48. Further, the voice input / output board 80 converts a voice signal input through the microphone 50 into voice data, and gives the voice data to the CPU 70.

  Although a detailed description is omitted, the control device 14a has a voice recognition function. Therefore, for example, when the content uttered by the human being to the android 12 is input through the microphone 50, the CPU 70 recognizes the speech content of the human speech by DP matching or the hidden Markov method. However, it is assumed that dictionary data for speech recognition is stored in the memory 74.

  The sensor input / output board 82 is an interface for giving outputs from various sensors to the CPU 70 and outputting control data to the various sensors. In this embodiment, the environmental sensor 16 is connected to the sensor input / output board 82, and the environmental sensor 16 includes an omnidirectional camera 60, a PTZ camera 62, and a floor sensor 64.

  The omnidirectional camera 60 is installed in a room or place (section) where the android 12 is arranged, and can photograph the entire room or place (360 degrees). Image data corresponding to the captured image of the omnidirectional camera 60 is given to the CPU 70. The CPU 70 not only detects a change in the captured video based on the image data, but also transmits the image data to the remote operation terminal 20. The remote operation terminal 20 outputs the received image data to the monitor 26. Therefore, the captured image of the omnidirectional camera 60 is displayed on the monitor 26.

  The PTZ camera 62 is a camera that can control (adjust) each of pan, tilt, and zoom according to an operator's instruction. For example, when the remote operator inputs pan, tilt, and zoom instructions, corresponding control signals (commands) are given from the remote operation terminal 20 to the first computer 14 a via the network 18. Then, the CPU 70 of the first computer 14a drives and controls the PTZ camera 62 according to the command. Image data corresponding to the video shot by the PTZ camera 62 is also given to the CPU 70. The CPU 70 not only detects a change in the captured video based on the image data, but also transmits the image data to the remote operation terminal 20. The remote operation terminal 20 outputs the received image data to the monitor 26. Therefore, the captured image of the PTZ camera 62 is also displayed on the monitor 26.

  In this embodiment, the captured images of the eye camera 44, the omnidirectional camera 60, and the PTZ camera 62 are displayed on the monitor 26 connected to the remote operation terminal 20 with the screen divided. Therefore, the remote operator can see the image of the line of sight of the Android 12 and the situation around the Android 12 by looking at the monitor 26.

  Although not shown, the floor sensor 64 or the floor pressure sensor includes a large number of detection elements (pressure-sensitive sensors), and the large number of detection elements are embedded (laid down) in the floor of the room or place where the android 12 is disposed. ). Therefore, based on the output from the floor sensor 64, whether or not there is a human around the android 12, the number of existing humans, the direction of the human viewed from the android 12, the distance between the android 12 and the human, and the like are known. be able to.

  The second computer 14 b includes an MPU 90, and a memory 94, a communication board 96, a control board 98, and a sensor input / output board 100 are connected to the MPU 90 via an internal bus 92.

  The memory 94 includes an HDD, a ROM, and a RAM, and the memory 94 stores a program and data for controlling the operation of the control device 14b. The communication board 96 is an interface for data communication with another computer (in this embodiment, the first computer 14a). The control board 98 is an interface for outputting control data for controlling a plurality of actuators 40 as control objects and inputting angle information (rotation angle, bending angle) from each actuator 40. Therefore, the MPU 90 can feedback control the plurality of actuators 40. However, the MPU 90 drives and controls each actuator 40 in accordance with an operation command (control data) from the CPU 70 of the first computer 14a.

  The sensor input / output board 100 is an interface for giving outputs from various sensors to the MPU 90 and outputting control data from the MPU 90 to various sensors. As described above, the tactile sensor 42, the eye camera 44, and the collision sensor 46 are connected to the sensor input / output board 100.

  In this embodiment, the control device 14 is provided separately from the android 12, but the control device 14 may be referred to as an android. Furthermore, it may be called an android including the environment sensor 16.

  In addition, the control device 14 and the remote operation terminal 20 are connected via the network 18 on the assumption that an operator is present in a remote place of the Android 12, but the operator is located at or near the place where the Android 12 exists. In such a case, the remote control terminal 20 can be directly connected to the control device 14 without going through the network 18.

  The Android 12 is arranged in a certain company, a certain event venue, or the like, and can work as a substitute for a human being. For example, the Android 12 functions as a receptionist or guide for a company or event venue, and responds to visitors. The android 12 performs a predetermined corresponding operation according to the external stimulus detected by the sensor group (42, 44, 46, 50, 60, 62, 64) of the android 12 and the environment sensor 16. The operation to be executed (executable) by the Android 12 according to the situation, that is, the control information to be instructed by the CPU 70 is determined in advance, and the contents thereof are stored in the memory 74.

  However, when the corresponding action according to the sensor group response is not stored in the memory 74 or when the android 12 cannot execute the corresponding action for the stimulus, such as when the stimulus from the outside cannot be recognized. The Android 12 calls a remote operator. That is, the control device 14 (CPU 70) notifies the remote operation terminal 20 of information that the Android 12 cannot be handled. For example, when the Android 12 cannot understand (speech recognition) a human question (utterance content), the called remote operator understands the human question and controls the operation of the Android 12 by remote operation.

  Further, as described above, since the captured images of the eye camera 44, the omnidirectional camera 60, and the PTZ camera 62 are displayed on the monitor 26 of the remote operation terminal 20, the remote operator has the Android 12 based on the captured images. It is possible to know the neighborhood, surroundings, and the appearance of a human being interacting with the Android 12. Therefore, even if there is no call from the android 12 (control device 14), the remote operator can remotely control the android 12 to execute a command operation as necessary.

  Instead of instructing the output of the predetermined synthesized voice, the remote operator can speak directly to the person by outputting his / her voice from the speaker 48 of the Android 12 by speaking to the microphone 24. That is, when the remote operator speaks, voice data corresponding to the voice input detected by the microphone 24 is acquired, and the voice data is given from the remote operation terminal 20 to the control device 14 via the network 18, and the speaker of the Android 12. 48.

  When a person interacting with the Android 12 speaks, the voice is input through the microphone 50 of the Android 12, and the corresponding voice data is transmitted from the control device 14 to the remote operation terminal 20 via the network 18 and from the speaker 22. Is output.

  Since the Android 12 is a robot having a shape very similar to that of a human and performing an operation very similar to that of a human, when outputting the speech voice of a remote operator, for example, the mouth is not moved or it is not related to the voice. Just moving your mouth gives a strong sense of incongruity to humans. Therefore, in the Android 12, the face mainly composed of the lips is operated according to the uttered voice of the remote operator that is output.

  The operation of the system 10 will be described with reference to the flowcharts shown in FIGS. FIG. 3 shows an example of the speech processing operation of the control device 14. The CPU 70 of the control device 14a repeatedly executes this utterance process at regular intervals.

  In step S1 of FIG. 3, it is determined whether audio data has been received. When the remote operator speaks, the voice data of the utterance voice acquired by the microphone 24 is transmitted from the remote operation terminal 20, so it is determined whether or not this voice data has been received via the network 18. The remote operation terminal 20 acquires the uttered voice as voice data at a predetermined sampling rate (for example, 8 kHz), and transmits the acquired voice data at a predetermined packet length (for example, 20 ms) at regular intervals.

  If “YES” in the step S1, the voice storing process is started in a step S3. The voice storage process is executed by the CPU 70 in parallel with other processes. By this voice storage process, the received voice data is sequentially stored in the memory 74. The voice storage process is terminated when speech data is no longer detected and voice data is no longer received.

  Subsequently, in step S5, the lip movement control process is started. The lip movement control process is executed by the CPU 70 in parallel with other processes. In this lip movement control process, the acquired utterance voice is analyzed, and the lip movement is controlled based on the voice. An example of the operation of the lip movement control process is shown in FIG.

  In step S7, it is determined whether or not a predetermined time has elapsed since the voice acquisition. In this embodiment, since the acquired uttered voice is reproduced with a certain amount of delay, the determination waits for a certain period of time from the acquisition (reception) of the voice data.

  If “YES” in the step S7, the sound reproduction process is started in a step S9. The audio reproduction process is executed by the CPU 70 in parallel with other processes. In this audio reproduction process, the acquired audio data is read from the memory 74 and applied to the audio input / output board 80, whereby the audio is output from the speaker 48 of the Android 12. The audio reproduction process is terminated when all the acquired audio data has been reproduced.

  If “NO” in the step S1, that is, if the utterance is not performed, the utterance process of FIG. 3 is ended as it is.

  An example of the operation of the lip motion control process started in step S5 will be described with reference to FIG. First, in step S21, the fluctuation amount of the acoustic feature is extracted.

  In the case of an object such as Android 12, it is difficult to control the lip shape for each frame like an image. Accordingly, first, the frequency and cepstrum of the remote operator's voice are analyzed to detect a position where the acoustic feature variation is high. The variation amount of the acoustic feature is calculated as, for example, a mean square error of a frame parameter (for example, MFCC) for a predetermined time before and after a certain time (for example, about 20 ms). Note that acoustic features acquired from the audio signal include LPC-Cepstral coefficient, MFCC coefficient, LSP coefficient, formant frequency, F0 (fundamental frequency), RMS (Root Mean Square), and the like. In the case of vowels, the formant frequency and the lip shape have an intuitive relationship, but accurate automatic detection is difficult. In this embodiment, MFCC coefficients that are often used in speech recognition are used.

  Next, in step S23, it is determined whether or not the fluctuation amount (MFCC mean square error or the like) exceeds a threshold value. By experiment, a threshold value is set for this variation so as to represent a change in phonemes. The peak position of the fluctuation amount exceeding the threshold is the basis for determining the operation command issuance time point of the Android 12.

  If “NO” in the step S23, the process returns to the step S21, and the process is repeated for the audio data having the next time as a base point.

  On the other hand, if “YES” in the step S23, an acoustic feature (for example, MFCC) is extracted from a sound for a predetermined time (for example, about 100 ms) before and after the time when a high variation amount of the acoustic feature is detected in a step S25, and the step S27. Then, the lip shape is estimated using a non-linear model. Examples of the estimation method include linear regression analysis, neural network, HMM, and KNN. In addition, there is a method of recognizing phoneme information once and mapping the lip shape in the phoneme, but since the accuracy of phoneme recognition is not so high, direct mapping of the lip shape is considered to be more efficient from the acoustic features . In addition, since there is a non-linear relationship between acoustic features and lip shape, a non-linear model such as a neural network is used. For this purpose, model learning is performed by using prerecorded video data or a lip shape database by motion capture, and information for nonlinear mapping by model learning is stored in the memory 74.

  Subsequently, control information for forming the estimated lip shape is set in step S29, and an operation delay is estimated in step S31. Specifically, regarding the control information of the actuator 40 of the Android 12, the static characteristics and dynamic characteristics of the actuator control are considered. That is, as a static feature, control information of the actuator 40 of the Android 12 for approaching a specific lip shape is manually acquired in advance, and a database in which the lip shape is associated with the control information is stored in the memory 74. Keep it. As dynamic features, when a lip is moved with a specific shape as a target, it takes time from when the command is issued until the Android 12 actually reaches the shape of the target (this is called an operation delay). ) Is obtained by experiment, and a database in which control information (lip shape) is associated with an operation delay is stored in the memory 74. In step S37, which will be described later, in order to synchronize with the voice based on the information of the operation delay, the time point at which the operation command is sent is advanced or delayed.

  Note that the delay time until the start of audio reproduction is determined based on the information on the operation delay. That is, as described above, in this embodiment, since the sound is always reproduced with a constant delay, the reproduction delay time is set to a value larger than the maximum operation delay value.

  In step S33, it is determined whether or not a predetermined time has been estimated. In this embodiment, since it is assumed that the lip movement is reconstructed in a range wider than the range from which the acoustic features are extracted, for example, a plurality of phonemes or words, the determination in step S33 is performed. If “NO” in the step S33, the process returns to the step S21 to repeat the process.

If “YES” in the step S33, the lip movement optimizing process is performed in the above-described predetermined time section in a step S35. That is, with respect to the voice of a relatively short period, the processing of step S21 and step S25 is performed, and the optimization of the operation is attempted through a longer section in which these voices are bundled. Since the estimated lip shape may not be completely reproduced by the Android 12, the lip movement is converted based on the time series of the estimated lip shape. For example, the following conversion can be considered.
(A) Simplification of operation: Move only some parts. The parts to be used differ depending on the operation, and the parts to be particularly emphasized differ depending on the sound and the action, and therefore the operations of the parts that can be omitted are omitted. For example, conversion is performed so that only the most characteristic movements such as the chin joint, the movement of the lips in the front-rear direction, or the movement of the side of the mouth are performed according to the sound and movement.
(B) Presentation with only important phonemes: Considering intonation and the like, only the highlighted and conspicuous actions are performed. For example, using prosodic features (pitch, strength, etc.), syllables with a high pitch, strong power, and long utterances are considered emphasized.
(C) Relativity of motion amount: Only the presence of motion is presented and the amount of motion is not allowed to follow. For example, instead of setting the magnitude of the movement of the designated actuator 40 as originally designated, if it is determined that the same effect can be obtained with a smaller number of movements, the movement is limited to a little. Change the amount of movement. As a result, a more natural operation can be shown and the operation can be performed quickly. Further, in order to make the movement stand out, the operation of the same part that is continuous in time is suppressed or amplified. For example, when the magnitude of the motion is expressed in 1-5 steps (5 = large), the motion sequence of 3 → 5 → 3 is executed for the single actuator 40, but the part “3” is not so important. At this time, it is replaced with 1 → 3 → 1 or 0 → 3 → 0. Since only the relative degree of movement is often important to the viewer, even if such a substitution is performed, it can appear the same. In this way, by suppressing or amplifying the operation for a certain period including the previous and subsequent operations, the operation can be performed quickly, and in some cases, a stronger effect such as showing a part to be emphasized more can be obtained. Can be given to the viewer.
(D) Conversion of motion speed: When a specific voice is emitted, the lip shape looks more natural when it shows its maximum change, so it quickly forms its target shape.

  In this optimization process, it is desirable to convert the operation to the simplest possible, taking into account delays, etc., but in order to make it appear more natural, not a few lip movements are necessary. The parameters are appropriately set by experiment. Note that optimization may not be performed depending on the spoken voice.

  If necessary, in step S35, after performing optimization, the operation delay of each operation in step S31 may be reestimated to obtain an appropriate delay for the optimized operation. Especially when the operation is amplified, it is necessary to re-estimate the operation delay. In addition, when motion suppression or speed conversion is performed, from the completion of the motion to the specific shape to the speech of the speech can be performed more quickly or at a more appropriate timing by re-estimating the motion delay.

  In another embodiment, the process of step S31 may be executed after step S35. In other embodiments, the processes of step S33 and step S35 may be omitted. That is, the estimated lip shape may be presented as it is.

  Subsequently, in step S37, the issuance timing of each operation command is set based on the operation delay with reference to the audio reproduction start timing. That is, the operation command issuance timing for forming a specific lip shape is set based on the audio reproduction start timing based on the estimated delay in order to synchronize with the audio. Basically, the operation command issuance timing is earlier than the playback time of the sound corresponding to the lip shape. This is not the case when the mouth is closed and the mouth is closed, and it may be delayed.

  In step S39, the operation command issuing process is started. The operation command issuing process is executed by the CPU 70 in parallel with other processes. In this operation command issuing process, when it is determined that the timing for issuing each operation command has come, the operation command is issued. Specifically, the CPU 70 outputs an operation command including actuator control information to the communication board 76 and gives it to the MPU 90 of the control device 14b. In response to this, the MPU 90 gives control information to the control board 98, whereby the actuator 40 specified in the control information is driven, and a lip shape corresponding to the control information is formed on the face of the Android 12. Become. As described above, since the operation timing at the time of forming each lip shape is considered in the issue timing, the sound corresponding to the lip shape is generated after the lip shape is actually formed on the face of the android 12. Is output. Further, the lip shape corresponding to the sound is changed after the output of the sound is finished. The operation command issuance process is terminated when it is determined that all the operation commands have been issued.

  In step S41, it is determined whether or not unprocessed audio data remains. If “YES”, the process returns to step S21 to repeat the process. In this way, in the android 12, the utterance voice of the remote operator is output with the lip movement adapted to the voice. On the other hand, if “NO” in the step S41, the lip movement control process is ended.

  According to this embodiment, the lip shape is estimated using the nonlinear model from the acoustic characteristics of the utterance voice of the remote operator, and the operation delay required until the lip shape is formed with reference to the reproduction start timing of the utterance voice is considered. Thus, the issuance timing of the operation command of the actuator 40 is set, so that the lip motion adapted to the voice of the remote operator in the Android 12 can be realized. Therefore, a natural dialogue can be performed without giving a sense of incongruity to the human being who receives the Android 12.

  In the above embodiment, the case where the utterance process is executed by the control device 14 has been described. However, in another embodiment, the remote operation terminal 20 executes the utterance process and issues an operation command to the control device 14. You may make it transmit. That is, for example, in FIG. 3, it is determined whether or not voice is detected by the microphone 24 in step S1. Further, when it is determined in step S7 that a predetermined time has elapsed since the acquisition (detection) of the sound, a reproduction instruction including the sound data is transmitted to the control device 14 in step S9. The control device 14 outputs the sound in response to reception of the reproduction instruction. Further, the remote operation terminal 20 transmits each operation command including control information to the control device 14 in the operation command issuing process started in step S39 of FIG. In response to receiving this operation command, the control device 14 controls the actuator 40 in accordance with the control information included in the operation command.

  In each of the above-described embodiments, the lip shape is estimated based only on the utterance voice of the remote operator. In other embodiments, the lip shape is estimated by combining the utterance voice and image processing. You may do it. That is, the remote operation terminal 20 is provided with a camera, the face of the remote operator who speaks is photographed, and the lip shape is detected by performing image processing on the photographed image. Then, the final lip shape is estimated based on the lip shape estimated from the uttered voice and the lip shape estimated from the captured image. Thereby, the accuracy of estimation of the lip shape can be improved. Alternatively, in another embodiment, the lip shape may be estimated by combining the speech sound and the motion capture. That is, the remote operation terminal 20 is combined with a motion capture system. Specifically, a marker is attached at an appropriate position on the face of the remote operator, and a plurality of cameras are provided. Then, the face of the speaking remote operator is photographed, the three-dimensional position of the marker is measured based on the photographed image, and the lip shape is detected based on the position of each marker. Then, based on the lip shape estimated from the speech voice and the lip shape estimated from the three-dimensional position, the final lip shape is estimated. This also improves the accuracy of lip shape estimation.

It is an illustration figure which shows an example of the android control system of this invention. It is a block diagram which shows the electrical structure of the android shown in FIG. 1, a control apparatus, and an environment sensor. It is a flowchart which shows an example of the operation | movement in the speech process of CPU70 shown in FIG. It is a flowchart which shows an example of operation | movement of the lip movement control process shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Android control system 12 ... Android 14 ... Control apparatus 20 ... Remote operation terminal 22, 48 ... Speaker 24, 50 ... Microphone 40 ... Actuator 70 ... CPU
74, 94 ... Memory 80 ... Voice I / O board 90 ... MPU
98… Control board

Claims (4)

A system for controlling an utterance operation of an Android remotely operated by an operator according to the utterance voice of the operator,
Voice acquisition means for acquiring voice data of the voice emitted by the operator;
Lip shape estimating means for estimating a lip shape for generating the voice data from the acoustic features of the acquired voice data using a non-linear model;
An operation delay estimating means for estimating an operation delay indicating time information taken from when an operation command for forming the lip shape is issued until the lip shape is formed;
Audio reproduction means for starting reproduction of audio data after the acquisition of the audio data and after the elapse of a reproduction delay time value larger than the value of the maximum operation delay estimated by the operation delay estimation means;
The operation command setting means for setting the issue timing of the operation command before the reproduction start timing and based on the operation delay with reference to the reproduction start timing by the audio reproduction means, and set by the operation command setting means A system comprising operation command issuing means for issuing each operation command according to an issue timing.   The lip shape estimating means determines whether or not the variation amount of the acoustic feature exceeds a threshold value, and the sound data before and after that time when the determination means determines that the variation amount of the acoustic feature exceeds the threshold value The system according to claim 1, further comprising extraction means for extracting lip, and estimating a lip shape based on the extracted voice data. The optimization unit according to claim 1, further comprising: an optimization unit that performs optimization including simplification of an operation through the section based on a time series of the lip shape of the section of a predetermined time estimated by the lip shape estimation unit. system.   The system according to claim 3, further comprising a re-estimation unit that re-estimates the operation delay based on the optimized operation after performing the optimization by the optimization unit.

Images