JP7185072B2 - Method and system for providing emotional modification during video chat - Google Patents

Description

TECHNICAL FIELD This disclosure relates generally to methods and systems for use during video chats, and in specific embodiments, to determine the consistency between one or more of a person's perceived emotions and a person's semantic emotions. The present invention relates to a method and system for modifying the video and audio signals of persons participating in a video chat to produce modified video and audio signals with enhanced video chat.

Drivers of automobiles and other types of vehicles often use their smartphones or other mobile computing devices to chat with others while driving. Such chats may be voice chats or video chats. For the purposes of this description, voice chat refers to audio-only communication, meaning that two people participating in the voice chat can hear each other, but cannot see each other. In contrast, a video chat refers to communication involving both audio and video between two people participating in a video chat, where the two people participating in the video chat can both hear and see each other. means Video telephony technology, which provides for receiving and transmitting audio and video signals, can be used to conduct video chats. Exemplary video telephony products include those from Apple Inc., to name a few. FaceTime® available from Microsoft, Google Duo and Google Hangouts, both available from Google LLC, Microsoft Corp. Skype available from Microsoft, as well as Tencent Corp. Includes WeChat available from In fact, 10% of drivers said they have used their smartphones to video chat while driving. This proportion is likely to increase in the future, especially as semi-autonomous and fully autonomous vehicles become more common.

Road rage, aggressive or angry behavior seen in vehicle drivers, is very common. In fact, research has shown that over the past year, the majority of drivers have expressed significant anger while driving. Road rage can have many kinds of direct harm. For example, for vehicle drivers and their passengers, road rage can lead to arguments, assaults and collisions that result in serious physical injury or even death. Roadrage can also have certain indirect negative effects. For example, a first person driving a first vehicle is participating in a video chat with a second person driving a second vehicle, during which the first driver experiences road rage. , the anger of the first person may be transmitted to the second person and/or otherwise distracted by the second person, thereby causing the second person to become involved in the conflict. may be at increased risk. In another example, a first person driving a first vehicle is participating in a business-related video chat with one or more other persons during which the first person is loading. If you feel rage, business relationships between the first person and one or more other persons may be impaired or otherwise adversely affected.

According to one aspect of the present disclosure, a method includes obtaining a video signal and an audio signal of a first person participating in a video chat with a second person; and determining the semantic emotion of the first person based on the audio signal. The method also modifies the video signal to increase consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. It also includes the step of

Optionally, in any of the preceding aspects, determining one or more types of perceived emotions of the first person based on the video signal comprises: detecting at least one of a facial expression or a body pose; and determining the facial expression of the first person based on at least one of the facial expression or the body pose of the first person. and determining at least one of a perceived emotion or a perceived emotion of body posture.

Optionally, in any of the preceding aspects, the step of determining one or more types of perceived emotion of the first person is also based on an audio signal, the pitch of the first person's speech; performing audio signal processing of the audio signal to determine at least one of vibrato or intonation; and determining the emotion. Such methods further include modifying the audio signal to increase the consistency between the perceived emotion of the first person's speech and the first person's semantic emotion.

Optionally, in any of the preceding aspects, modifying the video signal to produce a modified video signal comprises displaying images of the video signal corresponding to at least one of facial expressions or body poses. modifying the audio data to produce the modified audio signal, including modifying the audio data of the video signal corresponding to at least one of pitch, vibrato, or inflection; including.

Optionally, in any of the preceding aspects, the method comprises determining whether the second person has at least one of the one or more types of perceived emotions of the first person and the first person's meaning Participate in a video chat and send a modified video signal and a modified audio signal so that the first person's image and audio can be viewed and listened to with enhanced emotional integrity. providing to a subsystem associated with a second person who is

Optionally, in any of the preceding aspects, determining a semantic emotion of the first person based on the audio signal comprises performing natural language processing of the audio signal; and determining a semantic emotion of the first person based on the results of.

Optionally, in any of the preceding aspects, the step of determining one or more types of perceived emotion of the first person based on the video signal comprises determining the one or more types of perceived emotions of the first person based on the video signal. or to quantify the positivity and activity of the body posture of the first person based on the video signal. includes at least one of the steps of using a toric model of pose. Further, the step of determining the first person's semantic emotion based on the audio signal includes: determining the first person's linguistic positivity and activity based on the audio signal; Includes steps that use the model. In addition, the step of modifying the video signal to produce the modified video signal includes determining the positivity and activity of the first person's facial expressions and the positivity and activity of the first person's language. modifying the image data of the video signal to reduce the distance between or between the positivity and activity of the first person's body posture and the positivity and activity of the first person's language modifying the image data of the video signal so as to reduce the distance of .

Optionally, in any of the preceding aspects, determining the one or more types of perceived emotions of the first person is also based on an audio signal; Using a cyclic model of speech to quantify the positivity and activeness of a person's speech. To produce an audio signal modified to reduce the distance between the positivity and activity of the first person's speech and the positivity and activity of the first person's language, the method comprises: It may further comprise modifying audio data of the audio signal.

Optionally, in any of the preceding aspects, the method comprises determining whether the second person has at least one of the one or more types of perceived emotions of the first person and the first person's meaning Participate in a video chat and send a modified video signal and a modified audio signal so that the first person's image and audio can be viewed and listened to with enhanced emotional integrity. providing to a subsystem associated with the second person.

According to another aspect of the disclosure, a subsystem comprises one or more interfaces and one or more processors. The one or more interfaces are configured to receive video and audio signals of a first person participating in a video chat with a second person. One or more processors, communicatively coupled to the one or more interfaces, determine one or more types of perceived emotion of the first person based on the video signal, and determine the first person's perceived emotion based on the audio signal. is configured to determine the semantic emotion of a person of The one or more processors process the video signal to enhance consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. is also configured to change the The subsystem may also comprise one or more cameras configured to acquire video signals and one or more microphones configured to acquire audio signals.

Optionally, in any of the preceding aspects, the one or more processors determine the perceived emotion of the first person based on the video signal and the semantic emotion of the first person based on the audio signal. implement one or more neural networks configured to determine

Optionally, in any of the preceding aspects, the one or more processors combine at least one of the one or more types of perceived emotion of the first person with the semantic emotion of the first person. implement one or more neural networks configured to modify the video signal to increase the consistency between

Optionally, in any of the preceding aspects, to determine one or more types of perceived emotions of the first person based on the video signal, the one or more processors are configured based on the video signal to , detecting at least one of a facial expression or a body pose of a first person, and determining a facial expression of the first person based on at least one of the facial expression or the body pose of the first person; It is configured to determine at least one of a perceived emotion of facial expression or a perceived emotion of body posture.

Optionally, in any of the preceding aspects, the one or more processors perform audio signal processing of the audio signal to determine at least one of pitch, vibrato, or inflection of the first person's speech. and determining the perceived emotion of the first person's utterance based on the results of the audio signal processing, and comparing the perceived emotion of the first person's utterance with the semantic emotion of the first person. It is also configured to modify the audio signal to improve consistency between

Optionally, in any of the preceding aspects, the one or more processors modify at least one of facial expression or body posture to modify the video signal to produce a modified video signal. of an audio signal, corresponding to at least one of pitch, vibrato, or inflection, so as to modify the audio signal to modify the image data of the video signal and produce a modified audio signal, corresponding to configured to modify audio data;

Optionally, in any of the preceding aspects, the one or more processors perform natural language processing of the audio signal and determine the semantic emotion of the first person based on results of the natural language processing of the audio signal. configured to determine.

Optionally, in any of the preceding aspects, the one or more processors are configured, based on the video signal, to measure the facial annulus Using the postural ring model to quantify the positivity and activity of the body posture of the first person, based on the video signal, and using the cyclic model of posture, based on the audio signal, the first It is configured to use a cyclic model of language to quantify the positivity and activity of a person's language. In addition, the one or more processors reduce the distance between the positivity and activity of facial expressions of the first person and the positivity and activity of language of the first person, It is configured to modify the image data of the video signal in order to shorten the distance between the positivity and activity of the body posture of the person and the positivity and activity of the language of the first person.

Optionally, in any of the preceding aspects, the one or more processors, based on the audio signal, apply a cyclic model of speech to quantify positivity and activity of the first person's speech. to modify the audio data of the audio signal in order to reduce the distance between the positivity and activity of the first person's speech and the positivity and activity of the first person's language is also configured.

Optionally, in any of the preceding aspects, the subsystem determines whether the second person has at least one of the one or more types of perceived emotions of the first person and the meaning of the first person. Participate in a video chat and send a modified video signal and a modified audio signal to allow viewing of the first person's video and audio with enhanced emotional integrity. a transmitter configured to transmit to a subsystem associated with a second person who is

According to another aspect of the present disclosure, a non-transitory computer-readable medium, when executed by one or more processors, causes the one or more processors to participate in a video chat with a second person. obtaining video and audio signals of a first person; determining one or more types of perceived emotions of the first person based on the video signals; Determining a semantic emotion of one person and checking consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person modifying the video signal to enhance it; and storing computer instructions. The non-transitory computer-readable medium provides computer instructions that, when executed by one or more processors, cause the one or more processors to perform additional steps of the method summarized above and described in further detail below. can also be memorized.

SUMMARY This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. is not intended. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the background.

Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings, in which like reference numerals indicate like elements.

1 illustrates an example system that enables first and second persons to participate in a video chat; FIG. Allows a first and second person to participate in a video chat, where the audio and video of the first person being heard and seen by the second person is the actual audio and video of the first person FIG. 12 shows a system according to various embodiments of the present technology that also modifies the audio and video of at least a first person differently. Allows a first and second person to participate in a video chat, where the audio and video of the first person being heard and seen by the second person is the actual audio and video of the first person FIG. 12 shows a system according to various embodiments of the present technology that also modifies the audio and video of at least a first person differently. Allows a first and second person to participate in a video chat, where the audio and video of the first person being heard and seen by the second person is the actual audio and video of the first person FIG. 12 shows a system according to various embodiments of the present technology that also modifies the audio and video of at least a first person differently. FIG. 10 illustrates a modification subsystem in accordance with one embodiment of the present technology that may be used to modify the audio and video signals of a person participating in a video chat; Figure 6 shows further details of the emotion detector and emotion modifier of the modification subsystem introduced in Figure 5; It is a figure which shows a general ring model. It is a figure which shows the ring model of a face. It is a figure which shows the circular ring model of an attitude|position. It is a figure which shows the ring model of an utterance. How different kinds of perceived and semantic emotions can be mapped onto the cyclic model, how the distance between perceived and semantic emotions can be determined, and how such Fig. 2 illustrates how distance can be shortened to increase consistency between different types of perceived emotion and semantic emotion; To determine whether certain features of the video and audio signals are modified to increase consistency between the perceived and semantic emotions, FIG. 4 shows a high-level flow diagram describing how distance can be used; FIG. FIG. 4 shows a high-level flow diagram used to summarize a method according to certain embodiments of the present technology; FIG. 2 illustrates example components of an example mobile computing device in which embodiments of the present technology may be used;

Certain embodiments of the present technology modify the video and audio signals of a first person participating in a video chat with a second person such that the modified signals are When played back, what is seen and heard by the second person is different than the originally captured video and audio signals. While the first and second person are driving the vehicle, if both are participating in a video chat, such embodiments of the present technology provide that the first person is participating in the video chat. A first person's anger can be prevented from being transmitted to a second person if they feel road rage in between. If the first person driving the vehicle is participating in a business-related video chat with one or more other persons, such embodiments of the present technology allow the first person Anger can be prevented from being viewed by other persons, thereby impairing or otherwise adversely affecting the business relationship between the first person and one or more other persons avoid that. According to certain embodiments described in more detail below, one or more types of perceived emotions of the first person are incorporated into the video signal (and potentially the audio signal) of the first person. also), and the first person's semantic emotion may be determined based on the first person's audio signal. The first person's video and audio signals are then combined so that the resulting first person's modified video and audio are more sensitive to the first person's semantic emotion than the first person's perceived emotion. can be modified (also called altered) to be more consistent with More specifically, the video and audio signals are modified to shorten the difference between one or more types of perceived emotion of the person and the semantic emotion of the person.

Perceived emotion, as the term is used herein, generally refers to the emotions of a second person using the senses of a second person, e.g., using the second person's sight and hearing relates to the emotional state of the first person to become aware of. In contrast, semantic emotion, as the term is used herein, generally refers to the second to the oral language spoken by the first person (also called spoken language, or more succinctly language). Concerning the emotional state of a first person that a second person becomes aware of, using person understanding. For example, if a first person, during a conversation with a second person, says that they are having a good day, smiles and has positive body language, the perceived emotion is semantic emotion. can often be substantially matched. However, for example, the first person frowns and uses negative body language (such as looking down and folding their arms) while saying that they are having a great day during a conversation with a second person. , there may be a significant mismatch between perceived and semantic emotions. According to certain embodiments of the present technology, a first person frowns and uses negative body language while saying they are having a great day while video chatting with a second person. (such as looking down and folding arms), then when the video is played for the second person, the first person's body language has been transformed from negative to positive body language. As such, the first person's video has been altered so that the first person's body language is more consistent with the positive speech they used. In addition, the first person's audio is also used, for example, to transform the pitch, vibrato, and/or inflection of the first person's voice to be more consistent with the affirmative speech they used. , may be changed.

FIG. 1 shows an exemplary system that allows first and second persons to participate in a video chat. In FIG. 1, blocks 110A and 110B participate in a video chat using respective client computing devices, also referred to herein more generally as audio-video (AV) subsystems 120A and 120B. represents the first and second person AV subsystems 120 A and 120 B may be collectively referred to as AV subsystem 120 or individually as AV subsystem 120 . First person 110A and second person 110B may be collectively referred to as person 110 or individually as person 110. FIG. The AV subsystem 120A can acquire the video and audio signals of the first person 110A, and the AV subsystem 120B acquires the video and audio signals of the second person 110B. be able to. Accordingly, each AV subsystem 120 may comprise at least one microphone used to acquire audio signals and at least one camera used to acquire video signals. At least one camera is an RGB/NIR (near infrared) camera that includes an image sensor (e.g., CMOS image sensor) that can be used to capture multiple two-dimensional RGB/NIR images per second (e.g., 30 images per second). can do. At least one additional camera produces depth images rather than RGB/NIR images, for example reconstructing 3D structures in point clouds using structured light and/or time-of-flight (TOF) sensors. It can be a camera.

Additionally, the AV subsystem 120A can play video and audio of a second person (eg, 120B) for the first person 110A, and the AV subsystem 120B can play the video and audio of the second person 110A. Video and audio of the first person (eg, 120A) can be played for 110B. Accordingly, each AV subsystem 120 may comprise at least one audio-speaker used to output audible sound and at least one display used to display video images. One or both of AV subsystems 120A and 120B may be, but are not limited to, in-vehicle computer systems or mobile computing devices such as smart phones, tablet computers, notebook computers, laptop computers or the like. be able to. The possibility that one or both of the audio-video subsystems 120A and 120B, or portions thereof, may include microphones, cameras, audio speakers, and/or displays that are built into the vehicle, eg, as part of a vehicle entertainment system. There is also

When first person 110A and second person 110B are participating in a video chat using their respective AV subsystems 120A and 120B, at least one microphone of AV subsystem 120A is , acquires the audio signal of the first person 110A, and at least one camera of the AV subsystem 120A acquires the video signal of the first person 110A. Similarly, at least one microphone of the AV subsystem 120B captures the audio signal of the second person 110B and at least one camera of the AV subsystem 120B captures the video signal of the second person 110B. get. The audio and video signals of first person 110A acquired by AV subsystem 120A are transmitted over one or more communication networks 130 to AV subsystem 120B. Similarly, audio and video signals of second person 110B acquired by AV subsystem 120B are transmitted to AV subsystem 120B via one or more communication networks 130. FIG.

Communication network 130 can be, but is not limited to, any wired or wireless local area network (LAN) and/or wide area network (WAN), such as an intranet, extranet, or the Internet, or combinations thereof. not. Communication network 130 is sufficient to provide communication capabilities between AV subsystem 120 and, optionally, other devices and systems. In some implementations, communication network 130 uses Hypertext Transfer Protocol (HTTP) to transfer information using Transmission Control Protocol/Internet Protocol (TCP/IP). HTTP allows AV subsystem 120 to access various resources available through communication network 130 . However, various implementations described herein are not limited to the use of any particular protocol.

At least one audio-speaker of AV subsystem 120A uses audio signals of second person 110B to provide audible sounds (e.g., talking) of second person 110B that can be heard by first person 110A. word). At least one display of the AV subsystem 120A uses the video signal of the second person 110B to display a video image of the second person 110B that can be viewed by the first person 110A. Similarly, at least one audio-speaker of the AV subsystem 120B uses the audio signal of the first person 110A to provide the audible sound of the first person 110A ( for example, spoken words). At least one display of the AV subsystem 120B uses the video signal of the first person 110A to display a video image of the first person 110A that can be viewed by the second person 110B.

Conventionally, an unmodified version of the audio and video signal of the first person 110A (acquired by the AV subsystem 120A) is shown by the AV subsystem 120B (acquired by the second person 110B). using) to output and display the audio and video of the first person 110A to the second person 110B. Thus, when a first person 110A participates in a video chat with a second person 110B, an angry facial expression (e.g., frown), angry body posture (e.g., clasped upright), clenched fist), and an angry (e.g., high) tone, the second person 110B will see the angry facial expression and angry body posture of the first person 110A, and the first person 110A You will hear the angry tone of the Note that the term body posture, as used herein, also encompasses hand posture.

According to certain embodiments of the present technology, the audio and video signals of the first person 110A are modified before being provided to the AV subsystem 120B so that the actual appearance of the first person 110A is The audio and video signals of the first person 110A can be heard and viewed by the second person 110B differently than they are and how they are heard. Such modifications to the audio and video signals of first person 110A may be performed by the same AV subsystem that acquires the audio and video signals. More specifically, as shown in FIG. 2, the AV and modification subsystem 220A obtains the audio and video signals of the first person and presents the modified audio and video signals to the second person 110B. Such signals may be modified prior to providing such signals to the communication network 130 that provides them to the AV subsystem 120B that is in close proximity to them. Alternatively, such modifications to the audio and video signals of the first person 110A are performed by a further subsystem separate from the AV subsystem 120A that acquires the audio and video signals of the first person 110A. obtain. For example, as shown in FIG. 3, modification subsystem 320A may receive audio and video signals of a first person 110A, and modification subsystem 320A may transmit modified audio and video signals to a second person 110A. Such signals may be modified prior to providing such signals to communication network 130 serving AV subsystem 120B proximate person 110B. Another option is that the AV subsystem 120A (which obtains the audio and video signals of the first person 110A) transmits the audio and video signals of the first person 110A through one or more communication networks 130. to the modification subsystem 420A, which then provides the modified audio and video signals of the first person 110A after the modification subsystem 420 modifies such signals. may be provided to the AV subsystem 120B proximate to the second person 110B via the communication network 130. Other variations are possible and within the scope of the embodiments described herein. Although not shown in FIGS. 1-4, the video and audio signals of the second person 110B are also provided to a similar modification subsystem to modify the perceived emotion of the second person 110B. Such signals can be modified to appear more consistent with the semantic emotion.

A first person captured or otherwise obtained by AV subsystem 120A (in FIGS. 1, 3, and 4) or by AV and correction subsystem 220A (in FIG. 2) 110A's audio and video signals may also be referred to as first person 110A's captured audio and video signals. FIG. 5 shows a modification subsystem 520 that receives captured audio and video signals from AV subsystem 120A or that is part of AV and modification subsystem 220A. As shown in FIG. 5, modification subsystem 520 includes emotion detection block 530 (which may also be referred to as emotion detector 530) and emotion modification block 540 (which may also be referred to as emotion modifier 540). Emotion detector 530 may, for example, detect negative, positive and/or neutral emotions of first person 110A. Exemplary negative emotions include, but are not limited to anger, nervousness, distraction, and irritability. Emotion modifier 540 modifies the audio and video signals such that, for example, the one or more types of perceived emotions of first person 110A in the modified audio and video signals are neutral or positive emotions. can be fixed. Exemplary neutral or positive emotions include, but are not limited to, happy, calm, alert, and satisfied. Further details of emotion detector 530 and emotion modifier 540, according to specific embodiments of the present technology, are described below with reference to FIG.

Referring to FIG. 6, emotion detector 530 is shown as including face detection block 610 (also referred to as face detector 610) and facial expression recognition block 612 (also referred to as facial expression recognizer 612). ing. Emotion detector 530 is also shown to include a skeleton detection block 614 (also called skeleton detector 614) and a pose recognition block 616 (also called pose recognizer 616). As shown in FIG. 6, face detector 610 and skeleton detector 614 are shown receiving video signal 602 . Video signal 602 may be, for example, a video signal of first person 110A captured by AV subsystem 120A, and more specifically, one or more cameras thereof. Still referring to FIG. 6, emotion detector 530 includes audio signal processing block 624 (also called audio signal processor 624 or audio signal analyzer 624) and natural language processor 626 (also called natural language processor 626 or natural language analyzer 626). Also shown as including processing block 626 . As shown in FIG. 6, audio signal analyzer 624 and natural language analyzer 626 receive audio signal 622 . Audio signal 622 may be, for example, the audio signal of first person 110A captured by AV subsystem 120A, and more specifically its microphone. Video signal 602 and audio signal 622 are assumed to be digital signals unless specified otherwise. Interfaces 603 and 623 can receive video signal 602 and audio signal 622, for example, from a camera and microphone, respectively, or from one or more other subsystems.

According to particular embodiments, the face detector 610 can detect faces of people in images and can also detect facial features in images. Computer vision techniques already developed (or developed in the future) may be used by face detector 610 to detect such facial features. For example, the HSV (hue-saturation-value) color model, or some other computer vision technique, can be used to detect faces in images. A feature detection model or some other computer vision technique is used to identify facial features such as, but not limited to, eyes, nose, lips, chin, cheeks, eyebrows, forehead, and/or the like. obtain. Feature detection can also be used to detect wrinkles in specific facial regions such as the forehead, sides of the mouth, and/or around the eyes. In certain embodiments, a person's face and a person's facial features may be identified using a bounding box. Some features to be identified, such as eyes on a user's face, can be included within other features, in which case the including feature (e.g., face) is first identified and then included Contiguous bounding boxes may be used to identify features (eg, each eye of a pair of eyes). In other embodiments, a single bounding box may be used to identify each distinct feature. In certain embodiments, OpenCV (http://opencv.willowgarage.com/wiki/) computer vision library and/or Dlib algorithm library (http://opencv.willowgarage.com/wiki/) are used to identify these facial features and generate bounding boxes. http://dlib.net/),) can be used. In certain embodiments, the bounding box need not be rectangular, but rather may be another shape such as, but not limited to, an oval. In certain embodiments, machine learning techniques such as boosting may be used to increase confidence in detecting facial features (eg, eyes, nose, lips, etc.). More generally, the dataset may be used to train a deep neural network (DNN) and/or other computer model to detect facial features from images, after which the trained DNN and/or Or other computer models can be used for facial feature recognition.

Once the facial features are identified (also referred to as detection) by the face detector 610, a facial expression recognizer 612 can determine the facial expression of the person. Generally, the human face consists of different parts such as the chin, mouth, eyes and nose, as mentioned above. The shape, structure, and size of those facial features may vary for different facial expressions. In addition, wrinkles at specific facial locations may vary with certain facial expressions. For example, the shape of a person's eyes and mouth may be used to distinguish between different facial expressions such as a person's glabellar lines, and/or the like. Based at least in part on the person's detected facial expressions, one or more types of perceived emotions of the person may be determined by the perceived emotion detector 632 of FIG. Exemplary perceived emotions that may be detected based at least in part on the detected facial expression include anger, nervousness, distraction, and irritability, and happiness, calm, alertness, and contentment. , but not limited to. Certain techniques for quantifying perceived emotions are described below.

Skeletal detector 614 may use a skeletal detection model or some other computer model to identify body parts and joints, such as, but not limited to, arms, hands, elbows, wrists, and/or the like. Vision technology can be used. Posture Recognizer 616 may detect whether the person is holding the steering wheel of the vehicle with both hands while driving the vehicle, or whether the person is raising one arm with his or her fists while driving the vehicle. , the specific posture can be detected. The dataset can be used to train a deep neural network (DNN) and/or other computer model to detect human poses from images, and the trained DNN and/or other computer model can then be used to , can be used for pose recognition.

Once the body parts are detected in the image by the skeleton detector 614, the pose recognizer 616 can determine the pose of the person. Generally, the human body consists of different parts such as head, neck, torso, upper arm, elbow, forearm, wrist and hand. In a constant posture, the general and relative positions and orientations of such body parts may change. For example, while a person is driving a vehicle, that person often has both hands on the steering wheel of the vehicle. If the person becomes angry because of being made to change course and/or do the same, the person may raise one of their arms and make a fist. As can be seen from FIG. 6, the detected pose is also used to determine the person's perceived emotion, as represented by the line from pose recognizer 616 to perceived emotion detector 632. can be used.

As noted above, in FIG. 6, audio signal analyzer 624 and natural language analyzer 626 receive audio signal 622 . Audio signal 622 may be, for example, the audio signal of first person 110A captured by AV subsystem 120A. An audio signal analyzer 624 can analyze the audio signal 622 to detect various characteristics of the audio signal 622 that may change depending on the person's emotional state. Examples of such audio features include pitch, vibrato, and inflection. Pitch is related to the frequency of the signal and can therefore be quantified as frequency. Changes in the pitch of a person's voice are often correlated with a person's arousal state, or more generally with their emotional state. For example, an increase in pitch often correlates with a state of high arousal such as anger, joy, or fear, while a decrease in pitch often correlates with a state of low arousal such as sadness or calmness. Vibrato is a periodic modulation of the pitch (eg, fundamental frequency) of a person's voice that occurs at a given rate and depth. Vibrato is also related to jitter and is often correlated with emotional changes. Increased pitch variation, and thus vibrato, may indicate increased happiness, grief, or fear, for example. Inflection is a rapid modification of pitch at the beginning of each utterance, the modification overshooting its target by a few semitones but quickly decaying to normal values. The use of inflection results in additional pitch variation, which is associated with high emotional intensity and positive emotional valence. As can be seen from FIG. 6, the result of the audio signal analysis performed by the audio signal analyzer 624 is the human emotion, as represented by the line from the audio signal analyzer 624 to the perceived emotion detector 632 can also be used to determine the perceived emotion of As can be appreciated from the above description, constant changes in specific audio features can indicate an increase in either positive emotion (e.g., happiness) or negative emotion (e.g., anger). . For example, an increase in happiness or fear can both cause an increase in pitch. However, by analyzing multiple vocal features alone or in combination with facial expressions and/or body postures, it is possible to determine a relatively accurate perceived emotion of a person.

A natural language analyzer 626 performs natural language processing (NLP) on the audio signal 622 and the result is a person's meaning, as represented by the line from the natural language analyzer 626 to the semantic emotion detector 634 . can be used to determine emotional sentiment. NLP performed by natural language analyzer 626 can include speech recognition, which provides a textual representation of a person's speech. In natural speech, there are few pauses between successive words, so speech segmentation can be a subtask of speech recognition, where speech segmentation separates a person's audio clip into multiple words. including doing Natural language analyzer 626 may be configured to recognize a single language or multiple different languages such as English, Chinese, Spanish, French, and German, to name a few. If the natural language analyzer 626 can perform NPL for multiple different languages, the output of the natural language analyzer 626 can include an indication of the specific language the person is speaking.

Perceived emotion detector 632 determines one or more types of perceived emotions associated with the person based on the outputs of facial expression analyzer 612, pose recognizer 616, and audio signal analyzer 624. To do so, one or more look-up tables (LUTs) can be used. The output of the facial expression analyzer 612 may identify one or more facial expression characteristics of the person determined based on the person's video signal 602, and the output of the pose recognizer 616 may be the person's video. One or more body poses of the person determined based on the signal 602 can be identified, and the output of the audio signal analyzer 624 is one or more audio features determined based on the audio signal 622. identify. Alternatively, or in addition to using LUTs, the perceived emotion detector 632 may use facial expression training data, body posture training data, speech training data, and/or other perceived emotions. It can be implemented by one or more DNNs and/or one or more other computer models trained based on perceived emotional training data, which can include emotional training data.

The semantic emotion detector 634 can use one or more lookup tables (LUTs) to determine the perceived emotion associated with the person based on the output of the natural language analyzer 626. The output of the natural language analyzer 626, when determined based on the audio signal 622, can identify the words and sentences spoken by the person and can also indicate the language spoken. Alternatively, or in addition to using LUTs, the semantic emotion detector 634 may be trained by one or more DNNs and/or other computer models based on semantic emotion training data. can be implemented.

Still referring to FIG. 6, the outputs of perceived emotion detector 632 and semantic emotion detector 634 are also shown provided to emotion modification block 540, which is also referred to as emotion modifier 540. can be called Emotion modifier 540 is also shown receiving captured video signal 602 and captured audio signal 622 . Emotion modifier 540 is shown as including facial expression modifier block 642, posture modifier block 646, and audio modifier block 648, respectively, facial expression modifier 642, posture modifier 646, and audio modifier. Also known as 648. As described above, the perceived emotion detector 632 detects the detected facial expression, the detected body pose determined based on the video signal 602, and the detected emotion determined based on the audio signal 622. One or more types of perceived emotion of the person can be determined based on the perceived audio characteristics (eg, pitch, vibrato, and inflection). As also described above, the semantic emotion detector 634 determines the person's semantic emotion based on the person's spoken language using NLP.

According to certain embodiments of the present technology, the facial expression modifier 642 detects the perceived emotion of the person's facial expression (as determined by the perceived emotion detector 632) and (semantic emotion detection The facial expression image data of the video signal 602 is modified to be more consistent with the person's semantic emotion (determined by the detector 364). According to certain embodiments of the present technology, posture modifier 646 combines the perceived emotion of the person's body posture (determined by perceived emotion detector 632) and the modified) the image data of the video signal 602 so as to be more consistent with the person's semantic emotion. According to certain embodiments of the present technology, the audio modifier 648 combines the perceived emotion (as determined by the perceived emotion detector 632) of the person's utterances (as determined by the semantic emotion detector 634) ) modify the audio data of the audio signal 622 to be more consistent with the person's semantic emotion. Emotion modifier 540 is shown outputting modified video signal 652 and modified audio signal 662 .

Certain embodiments of the present technology may be used to determine how a person's emotions responsive to and/or induced by environmental factors interact with a person's emotions responsive to and/or induced by the context of a conversation and a particular feature space. and that the difference between emotions responsive to and/or evoked by environmental factors and emotions responsive to and/or evoked by the context of conversation can be quantified. depends on According to a particular embodiment, the feature space used to quantify the difference between perceived emotion and semantic emotion is the feature space defined by the arousal/emotional valence cyclic model. Yes, which was first developed by James Russell and published in an article entitled "A circumplex model of affect", Journal of Personality and Social Psychology, Vol. 39, No. 6, December 1980, pp. 1161-1178. was done. The arousal/valence cyclic model can also be more succinctly called a cyclic model, suggesting that emotions are distributed in a two-dimensional circular space containing the arousal dimension and the valence dimension. Arousal corresponds to the vertical axis, emotional valence corresponds to the horizontal axis, and the center of the circle corresponds to neutral emotional valence and moderate levels of arousal. In this model, emotional states can be represented by arbitrary levels of emotional valence and arousal, or neutral levels of one or both of these factors. James Russell and Lisa Feldman Barrett subsequently developed a modified arousal/emotional valence cyclic model, which they published in Journal of Personality and Social Psychology, Vol. 76, No. 5, May 1999. , pp. 805-819, in a paper entitled "Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant."

Perceived emotion detector 632, according to a particular embodiment of the present invention, uses one sensor to separately determine three types of perceived emotions based on facial expression, body posture, and speech. Use one or more arousal/emotional valence cyclic models. More specifically, in certain embodiments, a facial annulus model is used to determine the arousal and emotional valence associated with a person's facial expressions, and the arousal associated with a person's body posture. and emotional valence, a postural cyclic model is used, and an utterance cyclic model is used to define the arousal and emotional valence associated with human speech. The valence dimension is represented on the horizontal axis and ranges between positive and negative valences. Positive and negative emotional valences (along the horizontal axis) are also known as pleasant and unpleasant, respectively, or more commonly as positivity. The arousal dimension is represented on the vertical axis intersecting the horizontal "valence" axis and ranges between active and inactive. Activated and deactivated arousal (along the vertical axis) are also known as intense and non-intense arousal, respectively, or more commonly as activity. A general torus model is shown in FIG. 7A, a face torus model is shown in FIG. 7B, a pose torus model is shown in FIG. 7C, and a speech torus model is shown in FIG. 7D. .

According to certain embodiments of the present technology, feature vectors generated from facial expression detection, pose detection, and speech detection algorithms are input to the DNN. Facial expression detection may be performed by face detector 610 and facial expression analyzer 612, described above with reference to FIG. The result of facial expression detection can be one or more facial feature vectors. Pose detection may be performed by a skeleton detector 614 and a pose recognizer 616 . The result of body pose detection can be one or more pose feature vectors. Speech detection may be performed by audio signal analyzer 624 . The result of speech detection can be one or more speech feature vectors. According to a particular embodiment, the aforementioned feature vectors are concatenated together and fed to the DNN. Such a DNN can be used to implement the perceived emotion detector 632 of FIG.

According to certain embodiments of the present technology, the output of a DNN implementing the perceived emotion detector 632 is denoted as {aro _f , val _f , aro _p , val _p , aro _s , val _s }6 ``aro'' refers to arousal, ``val'' refers to emotional valence, and the subscripts f, p, and s refer to face, posture, and speech, respectively. Therefore, the arousal value and emotional value indicating the facial expression of the person, the arousal value and emotional value indicating the body posture of the person, and the arousal value and emotional value indicating the utterance of the person There exists a value of According to certain embodiments, these values are used to modify the person's facial expressions, body posture, and/or speech, as described in further detail below. The terms modification and alteration are used interchangeably herein.

According to certain embodiments of the present technology, a deep learning-based natural language processing (NLP) algorithm is applied to quantify a person's semantic emotion. The main idea is to use recognized utterances to determine the contextual dependence of emotions. In natural language processing, text instances are often represented as vectors in feature space. The number of features is often as high as hundreds of thousands and conventionally these features have known meanings. For example, whether the instance has a particular word previously observed in the training data, whether the word is listed as a positive/negative term in the emotional vocabulary, and so on. By using NLP algorithms, a person's semantic emotion can be estimated. According to a particular embodiment, the output of a DNN implementing semantic emotion detector 634 is two values denoted as {aro _sem , val _sem }, which can be collectively denoted as Emo _sem . According to certain embodiments, a person's semantic emotion Emo _sem is image data and audio data corresponding to a person's facial expressions, body posture, and/or speech, as described in further detail below. used to fix the

Each perceived emotion Emo ⁱ _perc denotes a point on the torus model, i={face, pose, speech}. This allows mapping of multiple types of perceived emotions to the cyclic model, as shown in FIG. As also shown in FIG. 8, the Semantic Emotion Emo _sem also shows points on the torus model that can be mapped to the same torus model. Referring to Figure 8, the "X" labeled 802 corresponds to the person's perceived facial emotion, and the "X" labeled 804 corresponds to the person's perceived posture emotion. and the "X" labeled 806 corresponds to the person's perceived utterance emotion. The positions of X802, X804 and X806 are defined by the six values noted above as {aro _f , val _f , aro _p , val _p , aro _s , val _s }. More specifically, the "X" position labeled 802 is defined by the values aro _f and val _f , and the "X" position labeled 804 is defined by the values aro _p and val _p . and labeled 806 is defined by the values aro _s and val _s . Still referring to FIG. 8, the dot labeled 808 corresponds to the person's semantic emotion Emo _sem . The position of the dot labeled 808 is defined by the values aro _sem and val _sem .

と意味的感情Emo_semとの間の距離dist_iは、以下の式を使用して計算され得る。

As mentioned above, activity is a measure of arousal and positivity is a measure of emotional valence. any perceived emotion

The distance dist _i between and the semantic emotion Emo _sem can be calculated using the following formula.

と意味的感情Emo_semとの間の距離は、知覚された感情と意味的感情とが、どの程度近くに整合しているかを示す。例えば、具体的な知覚された感情（例えば、身体の姿勢）と意味的感情との間の距離が比較的小さい場合、これは、知覚された感情が意味的感情と実質的に整合していることを示す。逆に、具体的な知覚された感情（例えば、身体の姿勢）と意味的感情との間の距離が比較的大きい場合、これは、知覚された感情が意味的感情と実質的に整合していないことを示す。特定の実施形態によれば、それぞれの種類の、判定された、知覚された感情に対して、知覚された感情と意味的感情との間の距離が判定される。これにより、顔の表情、身体の姿勢、および発話のそれぞれに対して1つずつ、3つの距離値が判定される。判定された距離が、指定された距離閾値を超える場合、知覚された感情が意味的感情と実質的に整合していないと判定され、その判定に応答して、知覚された感情と意味的感情との間の整合性を高めるために、それぞれの特徴（例えば、顔の表情、身体の姿勢、または発話）が修正される。より多くの例として、（図8において802とラベル付けされた「X］によって表される）顔の知覚された感情と（図8において808とラベル付けされたドットによって表される）意味的感情との間の判定された距離が、指定された距離閾値よりも大きい場合、そのときには、顔の知覚された感情が意味的感情と、より整合している、修正されたビデオ信号を作り出すように、ビデオ信号の顔の画像データが修正される。逆に、顔の知覚された感情と意味的感情との間の判定された距離が指定された距離閾値未満（以内とも呼ばれる）である場合、そのときには、ビデオ信号の顔の画像データは修正されない。この、それぞれの距離の判定、および判定された距離と距離閾値との比較は、身体の姿勢ならびに発話に対しても判定される。そのような比較の結果は、ビデオ信号の身体の姿勢データ、および／またはオーディオ信号の発話データを修正するか否かを判定するために使用される。 any perceived emotion

and the semantic emotion Emo _sem indicates how closely the perceived emotion and the semantic emotion match. For example, if the distance between a concrete perceived emotion (e.g., body posture) and a semantic emotion is relatively small, this indicates that the perceived emotion is substantially aligned with the semantic emotion. indicates that Conversely, if the distance between the concrete perceived emotion (e.g., body posture) and the semantic emotion is relatively large, this indicates that the perceived emotion is substantially aligned with the semantic emotion. indicates no According to a particular embodiment, for each type of determined perceived emotion, the distance between the perceived emotion and the semantic emotion is determined. This determines three distance values, one each for facial expression, body posture, and speech. If the determined distance exceeds a specified distance threshold, then it is determined that the perceived emotion is not substantially aligned with the semantic emotion, and in response to that determination, the perceived emotion and the semantic emotion are separated. Each feature (eg, facial expression, body posture, or speech) is modified to increase consistency between As more examples, the perceived emotion of the face (represented by the "X" labeled 802 in Figure 8) and the semantic emotion (represented by the dot labeled 808 in Figure 8) is greater than a specified distance threshold, then the perceived emotion of the face is more consistent with the semantic emotion to produce a modified video signal. , the image data of the face in the video signal is modified.Conversely, if the determined distance between the perceived emotion and the semantic emotion of the face is less than (also called within) a specified distance threshold, The facial image data of the video signal is then not modified, and this respective distance determination and comparison of the determined distance to the distance threshold is also determined for body pose and speech. The results of such comparisons are used to determine whether to modify the body pose data of the video signal and/or the speech data of the audio signal.

The foregoing distance determination and comparison is summarized in the flow diagram of FIG. Referring to FIG. 9, in step 902 the distance between one of the perceived emotions (face, pose, speech) and the semantic emotion is determined, more specifically, for example, the formula is calculated using At step 904 the calculated distance is compared to a distance threshold, and at step 906 there is a determination whether the calculated distance is within (ie, less than) the distance threshold. If the calculated distance is not within the distance threshold (i.e., the answer to the determination at step 906 is no), then flow proceeds to step 908, where the associated The signal, or part thereof, is modified in step 908 . If the calculated distance is within the distance threshold (i.e., the answer to the decision at step 906 is yes), then flow proceeds to step 910 without altering the relevant signal or portion thereof. proceed to The steps summarized above can be performed for each different type of perceived emotion, including faces, postures, and speech.

According to certain embodiments of the present technology, the first person's video and audio is modified by generating a composite image/audio to replace its original version. More specifically, the originally acquired video and audio signals of the first person are the modified video and audio signals that, when viewed by the second person (or several other persons) to produce a modified video and audio signal having a perceived emotion that is more consistent with the semantic emotion of the first person. According to certain embodiments, the perceived emotion of the generated image/audio should approach the semantic emotion as closely as possible.

Referring back to FIG. 6, emotion modifier 540, which may also be more specifically referred to as perceived emotion modifier 540, includes facial expression modifier 642, posture modifier 646, and audio modifier 648. shown as including Each of the modifiers 642, 646 and 648, which can also be referred to as modules, use algorithms to generate synthetic images or synthetic audio by modifying specific data within the captured audio and video signals. do. As a simple example, the first person's semantic emotion is happy, but their face's perceived emotion is nervous, their posture's perceived emotion is upset, and their speech perception is Suppose that the perceived emotion is determined to be mental stress. Using embodiments of the present technology, facial image data is modified such that the person's facial expression is happy (rather than nervous), and posture image data is modified such that the person's body posture is and the audio data is modified so that the person's speech is happy (rather than nervous). Such modifications should be made in real time or near real time so that there is no perceptible delay in the video chat.

As noted above, one or more DNNs and/or other computer models may be used to modify the captured video and audio signals to produce modified video and audio signals. According to a specific embodiment, a generative adversarial network (GAN) is used to implement such modifications. A GAN is a deep neural network architecture that includes two neural networks, a generative neural network and a discriminative neural network, that are pitted against each other in competition (hence the term "adversarial"). Therefore, the generative neural network and the discriminative neural network can be viewed as sub-networks of the GAN neural network. A generative neural network generates candidates while a discriminative neural network evaluates the candidates. Competition has an effect on data distribution. A generative neural network can learn to map from the latent space to a data distribution of interest, while a discriminative neural network can distinguish the candidates produced by the generative neural network from the true data distribution. can. The goal of training a generative neural network is to increase the error rate of a discriminative neural network by creating new candidates that the discriminator considers to be unsynthetic (part of the true data distribution) (i.e., discriminative "Tricking" the device neural network). A known dataset serves as initial training data for the discriminative neural network. Training the discriminative neural network may include feeding the discriminative neural network with samples from a training data set until it achieves acceptable accuracy. A generative neural network may be trained based on whether the generative neural network succeeds in fooling the discriminating neural network. A generative neural network may be seeded with randomly sampled inputs sampled from a predetermined latent space (eg, a multivariate normal distribution). The candidates synthesized by the generative neural network can then be evaluated by the discrimination neural network. Backpropagation can be applied to both networks such that the generative neural network produces better images, while the discriminative neural network becomes better at flagging the composite image. The generation neural network can be, for example, a deconvolutional neural network, and the discrimination neural network can be, for example, a convolutional neural network. GANs should be trained before being used to modify signals during video chats.

Referring back to FIG. 6, facial expression modifier 642 may be implemented by a GAN. More specifically, GANs can be used to modify image data within a video signal to produce a modified video signal that can be used to display a realistic image of a person. , where the image has been modified to better match the perceived emotion of the person's face and posture to the person's semantic emotion. GANs can also be used to modify audio signals so that the perceived emotion of a person's speech is more consistent with the person's semantic emotion. In a specific embodiment, StarGAN may be used to perform image and/or audio modification. Y. A paper entitled "StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation," by Choi et al. , discusses how StarGAN has been used. The use of additional and/or alternative types of neural networks and/or other types of computer models are within the scope of the embodiments described herein.

Still referring to FIG. 6, a GAN can also be used to implement pose modifier 646 . Alternatively, a pre-trained visual generator model can be used to implement pose modifier 646 . As shown in FIG. 6, original video signal 602 is provided to skeleton detector 614 . Original video signal 602 may also be referred to as original image stream 602 . A skeleton detector 614 extracts skeleton information from the original image stream. Skeletal information can be represented as a vector X that stores all joint positions in a frame. According to one embodiment, vector X is combined with a semantic affective signal represented by vector e. These two vectors are concatenated into vector X and can be used as input for a pre-trained visual generator model. For example, but not limited to, convolutional layers, maximum pooling layers, deconvolutional layers, and batch normalization layers may be implemented in the pre-trained visual generator model. The output of the pre-trained visual generator model can be used to generate a modified video signal 652 containing modified body poses that are more consistent with the semantic emotion.

Still referring to FIG. 6, original audio signal 622 is shown as being provided to audio signal analyzer 624, as already described above. Features of the audio signal that can be modified to better match the perceived emotion corresponding to a person's speech with its semantic emotion include, but are not limited to, pitch, vibrato, and inflection. More specifically, the pitch may be shifted, where pitch shift refers to the multiplication of the pitch of the original speech signal by a factor α. Elevated pitch (α > 1) often correlates with highly arousal states such as happiness, while decreased pitch (α < 1) correlates with low emotional valence such as sadness. Vibrato is a periodic modulation of the pitch (fundamental frequency) of a voice that occurs at a given rate and depth. Vibrato, which is also associated with jitter, is frequently reported to correlate with high arousal and is an important marker of emotion even in single vowel sounds. Vibrato can be modified to change the perceived emotion associated with speech. Inflection is a rapid modification of pitch (eg, ~500ms) at the beginning of each utterance, the modification overshooting its target by a few semitones but quickly decaying to normal values. The use of inflection results in additional pitch variation, which is associated with high emotional intensity and positive emotional valence. Inflection can also be modified to change the perceived emotion associated with the utterance. Audio signals can also be filtered to alter the perceived emotion corresponding to speech, where filtering refers to the process of enhancing or attenuating the energy contribution of particular regions of the frequency spectrum. For example, emotions of high arousal tend to be associated with increased high frequency energy, making speech sharper and brighter. If a person's semantic emotion corresponds to a less activated arousal than the person's perceived emotion, which corresponds to the utterance, then in order to better align the perceived emotion with the semantic emotion, can be attenuated using filtering. The emotional tone of the modified audio signal should be recognizable and the voice should sound natural and not be perceived as synthetic. As noted above, the terms modification and alteration are used interchangeably herein.

One or more processors may be used to implement the neural networks described above. Where multiple processors are used, they may be collocated or widely interlocked, or a combination thereof.

The high-level flow diagram of FIG. 10 is now used to summarize the method according to certain embodiments of the present technology. Referring to FIG. 10, step 1002 includes obtaining video and audio signals of a first person participating in a video chat with a second person. Referring back to FIGS. 1-4, step 1002 performs an AV subsystem (eg, 120A), or more specifically, one or more cameras and one or more microphones of the AV subsystem. , or by some other subsystem or system.

Referring again to FIG. 10, Step 1004 includes determining one or more types of perceived emotions of the first person based on the video signal. Step 1006 includes determining a semantic emotion of the first person based on the audio signal. The types of perceived emotion that may be determined in step 1004 include perceived emotion of facial expressions, perceived emotion of body posture, and perceived emotion of speech, as described above.

Referring briefly back to FIGS. 5 and 6, various types of perceived emotions may be detected by, for example, emotion detector 530 or, more specifically, the perceived emotion detector of emotion detector 530. 632. More specifically, based on the video signal obtained in step 1002, the facial expression and body posture of the first person can be determined, and based on these, the facial expression of the first person can be determined. The perceived emotion of facial expressions and the perceived emotion of body posture can be determined. Additionally, audio signal processing of the audio signal obtained in step 1002 can be performed to determine at least one of pitch, vibrato, or inflection of the first person's speech, The perceived emotion of the person's utterance may be determined based on the results of the audio signal processing. Additional and/or alternative variations are possible and within the scope of the embodiments described herein.

According to a specific embodiment, in step 1004, a facial torus model is used to quantify the positivity and activity of the facial expression of the first person based on the video signal, the video A cyclic model of posture is used to quantify the positivity and activeness of the body posture of the first person based on the signal, and the positivity of the speech of the first person based on the audio signal. and a cyclic model of speech is used to quantify activity.

The semantic emotion determined in step 1006 may be determined, for example, by emotion detector 530 , and more specifically by semantic emotion detector 634 of emotion detector 530 . As described in further detail above, Step 1006 performs natural language processing of the audio signal and determines the semantic emotion of the first person based on the results of the natural language processing of the audio signal. and According to a specific embodiment, in step 1006, the first person's semantic emotion is used in a linguistic circle to quantify the first person's linguistic positivity and activity based on the audio signal. It is determined based on the audio signal using an annulus model.

Referring again to FIG. 10, step 1008 is to increase the consistency between at least one of the one or more types of perceived emotions of the first person and the semantic emotions of the first person. includes modifying video and audio signals. As detailed above, one or more computer-implemented neural networks may be used to perform step 1008 . Other types of computer-implemented models may alternatively or additionally be used to implement step 1008 . Step 1008 includes modifying at least one of facial expression and body posture of image data included in the video signal and modifying pitch, vibrato, or intonation of audio data included in the audio signal. can contain.

According to a specific embodiment, step 1008 is performed to reduce the distance between the positivity and activity of the first person's facial expressions and the positivity and activity of the first person's language. includes modifying the image data contained in the video signal. Step 1008 extracts the image data contained in the video signal to reduce the distance between the positivity and activity of the first person's body posture and the positivity and activity of the language of the first person. can also include changing the In addition, step 1008 removes the audio data contained in the audio signal to reduce the distance between the positivity and activity of the first person's speech and the positivity and activity of the first person's language. can also include changing the

Still referring to FIG. 10, step 1010 sends the modified video signal and modified providing (e.g., transmitting) an audio signal so that the second person communicates with at least one of the one or more types of perceived emotions of the first person and the emotion of the first person It is possible to view modified images and audio of the first person that are more consistent with the person's semantic emotion.

For example, the methods described above with reference to FIGS. 9 and 10 may be applied to in-vehicle computer systems or mobile computing devices such as, but not limited to, smart phones, tablet computers, notebook computers, laptop computers, or the like. can be performed at least in part by Such method steps may be performed solely by a mobile computing device or by a mobile computing device in network communication with one or more servers over one or more communication networks. FIG. 11 illustrates exemplary components of an exemplary mobile computing device that may be used with embodiments of the present technology. Such a mobile computing device may be used, for example, but not by way of limitation, to implement an AV subsystem (eg, 120A or 220A in FIGS. 1-4).

FIG. 11 shows an exemplary mobile computing device 1102 that can be used with the embodiments of the technology described herein. The mobile computing device 1102 can be a smart phone, such as, but not limited to, an iPhone®, Blackberry™, Andriod™-based, or Windows™-based smart phone. Mobile computing device 1102 may alternatively be a tablet computing device, such as, but not limited to, an iPad®, Andriod-based, or Windows™-based tablet. As another example, mobile computing device 1102 may be an iPod® Touch™, or the like.

Referring to the block diagram of FIG. 11, mobile computing device 1102 includes camera 1104, accelerometer 1106, magnetometer 1108, gyroscope 1110, microphone 1112, display 1114 (which may or may not be a touch screen display), It is shown to include processor 1116 , memory 1118 , transceiver 1120 , speaker 1122 and drive unit 1124 . Each of these elements are shown connected to a bus 1128 that allows the various components to communicate with each other and to transfer data from one element to another. It is also possible that some elements may communicate with each other without using bus 1128 .

Camera 1104 may be used to capture video signals, including images of people using mobile computing device 1102 . Microphone 1112 may be used to produce an audio signal indicative of what is being spoken by the person using mobile computing device 1102 .

The accelerometer 1106 can be used to measure linear acceleration with respect to a coordinate system, thus detecting movement of the mobile computing device 1102, as well as the angle of the mobile device 1102 with respect to the horizon or ground. can be used for Magnetometer 1108 can be used as a compass to determine the direction of magnetic north and the bearing relative to magnetic north. The gyroscope 1110 can be used to detect both vertical and horizontal orientations of the mobile computing device 1102 and, together with the accelerometer 1106 and magnetometer 1108, can provide highly accurate orientation of the mobile computing device 1102. can be used to obtain information. Mobile computing device 1102 may also include additional sensor elements such as, but not limited to, ambient light sensors and/or proximity sensors.

The display 1114, which may or may not be a touchscreen display, is used as a user interface for visually displaying items (eg, images, options, instructions, etc.) to the user and for receiving input from the user. obtain. Display 1114 may also be used to allow users of mobile computing devices 1102 to participate in video chats. In addition, mobile computing device 1102 may include additional elements such as keys, buttons, trackpads, trackballs, or the like for receiving input from a user.

The memory 1118 may be used, but not limited to, to store images captured using the camera 1104, as well as software and/or firmware that control the mobile computing device 1102. . A variety of different types of memory may be included in mobile computing device 1102, including non-volatile and volatile memory. A drive unit 1124, such as but not limited to a hard drive, may be used to store images captured using the camera 1104 as well as software that controls the mobile computing device 1102. , but not limited to. Memory 1118 and disk unit 1124 are machine-readable media in which one or more sets of executable instructions (eg, applications) that embody one or more of the methods and/or functions described herein are stored. can include Instead of or in addition to the drive unit 1124, the mobile computing device may include solid state storage devices such as flash memory or any form of non-volatile memory. The term "machine-readable medium" as used herein refers to any one of, for example, centralized or distributed databases and/or associated caches and servers, storage drives (including, for example, magnetic and optical drives and storage mechanisms), etc. a single medium or medium in all forms such as the above storage devices and one or more instances of memory devices or modules (whether main memory, cache storage either internal or external to the processor, or buffers); Any of a plurality of media should be construed to include all forms of storage media. The terms "machine-readable medium" or "computer-readable medium" mean any tangible, non-tangible medium capable of storing or encoding a sequence of instructions for execution by a machine and causing the machine to perform any one method. should be construed to include transitory media. The term "non-transitory media" includes all forms of storage drives (optical, magnetic, etc.) and all forms of memory devices (e.g., DRAM, Flash (of all storage designs), SRAM, MRAM, phase change, etc.), as well as all other structures designed to store any kind of information for later retrieval.

A transceiver 1120 connected to an antenna 1126 may be used to wirelessly transmit and receive data using, for example, Wi-Fi, cellular communications, or mobile satellite communications. Mobile computing device 1102 may be capable of implementing wireless communications using Bluetooth and/or other wireless technologies. Mobile computing device 1102 may also include multiple types of transceivers and/or multiple types of antennas. Transceiver 1120 can include a transmitter and a receiver.

The speaker 1122 is used to provide auditory instructions, feedback and/or indicators to the user, playback of recordings (eg, music recordings), and to enable the mobile computing device 1102 to operate as a mobile phone. obtain. Speakers 1122 may also be used to allow users of mobile computing devices 1102 to participate in video chats.

Processor 1116 may be used to control various other elements of mobile computing device 1102 under the control of software and/or firmware stored in memory 1118 and/or drive unit 1124, for example. There may also be multiple processors 1116, such as a central processing unit (CPU) and a graphics processing unit (GPU). Processor 1116 can execute computer instructions (stored on a non-transitory computer-readable medium) to cause the processor to perform steps used to implement embodiments of the technology described herein. can.

Certain embodiments of the technology described herein may be implemented using hardware, software, or a combination of both hardware and software. The software used is stored in one or more of the processor readable storage devices described above to program one or more processors to perform the functions described herein. Processor readable storage can include computer readable media such as volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and communication media. Computer-readable storage media may be implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Examples of computer readable storage media are RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices. or other magnetic storage device, or any other medium that can be used to store desired information and that can be accessed by a computer. One or more computer-readable media does not include a propagated, modulated, or transitory signal.

Communication media typically embody computer readable instructions, data structures, program modules, or other data in a propagated, modulated, or transitory data signal, such as a carrier wave or other transport mechanism. including any information delivery medium. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of the any of the above are also included within the scope of computer readable media.

In alternative embodiments, part or all of the software may be replaced with dedicated hardware logic components. For example, without limitation, illustrative types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System On Including systems on chip (SOC), complex programmable logic devices (CPLD), special purpose computers, etc. In one embodiment, software (stored on a memory device) that implements one or more embodiments is used to program one or more processors. One or more processors may communicate with one or more computer readable media/storage devices, peripherals and/or communication interfaces.

It is understood that the subject matter may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. rather, these embodiments are provided so that this subject matter will be thorough and complete, and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments that fall within the scope and spirit of the subject matter as defined by the appended claims. Moreover, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be apparent to those skilled in the art that the present subject matter may be practiced without such specific details.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions execute via a processor of a computer or other programmable instruction execution device to implement the functions/acts specified in one or more blocks of the flowchart illustrations and/or block diagrams. may be provided to the processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine for producing a mechanism for.

The description of this disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of this disclosure. The aspects of the disclosure herein best describe the principles and practical application of the disclosure, and enable those skilled in the art to comprehend the disclosure, including various modifications suitable for the particular uses envisioned. selected and described for

The disclosure has been described in conjunction with various embodiments. Other variations and modifications to the disclosed embodiments, however, can be understood and attained from a study of the drawings, the disclosure, and the appended claims, and such variations and modifications come within the scope of the appended claims. should be construed as included. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality.

It is noted that dimensions of the various features illustrated in the figures herein may not necessarily be drawn to scale.

In this specification, references to "an embodiment," "one embodiment," "some embodiments," or "another embodiment" References herein may be used to describe different embodiments or the same embodiment.

As used herein, a connection may be a direct connection or an indirect connection (eg, via one or more other parts). Occasionally, when an element is referred to as being connected or coupled to another element, that element may be directly connected to the other element or indirectly to the other element through intervening elements. may be connected to When an element is referred to as being directly connected to another element, then there are no intervening elements between that element and the other element. Two devices are "communicating" when they are directly or indirectly connected such that electronic signals can be conveyed between them.

As used herein, the term "based on" may be interpreted as "based at least in part on."

In this specification, unless there is additional context, the use of numerical terms such as "first" object, "second" object, and "third" object does not imply order of objects. It may alternatively be used for identification purposes to identify different objects.

The foregoing detailed description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the subject matter claimed herein to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments best illustrate the principles of the disclosed technology and its practical application, thereby enabling those skilled in the art to include various modifications suitable for the particular uses envisioned in the various embodiments. selected to allow the best use of the technology. It is intended that the scope be defined by the claims appended hereto.

While the subject matter has been described in language specific to structural features and/or methodological acts, the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. It should be understood that the Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

110A First person
110B Second person
120A Audio-Video Subsystem (Audio-Video (A-V) Subsystem, A-V Subsystem, AV Subsystem)
120B Audio-Video Subsystem (Audio-Video (A-V) Subsystem, A-V Subsystem)
130 Communications Network
220A Audio-Video and Correction Subsystem (AV and Correction Subsystem, AV and Correction Subsystem)
320A correction subsystem
420A correction subsystem
520 correction subsystem
530 Emotion Detector (Emotion Detection Block)
540 Emotion Modifier (Emotion Modifier Block)
602 video signal
603 interface
610 Face Detector (Face Detection Block)
612 Facial Expression Recognizer (Facial Expression Recognition Block, Facial Expression Analyzer)
614 Skeleton Detector (Skeleton Detection Block)
616 Posture Recognizer (Posture Recognition Block)
622 audio signal
623 interface
624 Audio Signal Analyzer (Audio Signal Processing Block, Audio Signal Processor)
626 Natural Language Analyzer (Natural Language Processing Block, Natural Language Processor)
632 Perceived Emotion Detector
634 Semantic Emotion Detector
642 Facial Expression Modifier (Facial Expression Modification Block)
646 Posture Corrector (Posture Correction Block)
648 Audio Modifier (Audio Modifier Block)
652 modified video signal
662 modified audio signal
802 Perceived Facial Emotions of a Person
804 Perceived Posture Emotions of a Person
806 Human Perceived Speech Emotion
808 Human Semantic Feelings
1102 mobile computing devices
1104 camera
1106 accelerometer
1108 Magnetometer
1110 gyroscope
1112 Microphone
1114 Display
1116 processor
1118 memory
1120 Transceiver
1122 speaker
1124 drive unit
1126 Antenna
1128 Bus

Claims (22)

obtaining video and audio signals of a first person participating in a video chat with a second person;
determining one or more types of perceived emotions of the first person based on the video signal;
determining a semantic emotion of the first person based on the audio signal;
modifying the video signal to increase consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person; A method comprising steps and . determining the one or more types of perceived emotions of the first person based on the video signal,
detecting a facial expression of said first person based on said video signal ; and
determining a perceived emotion of a facial expression of said first person based on said facial expression of said first person ; or
detecting a body pose of the first person based on the video signal; and a perceived motion of the body pose of the first person based on the body pose of the first person. or
detecting a facial expression and body posture of the first person based on the video signal; and based on the facial expression and the body posture of the first person, determining the perceived emotion of facial expressions and the perceived emotion of body posture of
2. The method of claim 1, comprising: wherein said step of determining said one or more types of perceived emotion of said first person is also based on said audio signal, wherein said step of determining said one or more types of perceived emotion of said first person is also based on said first person's speech pitch, vibrato, or inflection; performing audio signal processing of the audio signal to determine at least one; and determining perceived emotion of the first person's speech based on results of the audio signal processing of the audio signal. and
the method further comprises modifying the audio signal to increase consistency between the perceived emotion of the utterance of the first person and the semantic emotion of the first person;
3. The method of claim 2. The step of modifying the audio signal to produce the modified audio signal includes modifying audio data of the video signal corresponding to at least one of the pitch, vibrato, or intonation. and the method is
further comprising providing the modified audio signal to a subsystem associated with the second person participating in the video chat;
4. The method of claim 3. said modifying said video signal to produce a modified video signal comprises modifying image data of said video signal corresponding to at least one of facial expressions or body posture; the method is
further comprising providing the modified video signal to a subsystem associated with the second person participating in the video chat;
The method of Claim 1.
Determining the semantic emotion of the first person based on the audio signal comprises:
performing natural language processing of the audio signal;
and determining the semantic emotion of the first person based on results of the natural language processing of the audio signal. determining one or more types of perceived emotions of the first person based on the video signal,
using a facial torus model to quantify the positivity and activity of facial expressions of said first person based on said video signal; or said first person based on said video signal. using a cyclic model of posture to quantify the positivity and activity of the body posture of
for said step of determining said semantic emotion of said first person based on said audio signal to quantify positivity and activity of said first person's language based on said audio signal; using a cyclic model of language;
said step of modifying said video signal to produce a modified video signal;
of said video signal to reduce the distance between said positivity and said activity of said facial expression of said first person and said positivity and said activity of said language of said first person. modifying image data, or adjusting the distance between said positivity and said activity of said body posture of said first person and said positivity and said activity of said language of said first person; modifying image data of said video signal to shorten it;
7. A method according to any one of claims 1-6. The step of determining the one or more types of perceived emotions of the first person is also based on the audio signal, and based on the audio signal, the positivity of the first person's speech. and using a cyclic model of speech to quantify activity,
The method is modified to reduce the distance between the positivity and activity of the utterance of the first person and the positivity and activity of the language of the first person. further comprising modifying audio data of said audio signal to produce an audio signal;
8. The method of claim 7. 9. The method of claim 8, further comprising providing the modified video signal and the modified audio signal to a subsystem associated with the second person participating in the video chat. . one or more interfaces configured to receive video and audio signals of a first person participating in a video chat with a second person;
communicatively coupled to the one or more interfaces;
determining one or more types of perceived emotions of the first person based on the video signal;
determining a semantic emotion of the first person based on the audio signal;
modifying the video signal to increase consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person; A subsystem comprising one or more processors configured to: one or more cameras configured to acquire said video signal;
11. The subsystem of Claim 10, further comprising: one or more microphones configured to capture said audio signal. The one or more processors determine the one or more types of perceived emotions of the first person based on the video signal, and determine the semantic emotion of the first person based on the audio signal. 12. The subsystem of claim 10 or 11, implementing one or more neural networks configured to determine emotion. The one or more processors enhance consistency between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. 13. A subsystem according to any one of claims 10 to 12, implementing one or more neural networks configured to modify said video signal to. to determine the one or more types of perceived emotions of the first person based on the video signal, the one or more processors comprising:
detecting a facial expression of the first person based on the video signal;
determining a perceived emotion of the facial expression of the first person based on the facial expression of the first person ; or
Detecting a body pose of the first person based on the video signal and determining a perceived motion of the body pose of the first person based on the body pose of the first person. do, or
detecting a facial expression and a body posture of the first person based on the video signal, and detecting a face of the first person based on the facial expression and the body posture of the first person; determine the perceived emotion of facial expressions and the perceived emotion of body posture
14. A subsystem according to any one of claims 10 to 13, configured to: the one or more processors
performing audio signal processing of the audio signal to determine at least one of pitch, vibrato, or inflection of speech of the first person; determine the perceived emotion of the person's utterance of
also configured to modify the audio signal to increase consistency between the perceived emotion of the utterance of the first person and the semantic emotion of the first person. Subsystem according to 14. the one or more processors
configured to modify audio data of the audio signal corresponding to at least one of the pitch, vibrato, or inflection to modify the audio signal to produce a modified audio signal. 16. The subsystem of claim 15, wherein the one or more processors
configured to modify image data of said video signal corresponding to at least one of facial expression or body posture to modify said video signal to produce a modified video signal. 11. The subsystem according to claim 10. the one or more processors
performing natural language processing of the audio signal;
18. A subsystem according to any one of claims 10 to 17 , configured to determine said semantic emotion of said first person based on results of said natural language processing of said audio signal. the one or more processors
using a facial torus model to quantify the positivity and activity of facial expressions of the first person based on the video signal; or
configured to use a toric model of posture to quantify the positivity and activeness of body posture of the first person based on the video signal;
using a linguistic cyclic model to quantify linguistic positivity and activity of the first person based on the audio signal;
the distance between said positivity and said activity of said facial expression of said first person and said positivity and said activity of said language of said first person , or said of said first person for reducing at least one of the distance between the positivity and the activity of body posture and the positivity and activity of the language of the first person. 19. The subsystem of any one of claims 10-18 , configured to modify image data. the one or more processors
using an utterance cyclic model to quantify the positivity and activeness of the first person's utterance based on the audio signal;
audio data of said audio signal for reducing the distance between said positivity and activity of said utterance of said first person and said positivity and said activity of said language of said first person; 20. The subsystem of claim 19 , also configured to modify the . a transmitter configured to transmit a modified video signal and a modified audio signal to a subsystem associated with said second person participating in said video chat; 21. Subsystem according to claim 20 . A computer readable medium storing computer instructions which, when executed by one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 9 .

Images