JP7728580B2 - Video conference analysis system and video conference analysis program - Google Patents

Description

This invention relates to a video conference analysis system and a video conference analysis program that analyze the comments made by participants in video conferences held over a network.

With the spread of high-speed, high-capacity communication environments, it has become common to hold video conferences by connecting participants in remote locations over a network. Participants do not need to gather in one place, which frees them from travel time and the associated physical exhaustion, allowing them to participate in conferences efficiently and without waste. Proposals have also been made to improve the efficiency of video conferences themselves, such as the conference support system described in Patent Document 1. This conference support system identifies attendees through facial recognition and can display and record attendees' comments during the conference as text on a display in real time. It can also generate minutes in the form of summaries and record them in association with information identifying the conference in question. As such, video conferences are well suited to improving efficiency, and various proposals have been made.

JP 2019-61594 A

However, because video conferencing is conducted via a screen rather than face-to-face, there are concerns that it may be harder to control emotions. For example, there is a risk that participants may become emotional and be more likely to say or do things that offend other participants.

Due to these concerns, there was a demand for a system or program that could analyze the emotional state of participants during a video conference and the level of harassment contained in their comments.

The present invention was made in light of these circumstances, and its objective is to provide a video conference analysis system and a video conference analysis program that can analyze the comments made by video conference participants to understand the emotional state of the participants and the level of harassment contained in their comments.

In order to solve this problem, the invention described in claim 1 is a video conference analysis system for analyzing speech made by participants in a video conference held over a network, comprising: an image/audio receiving unit that receives audio data of the speech made by the participants over the network; a voice/speech conversion unit that converts the audio data received by the image/audio receiving unit into text; a text/sentence analysis unit that analyzes text indicating the content of the speech made by the participants, which is composed of text converted from the audio data by the voice/speech conversion unit, to obtain a speech analysis result; a screen configuration unit that configures a display screen that displays the text and the speech analysis result obtained by the text/sentence analysis unit; and a display screen configured by the screen configuration unit. The system comprises a screen providing unit that transmits a screen to the participants taking part in the video conference via the network, and a control unit that converts the audio data received by the image and audio receiving unit into text using the speech conversion unit to obtain the text, analyzes the text using the text analysis unit to obtain the speech analysis results, configures the display screen using the screen composition unit, and controls the transmission of the display screen from the screen providing unit, and is characterized in that the text analysis unit performs an analysis based on the text indicating the content of the participants' speech, and estimates an emotional level indicating an emotional state and a harassment level indicating the degree of harassment to obtain the speech analysis results.

The invention of claim 2, in addition to the configuration of claim 1, is characterized in that the speech-to-text conversion unit inputs the speech data received by the image and audio receiving unit into a trained speech-to-text conversion model generated by machine learning using a combination of training speech data and correct text data corresponding to the training speech data, and performs calculations to convert the speech data into text and obtain the text; and the text analysis unit inputs the acquired text into a trained text analysis model generated by machine learning using a combination of training text and correct emotion data corresponding to the training text, and a combination of the training text and correct harassment data corresponding to the training text, and performs calculations to estimate the emotion level and the harassment level and obtain the utterance analysis result.

The invention of claim 3, in addition to the configuration of claim 1 or 2, further comprises a translation unit that translates an original text written in a source language into a predetermined translation language, the control unit controls the operation of translating the text indicating the content of the participant's remarks using the translation unit to obtain a translation, the screen composition unit composes a translation display screen that displays the translation of the text translated by the translation unit, and the screen providing unit transmits the translation display screen composed by the screen composition unit to the participants participating in the video conference via the network.

The invention of claim 4 is characterized in that, in addition to the configuration of claim 3, the translation unit obtains a translation of a training original sentence by inputting the original sentence into a trained translation model generated by machine learning, using a combination of a training original sentence and a correct translation that is the correct translation of the training original sentence as training data, and performing calculations on the trained translation model.

The invention of claim 5, in addition to the configuration of any one of claims 1 to 4, further comprises an image analysis unit that performs analysis based on image data of the participants and estimates the emotional level to obtain image analysis results; the image and audio receiving unit receives the image data of the participants via the network; the control unit controls the operation of the image analysis unit to analyze the image data received by the image and audio receiving unit; the image analysis unit inputs the image data into a trained image analysis model generated by machine learning using combinations of training face images and correct emotion types corresponding to the training face images as training data, and performs calculations to obtain the image analysis results; the screen composition unit composes an image analysis display screen that displays the image analysis results obtained by the image analysis unit; and the screen providing unit transmits the image analysis display screen composed by the screen composition unit to the participants participating in the video conference via the network.

The invention of claim 6, in addition to the configuration of any one of claims 1 to 4, further comprises an image analysis unit that performs analysis based on image data of the participants, estimates the emotional level, and obtains image analysis results; the image and audio receiving unit receives the image data of the participants via the network; the control unit controls the image analysis unit to analyze the image data received by the image and audio receiving unit; the image analysis unit extracts a facial image from the image data and performs analysis using at least one of eye shape, changes in eye shape, eyebrow shape, changes in eyebrow shape, shape of the corners of the mouth (the sides of the lips), changes in the shape of the corners of the mouth, cheek shape, changes in the shape of the cheeks, frequency of teeth, and changes in the frequency of teeth to obtain the image analysis results; the screen composition unit creates an image analysis display screen that displays the image analysis results obtained by the image analysis unit; and the screen providing unit transmits the image analysis display screen created by the screen composition unit to the participants participating in the video conference via the network.

The invention of claim 7, in addition to the configuration of any one of claims 1 to 4, further comprises a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotional level and the harassment level, and obtains a voice analysis result; the control unit controls the operation of the voice analysis unit to analyze the voice data; the voice analysis unit inputs the received voice data into a trained voice analysis model generated by machine learning using combinations of training voice data and correct emotion data corresponding to the training voice data, and combinations of the training voice data and correct harassment data corresponding to the training voice data, as training data, and performs calculations to estimate the emotional level and the harassment level and obtain the voice analysis result; the screen composition unit composes a voice analysis display screen that displays the voice analysis result obtained by the voice analysis unit; and the screen provision unit transmits the voice analysis display screen composed by the screen composition unit to the participants participating in the video conference via the network.

The invention of claim 8, in addition to the configuration of any one of claims 1 to 4, further comprises a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotion level and the harassment level, and obtains a voice analysis result; the control unit controls the operation of the voice analysis unit to analyze the voice data; the voice analysis unit performs analysis using at least one of voice volume, changes in voice volume, voice pitch, changes in voice pitch, speaking rate, changes in speaking rate, frequency of speaking over the words of other participants, and changes in frequency of speaking over the words of the other participants, and obtains the voice analysis result; the screen composition unit composes a voice analysis display screen that displays the voice analysis result obtained by the voice analysis unit; and the screen provision unit transmits the voice analysis display screen composed by the screen composition unit to the participants participating in the video conference via the network.

The invention according to claim 9, in addition to the configuration of claim 1, further comprises an image analysis unit that performs analysis based on image data of the participants, estimates the emotional level, and obtains an image analysis result; a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotional level and the harassment level, and obtains a voice analysis result; and a judgment unit that comprehensively evaluates the utterance analysis result analyzed by the character and sentence analysis unit, the image analysis result obtained by the image analysis unit, and the voice analysis result obtained by the voice analysis unit, and estimates at least one of the overall emotional level and the overall harassment level, and obtains a comprehensive judgment result; and the image and voice receiving unit performs analysis based on image data of the participants, estimates the emotional level, and obtains an overall judgment result. The control unit receives the projected image data via the network, and controls the image analysis unit to analyze the image data and the audio analysis unit to analyze the audio data, and the judgment unit to comprehensively evaluate the utterance analysis results analyzed by the text analysis unit, the image analysis results obtained by the image analysis unit, and the audio analysis results obtained by the audio analysis unit to determine the comprehensive judgment result. The screen composition unit creates a comprehensive judgment display screen that displays the comprehensive judgment result obtained by the judgment unit, and the screen provision unit transmits the comprehensive judgment display screen created by the screen composition unit to the participants participating in the video conference via the network.

The invention of claim 10, in addition to the configuration of any one of claims 1 to 9, further comprises an inappropriate phrase detection unit that detects whether the text contains pre-registered inappropriate phrases, and when the inappropriate phrase detection unit detects the inappropriate phrase, the control unit takes inappropriate phrase blocking measures that include at least one of stopping the transmission of the portion of the audio data corresponding to the inappropriate phrase to be transmitted via the network to the participants in the video conference, deleting the inappropriate phrase contained in the text, and replacing the inappropriate phrase contained in the text with an appropriate phrase corresponding to the inappropriate phrase.

The invention of claim 11 is characterized in that, in addition to the configuration of claim 10, when the detection frequency of the inappropriate phrases detected by the inappropriate phrase detection unit exceeds a predetermined threshold, the control unit controls the screen composition unit to compose a warning display screen and the screen provision unit to transmit the warning display screen composed by the screen composition unit to the participants taking part in the video conference via the network.

The invention of claim 12 is a video conference analysis program that analyzes speech made by participants in a video conference held over a network, comprising: an image and audio reception process that receives audio data of the participants' speech over the network; a voice-to-text conversion process that converts the audio data received in the image and audio reception process into text; and a text analysis process that analyzes text indicating the content of the participants' speech, which is composed of text converted from the audio data in the voice-to-text conversion process, to obtain a speech analysis result. The text analysis process performs an analysis based on the text indicating the content of the participants' speech, and estimates an emotional level that indicates their emotional state and a harassment level that indicates the degree of harassment to obtain the speech analysis result.

The invention of claim 13, in addition to the configuration of claim 12, further comprises a translation process for translating an original text written in a source language into a predetermined translation language, and the translation process is characterized in that it translates the text indicating the content of the participant's remarks to generate a translated text.

The invention of claim 14, in addition to the configuration of claim 12, further comprises an image analysis process that performs analysis based on image data of the participant, estimates the emotional level, and obtains an image analysis result; the image and audio reception process receives the image data of the participant via the network; and the image analysis process extracts a facial image from the image data and performs analysis using at least one of the following: eye shape, changes in eye shape, eyebrow shape, changes in eyebrow shape, shape of the corners of the mouth (the parts on both sides of the lips), changes in the shape of the corners of the mouth, cheek shape, changes in cheek shape, frequency of tooth appearance, and changes in the frequency of tooth appearance, thereby obtaining the image analysis result.

The invention of claim 15, in addition to the configuration of claim 12, includes a voice analysis process that performs analysis based on the voice data received in the image and voice reception process, estimates the emotional level and the harassment level, and obtains a voice analysis result, and is characterized in that the voice analysis process performs analysis using at least one of the following: voice volume, changes in voice volume, voice pitch, changes in voice pitch, speaking rate, changes in speaking rate, frequency of speaking over the words of other participants, and changes in frequency of speaking over the words of the other participants, to obtain the voice analysis result.

The invention of claim 16, in addition to the configuration of claim 12, comprises an image analysis process that performs an analysis based on image data of the participants and estimates the emotional level to obtain an image analysis result; a voice analysis process that performs an analysis based on the voice data received in the image and voice reception process and estimates the emotional level and the harassment level to obtain a voice analysis result; and a judgment process that comprehensively evaluates the utterance analysis result analyzed in the character and sentence analysis process, the image analysis result obtained in the image analysis process, and the voice analysis result obtained in the voice analysis process to estimate at least one of an overall emotional level and an overall harassment level to obtain an overall judgment result, wherein the image and voice reception process receives the image data of the participants via the network, the image analysis process analyzes the image data to obtain the image analysis result, the voice analysis process analyzes the voice data to obtain the voice analysis result, and the judgment process comprehensively evaluates the utterance analysis result analyzed in the character and sentence analysis process, the image analysis result obtained in the image analysis process, and the voice analysis result obtained in the voice analysis process to obtain the overall judgment result.

According to the invention of claim 1, audio data of speeches made by video conference participants is converted into text, and a text sentence indicating the content of the speech is constructed. Analysis is then performed based on the constructed text sentence to estimate an emotional level indicating the emotional state and a harassment level indicating the degree of harassment, and a speech analysis result is obtained. A display screen is then constructed that displays the obtained text sentence indicating the content of the participants' speeches and the speech analysis result, and this display screen is transmitted to the video conference participants.

In this way, the emotional and harassment levels of video conference participants can be analyzed based on their comments, and the results of this comment analysis can be shared among the video conference participants. This means that if a participant becomes emotional, they can understand their own state and control themselves, and other participants can also understand their state and encourage them to calm down.

In addition, text indicating the content of each participant's remarks is sent to video conference participants and can be viewed on the screen, so even if the connection is poor and it is difficult to hear the audio, the conference can continue without interruption. Participants with hearing impairments can also participate in the conference.

Furthermore, according to the invention of claim 2, the speech-to-text conversion unit converts speech data into text using a trained speech-to-text conversion model generated by machine learning, which is artificial intelligence (AI). This allows for highly accurate and stable conversion of speech data into text. Furthermore, the text analysis unit determines emotion levels and harassment levels using a trained text analysis model generated by machine learning, which is artificial intelligence. This allows for accurate and stable estimation of emotion levels and harassment levels.

Furthermore, according to the invention of claim 3, text indicating the content of statements made by video conference participants is translated into a specified translation language, allowing participants who speak different languages to refer to the translation text and achieve smooth communication.

Furthermore, according to the invention of claim 4, the translation unit translates the source text using a trained translation model generated by machine learning. This allows for highly accurate and reliable translation.

Furthermore, according to the invention of claim 5, the image analysis unit estimates the emotional level from the image data using a trained image analysis model generated by machine learning. This makes it possible to stably determine the emotional level from the image data with high accuracy.

Furthermore, according to the invention of claim 6, emotional levels are estimated based on image data captured of video conference participants, and image analysis results are obtained. An image analysis display screen displaying these image analysis results is then constructed and transmitted to the video conference participants. In this way, emotional levels are estimated based on the image data of the participants, and the image analysis results are shared with the conference participants. Because the image analysis results are obtained from the image data of the participants, emotional levels can be analyzed using data other than text that indicates the content of the participants' comments, and multifaceted analysis results can be obtained.

Furthermore, according to the invention of claim 7, the voice analysis unit estimates the emotion level and harassment level from the voice data using a trained voice analysis model generated by machine learning. This makes it possible to accurately and reliably detect the emotion level and harassment level from the voice data.

Furthermore, according to the invention of claim 8, emotional and harassment levels are estimated based on the audio data of the speech of the video conference participants, and audio analysis results are obtained. A audio analysis display screen displaying these audio analysis results is then constructed and transmitted to the video conference participants. In this way, emotional and harassment levels are estimated based on the audio data of the participants' speech, and the audio analysis results are shared with the conference participants. Because the audio analysis results are obtained from the participants' audio data, the emotional and harassment levels can be analyzed using the participants' audio data itself, rather than text that indicates the content of the participants' speech, and analysis results can be obtained from multiple perspectives.

Furthermore, according to the invention of claim 9, a comprehensive assessment result is obtained by comprehensively evaluating the utterance analysis result obtained based on the text indicating the content of the participants' utterances, the image analysis result obtained based on the image data of the participants, and the audio analysis result obtained based on the audio data of the participants' utterances. Then, a comprehensive assessment display screen is constructed to display this comprehensive assessment result, and is transmitted to the participants in the video conference. In this way, the utterance analysis result, image analysis result, and audio analysis result are combined, thereby obtaining a more multifaceted analysis result.

Furthermore, according to the invention of claim 10, the text indicating the content of participants' remarks is detected to determine whether it contains inappropriate words, and if inappropriate words are detected, an inappropriate word blocking measure is taken. This inappropriate word blocking measure prevents inappropriate words that may offend others from being communicated to conference participants, allowing participants to participate in the conference with peace of mind.

Furthermore, according to the invention of claim 11, if the detection frequency of inappropriate words and phrases exceeds a predetermined threshold, a warning display screen is sent to the participants. This warning display screen allows meeting participants to objectively recognize that inappropriate remarks are increasing, and they can take measures such as taking a break.

According to the invention of claim 12, audio data of speeches made by video conference participants is converted into text, and a text sentence indicating the content of the speech is constructed. Furthermore, an analysis is performed based on the constructed text sentence, and an emotional level indicating the emotional state and a harassment level indicating the degree of harassment are estimated, and a speech analysis result is obtained. In this way, emotional levels and harassment levels can be analyzed based on the speeches made by video conference participants.

The video conferencing analysis program can be provided as an API (Application Programming Interface), which means that the API can be used in a variety of video conferencing systems, making it highly versatile.

Furthermore, according to the invention of claim 13, text indicating the content of statements made by video conference participants is translated into a specified translation language, allowing participants who speak different languages to refer to the translation text and achieve smooth communication.

Furthermore, according to the invention of claim 14, emotional levels are estimated based on image data captured of video conference participants, and image analysis results are obtained. This makes it possible to analyze emotional levels using data other than text that indicates the content of participants' comments, thereby obtaining multifaceted analysis results.

Furthermore, according to the invention of claim 15, the emotional and harassment levels of video conference participants are estimated based on the audio data of their speech, and audio analysis results are obtained. Therefore, unlike text that indicates the content of participants' speech, the emotional and harassment levels can be analyzed using the participants' audio data itself, and multifaceted analysis results can be obtained.

Furthermore, according to the invention of claim 16, a comprehensive assessment is made by comprehensively evaluating the utterance analysis results obtained based on text indicating the content of the participants' utterances, the image analysis results obtained based on the participants' image data, and the audio analysis results obtained based on the audio data of the participants' utterances. In this way, the utterance analysis results, image analysis results, and audio analysis results are combined, making it possible to obtain more multifaceted analysis results.

1 is a configuration block diagram showing in outline a video conference system including a video conference analysis system according to a first embodiment of the present invention; 2 is a schematic block diagram showing the configuration of a WEB/APP server according to the first embodiment. FIG. 2 is a schematic block diagram showing the configuration of a voice-to-text conversion server according to the first embodiment. FIG. 2 is a schematic block diagram showing the configuration of a character and sentence analysis server according to the first embodiment. FIG. FIG. 2 is a schematic block diagram showing the configuration of a translation server according to the first embodiment. 2 is a schematic block diagram showing the configuration of an image and audio analysis server according to the first embodiment. FIG. 2 is a schematic block diagram showing the configuration of a client terminal of a videoconference system including the videoconference analysis system according to the first embodiment. FIG. 10 is a diagram showing an example of a video conference screen displayed on a client terminal according to the first embodiment; FIG. FIG. 10 is a diagram showing another example of the video conference screen displayed on the client terminal according to the first embodiment. 1A is a diagram showing an example of a pictogram for displaying an emotion level according to the first embodiment, and FIG. 1B is a diagram showing an example of a pictogram for displaying a harassment level. FIG. 10 is a diagram showing an example of a screen displaying an emotion level according to the first embodiment. 10 is a diagram showing a schematic flowchart at the start of a video conference in the video conference analysis system according to the first embodiment. FIG. 10 is a diagram showing a schematic flowchart when a video conference participant makes a statement in the video conference analysis system according to the first embodiment. FIG. FIG. 10 is a block diagram illustrating a configuration of a video conference system including a video conference analysis system according to a second embodiment of the present invention. FIG. 10 is a schematic block diagram showing the configuration of a WEB/APP server according to the second embodiment. FIG. 10 is a diagram illustrating an example of the specifications of a video conference analysis API according to the second embodiment. FIG. 10 is a schematic block diagram showing the configuration of a videoconference server of a videoconference system including a videoconference analysis system according to the second embodiment.

[First embodiment of the invention]
A first embodiment of the present invention will be described with reference to FIGS. 1 to 13. FIG.

FIG. 1 is a block diagram illustrating a configuration of a videoconference system 100 including a videoconference analysis system 1 according to a first embodiment of the present invention. This videoconference analysis system 1 includes a WEB/APP (Web/Application) server 2, a speech-to-text conversion server 3, a text analysis server 4, a translation server 5, and an image/speech analysis server 6. Each server included in this analysis system 1 is connected to the Internet 8, which serves as a "network." Client terminals 7 ₁ , 7 ₂ , 7 ₃ , . . . , 7 _n (hereinafter referred to as "client terminals 7") operated by videoconference participants are also connected to the Internet 8, allowing participants to participate in the videoconference via the Internet 8. Thus, this videoconference system 100 is configured to include the videoconference analysis system 1 and the client terminals 7.

This videoconferencing system 100 not only allows participants who are far apart to hold videoconferences, but also analyzes the participants' comments using the videoconferencing analysis system 1, estimating their emotional state and the degree of harassment contained in the comments, and displaying this information on each client terminal 7.

The following describes each server and client terminal 7 that make up this videoconference analysis system 1.

<WEB/APP server>
The WEB/APP server 2 accepts image and audio data of the participants transmitted from each client terminal 7 of the conference participants, composes a video conference screen and conference audio based on the image and audio data, and transmits the video conference screen and audio to each client terminal 7. By performing these operations, the conference participants can view the conference screen and audio via each client terminal 7, allowing the video conference to proceed. In this way, the WEB/APP server 2 has the function of realizing a video conference. In addition, the WEB/APP server 2 processes and analyzes the image and audio data of the participants transmitted from each client terminal 7 using the speech-to-text conversion server 3, the text analysis server 4, the translation server 5, and the image and audio analysis server 6 (described below). The WEB/APP server 2 composes a display screen to be displayed on the client terminal 7 based on the analysis results obtained in this manner, and transmits the display screen to each client terminal 7. Participants can check their own analysis results and the analysis results of other participants through the screen displayed on the client terminal 7.

As shown in the schematic block diagram of Figure 2, the WEB/APP server 2 is configured to include a WEB/APP server control unit 20, a screen configuration unit 21, a conference setting unit 22, a minutes recording unit 23, a determination unit 24, an image and audio receiving unit 25, a screen providing unit 26, a communication unit 28, and a memory unit 29.

The WEB/APP server control unit 20, which serves as a "control unit," is configured to include a CPU (not shown) that executes programs, performs arithmetic processing, and controls the various elements that make up the WEB/APP server 2. The WEB/APP server control unit 20 executes programs stored in an auxiliary storage device (not shown), a non-volatile storage device that makes up the storage unit 29, and operates the various elements that make up the WEB/APP server 2. The auxiliary storage device may be, for example, an HDD (hard disk drive) or an SSD (solid state drive). When a program is executed, the RAM (not shown), a volatile memory that makes up the storage unit 29, is used as a work area for program execution and arithmetic processing by the CPU.

Specifically, the auxiliary storage device of the storage unit 29 stores programs such as a program that runs the video conference and a program that causes each server to analyze the image and audio data of participants sent from each client terminal 7, creates a screen displaying the analysis results, and sends these to each client terminal 7. These programs are executed by the CPU of the WEB/APP server control unit 20 using the RAM of the storage unit 29.

The communication unit 28 is connected to the Internet 8 and transmits and receives data between each server and each client terminal 7 that make up this videoconference analysis system 1.

Under the control of the WEB/APP server control unit 20, the image and audio receiving unit 25 receives captured images of participants and audio data of participants' speech transmitted from each client terminal 7 of the conference participants via the Internet 8 (image and audio reception process). This image and audio receiving unit 25 may be implemented by the communication unit 28. The image and audio receiving unit 25 or the communication unit 28 may also receive character data constituting sentences input from a character input unit 74 such as a keyboard on each client terminal 7 of the participants, as well as chart data transmitted from the client terminal 7.

The WEB/APP server control unit 20 transmits the audio data of the participants' comments received by the image and audio receiving unit 25 from the communication unit 28 to the speech-to-text conversion server 3, converts the audio data into text, and obtains text indicating the content of the comments. Next, the WEB/APP server control unit 20 transmits the text data of the obtained text from the communication unit 28 to the text analysis server 4, analyzes the text, and obtains the results of the comment analysis. The WEB/APP server control unit 20 also transmits the image and audio data of the participants received by the image and audio receiving unit 25 from the communication unit 28 to the image and audio analysis server 6, analyzes the image data, and obtains the results of the image analysis, and analyzes the audio data, and obtains the results of the audio analysis.

The screen configuration unit 21 configures video conference screens 200, 210 (described below) and the like, which are displayed on each client terminal 7 under the control of the WEB/APP server control unit 20. The display screen displays images of participants, text indicating the content of their speech obtained by converting the audio data of their speech into text, speech analysis results obtained by analyzing the text, translations of the text into a specified translation language (translation display screen), image analysis results obtained by analyzing image data of participants (image analysis display screen), audio analysis results obtained by analyzing the audio data of participants (audio analysis display screen), overall judgment results obtained by comprehensively evaluating the speech analysis results, image analysis results, and audio analysis results (overall judgment display screen), and text and chart data entered from the participants' client terminals 7.

The screen providing unit 26 transmits and provides the screen created by the screen composition unit 21 to each of the participants' client terminals 7 under the control of the WEB/APP server control unit 20. This screen providing unit 26 may be realized by the communication unit 28.

Prior to the conference, the conference setting unit 22 sets up the video conference, such as setting the language to be used in the conference, whether or not translation will be used, and the content of the conference analysis, as well as setting the conference participants and the languages used by those participants. When translation is set, the language of the conference participant making a statement is translated into the language used by the other participants whose comments are received at the client terminal 7. The video conference proceeds based on the settings of this conference setting unit 22.

The minutes recording unit 23 records the audio data of participants' remarks received by the image and audio receiving unit 25, and records text data converted from participants' audio data. Furthermore, it may record the results of analysis based on the participants' images and audio data, or may record text entered from the keyboard of each client terminal 7. It may also record chart data, image data, etc. sent from each client terminal 7.

Based on the control of the WEB/APP server control unit 20, the judgment unit 24 comprehensively evaluates the utterance analysis results analyzed by the character and sentence analysis server 4 and the image and audio analysis results obtained by the image and audio analysis server 6, estimates the overall emotional level and overall harassment level, and determines an overall judgment result (judgment process). As a method for comprehensively evaluating the utterance analysis results, image analysis results, and audio analysis results, for example, weighting or other processing may be performed on each result to make a comprehensive judgment.

Furthermore, if inappropriate words are detected in a text sentence obtained by converting the audio data of a participant's speech received by the image and audio receiving unit 25 into text by the speech-to-text server 3, the WEB/APP server control unit 20 may take inappropriate word blocking measures, such as stopping the transmission of the portion of the audio data corresponding to the inappropriate words from the audio data to be sent to each participant's client terminal 7, or deleting the inappropriate words from the display screen of the text sentence to be sent to each participant's client terminal 7. Furthermore, as an inappropriate word blocking measure, appropriate words corresponding to inappropriate words may be stored in advance in the WEB/APP server 2, speech-to-text server 3, etc., and inappropriate words contained in the text sentence may be replaced with appropriate words corresponding to the inappropriate words.

Furthermore, if the frequency at which inappropriate words are detected in text exceeds a number preset in the conference setting unit 22, the WEB/APP server control unit 20 may cause the screen configuration unit 21 to configure a warning display screen indicating that the frequency at which inappropriate words are detected has exceeded a predetermined threshold, and may transmit the configured warning display screen from the screen providing unit 26 to each of the participants' client terminals 7. The warning display screen may, for example, be a warning that inappropriate remarks are becoming more frequent or a display encouraging participants to take a break. Furthermore, if a particular participant repeatedly makes remarks containing inappropriate words, a display may be displayed advising that the participant should leave the video conference.

<Speech-to-text server>
The speech-to-text server 3 receives speech data of conference participants and converts the speech data into text to create text that shows the content of the speech. In addition, the speech-to-text server 3 stores inappropriate words and phrases in advance, and detects whether or not the text obtained from the speech data contains inappropriate words and phrases.

As shown in the schematic block diagram of Figure 3, the speech-to-text server 3 is configured to include a speech-to-text server control unit 30, a speech-to-text conversion unit 31, a speech-to-text dictionary database 32, a trained speech-to-text model storage unit 33, an inappropriate phrase storage unit 34, an inappropriate phrase detection unit 35, a communication unit 38, and a memory unit 39.

The speech-to-text server control unit 30 is configured to include a CPU (not shown) that executes programs, performs calculations, and controls the various elements that make up the speech-to-text server 3. The speech-to-text server control unit 30 executes programs stored in an auxiliary storage device (not shown) that makes up the storage unit 39, causing the various elements that make up the speech-to-text server 3 to operate. When a program is executed, the RAM (not shown), a volatile memory that makes up the storage unit 39, is used as a work area for the CPU to execute the program and perform calculations.

The communication unit 38 is connected to the Internet 8 and transmits and receives data to and from the WEB/APP server 2 and the like. Under the control of the voice-to-text conversion server control unit 30, this communication unit 38 accepts the voice data of participants' speech and the language of that voice data transmitted from the WEB/APP server 2.

The speech-to-text dictionary database 32 stores characters corresponding to voice data as a database, and this database 32 is used as a dictionary when extracting characters from voice data. The speech-to-text dictionary database 32 also has a multilingual dictionary database, and is used as a dictionary for converting voice data into text by specifying a language.

The trained speech-to-text model storage unit 33 stores trained speech-to-text models generated through machine learning using a large number of combinations of training speech data and corresponding ground truth text data as training data. To support multiple languages, a generation model for each language is stored. A language is specified, and speech data is input into this trained speech-to-text model for calculation. This calculation outputs characters in the specified language that are estimated to correspond to the speech data. Because machine learning, an artificial intelligence, is used, speech data is converted into text with high accuracy and stability.

Based on the control of the speech-to-text server control unit 30, the speech-to-text conversion unit 31 converts the speech data received by the communication unit 38 into characters in that language (speech-to-text conversion process). Specifically, the speech-to-text conversion unit 31 converts the speech data into characters using the speech-to-text dictionary database 32. The speech-to-text conversion unit 31 also converts the speech data into characters using a trained speech-to-text model stored in the trained speech-to-text model storage unit 33 for the speech data and its language. The speech-to-text conversion unit 31 then compares and adjusts the characters obtained using the speech-to-text dictionary database 32 with the characters obtained using the trained speech-to-text model, correcting them to obtain the optimal characters. Note that the speech-to-text conversion unit 31 does not need to use both the speech-to-text dictionary database 32 and the trained speech-to-text model; it may use either one to convert speech data into characters.

The voice-to-text conversion server control unit 30 transmits the text data of the text sentence obtained by converting the voice data into text from the communication unit 38 to the WEB/APP server 2.

The inappropriate phrase storage unit 34 pre-registers and stores inappropriate phrases that may cause discomfort or harassment to others.

The inappropriate phrase detection unit 35, under the control of the speech-to-text server control unit 30, detects whether the text obtained by the speech-to-text conversion unit 31 contains any inappropriate phrases stored in the inappropriate phrase storage unit 34.

If an inappropriate word or phrase is detected in the text, the voice-to-text server control unit 30 sends a notification indicating that the text composed of the received voice data contains an inappropriate word or phrase, along with the detected inappropriate word or phrase, to the WEB/APP server 2 via the communication unit 38. The voice-to-text server 3 may also store appropriate words or phrases corresponding to inappropriate words, and transmit the detected inappropriate word or phrase and the appropriate words or phrases corresponding to the inappropriate word or phrase to the WEB/APP server 2. This allows the WEB/APP server 2 to replace the inappropriate word or phrase with an appropriate word or phrase.

The WEB/APP server 2, which receives a notification indicating that inappropriate words are included and the inappropriate words, can take measures to block inappropriate words as described above.

<Text analysis server>
The character sentence analysis server 4 analyzes the character sentences that are constructed by converting the audio data of the speeches of the conference participants into text, and estimates the emotional level that indicates the emotional state and the harassment level that indicates the degree of harassment to obtain the speech analysis results (character sentence analysis processing).

Emotion levels may be a combination of emotion types such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," along with a percentage display of the intensity of the emotion. Alternatively, the estimated probability of emotions such as "joy" and "sadness" may be displayed as a percentage. Alternatively, the emotional state may be displayed as a percentage ranging from good to bad.

Harassment levels may be a combination of the type of harassment, such as "power harassment" or "sexual harassment," and a percentage display of the intensity of that harassment. Alternatively, the estimated probability of harassment, such as "power harassment" or "sexual harassment," may be displayed as a percentage. Alternatively, the percentage display may be a range between the presence and absence of harassment.

As shown in the schematic block diagram of Figure 4, the text analysis server 4 is configured to include a text analysis server control unit 40, a text analysis unit 41, a text analysis dictionary database 42, a trained text analysis model storage unit 43, a communication unit 48, and a memory unit 49.

The character and text analysis server control unit 40 is configured to include a CPU (not shown) that controls the various elements that make up the character and text analysis server 4. The memory unit 49 is configured from an auxiliary storage device (not shown) and RAM (not shown).

The communication unit 48 is connected to the Internet 8 and transmits and receives data to and from the WEB/APP server 2 and the like. Under the control of the character and sentence analysis server control unit 40, this communication unit 48 accepts character data and the language of text sentences converted from participants' voice data transmitted from the WEB/APP server 2.

The character and sentence analysis dictionary database 42 stores emotional and harassment levels corresponding to the words and phrases that make up sentences, and this database 42 is used as a dictionary for estimating emotional and harassment levels from text. The character and sentence analysis dictionary database 42 also has a multilingual dictionary database, and is used as a dictionary for estimating emotional levels, etc. from text by specifying a language.

The learned character sentence analysis model storage unit 43 stores a learned character sentence analysis model generated through machine learning using numerous combinations of training sentences and their corresponding correct emotion data, and training sentences and their corresponding correct harassment data, as training data. This learned character sentence analysis model is composed of a learned character sentence analysis emotion generation model and a learned character sentence analysis harassment generation model. The learned character sentence analysis emotion generation model is generated using combinations of training sentences and their corresponding correct emotion types, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," as training data. The emotion level is determined by inputting the character sentence and performing a calculation. At least one of the emotions, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," can be used as the emotion type. Meanwhile, the trained text analysis harassment generation model is generated using training data consisting of combinations of training text and corresponding correct answers for types of harassment, such as "power harassment," "sexual harassment," and "no harassment." The harassment level is determined by inputting the text and performing a calculation. At least one of "power harassment," "sexual harassment," and "no harassment" can be used as the type of harassment. To support multiple languages, a generation model for each language is maintained. By specifying a language and inputting text into this trained text analysis model and performing a calculation, the estimated emotion level and harassment level corresponding to the text are output.

Under the control of the character/text analysis server control unit 40, the character/text analysis unit 41 uses the character/text analysis dictionary database 42 to estimate the emotional level and harassment level from a character/text composed of character data received by the communication unit 48 and its language. Using the character/text analysis dictionary database 42, the emotional level and harassment level for each word/phrase that makes up the character/text is determined. Then, the emotional level and harassment level for each word/phrase are weighted and other processing is performed to estimate the emotional level and harassment level for the character/text as a whole. Furthermore, the character/text analysis unit 41 uses a trained character/text analysis model stored in the trained character/text analysis model storage unit 43 to estimate the emotional level and harassment level from the character/text for the character/text and its language. Then, the emotional level and harassment level estimated using the character/text analysis dictionary database 42 are compared with the emotional level and harassment level estimated using the trained character/text analysis model, and adjustments are made to correct them to the optimal emotional level and harassment level.

The character and sentence analysis unit 41 does not need to use both the character and sentence analysis dictionary database 42 and the trained character and sentence analysis model, and may use either one to estimate the emotion level or harassment level from the text. The character and sentence analysis unit 41 may also estimate either the emotion level or the harassment level.

In this way, the emotional level and harassment level estimated by analyzing the text that indicates the content of the participants' comments become the comment analysis results. The text analysis server control unit 40 transmits this comment analysis result from the communication unit 48 to the WEB/APP server 2.

The character and sentence analysis unit 41 may also analyze sentences entered via the keyboard of the client terminal 7, etc., to estimate the emotional level and harassment level.

<Translation server>
The translation server 5 converts the speech data of the conference participants into text, and translates the resulting text into a target language to generate a translation of the text, which is the original text written in the source language (translation process).

As shown in the schematic block diagram of Figure 5, the translation server 5 is configured to include a translation server control unit 50, a translation unit 51, a translation dictionary database 52, a trained translation model storage unit 53, a communication unit 58, and a memory unit 59.

The translation server control unit 50 is configured to include a CPU (not shown) that controls the various elements that make up the translation server 5. The memory unit 59 is configured from an auxiliary storage device (not shown) and RAM (not shown).

The communication unit 58 is connected to the Internet 8 and transmits and receives data to and from the WEB/APP server 2 and the like. Under the control of the translation server control unit 50, this communication unit 58 receives text data of text sentences converted from participant voice data transmitted from the WEB/APP server 2, the source language in which the text sentences are written, and the translation language into which the text is to be translated.

Translation dictionary database 52 stores translation words corresponding to words in the source language as a database, and is used as a dictionary when generating translations of text. Translation dictionary database 52 also has a multilingual dictionary database, and is used as a dictionary when generating translations of text in the source language into a target language by specifying the source language and target language.

The trained translation model storage unit 53 stores trained translation models generated by machine learning using as training data a large number of combinations of training source sentences and their correct translations. To support multiple languages, a generation model for each language is stored, and by specifying the source language and translation language, a text sentence in the source language is input into this trained translation model and calculations are performed to output a translation of the text sentence.

Under the control of the translation server control unit 50, the translation unit 51 uses the translation dictionary database 52 to generate a translation from a text sentence composed of character data received by the communication unit 58, and the source language and translation language of the text sentence. The translation unit 51 also generates a translation of the text sentence using a trained translation model stored in the trained translation model storage unit 53 for the text sentence, the language, and the translation language. The translation generated using the translation dictionary database 52 is then adjusted with the translation generated using the trained translation model to obtain an optimal translation. Note that the translation unit 51 does not need to use both the translation dictionary database 52 and the trained translation model; it may use either one to generate a translation of the text sentence.

The translation server control unit 50 transmits the text data of the translation of the text sentence generated in this way from the communication unit 58 to the WEB/APP server 2.

The translation unit 51 may also translate sentences entered via a keyboard or other device on the client terminal 7 to generate translated text.

<Image and audio analysis server>
The image and audio analysis server 6 analyzes image data of the conference participants to estimate their emotional levels and obtain image analysis results (image analysis processing), and also analyzes audio data of the conference participants' speech to estimate their emotional levels and harassment levels and obtain audio analysis results (audio analysis processing).

As shown in the schematic block diagram of Figure 6, the image and audio analysis server 6 is configured to include an image and audio analysis server control unit 60, an image analysis unit 61, an image analysis dictionary database 62, a trained image analysis model holding unit 63, a sound analysis unit 64, a sound analysis dictionary database 65, a trained sound analysis model holding unit 66, a communication unit 68, and a memory unit 69.

The image and audio analysis server control unit 60 is configured to include a CPU (not shown) that controls each element that makes up the image and audio analysis server 6. The memory unit 69 is configured from an auxiliary storage device (not shown) and RAM (not shown).

The communication unit 68 is connected to the Internet 8 and transmits and receives data to and from the WEB/APP server 2 and the like. Based on the control of the image and audio analysis server control unit 60, this communication unit 68 receives image data of participants and audio data of participants' speech sent from the WEB/APP server 2.

The image analysis dictionary database 62 stores a database of emotional levels corresponding to the shapes of facial image parts such as the eyes, eyebrows, corners of the mouth (the parts on either side of the lips), cheeks, and mouth, as well as changes in these shapes. This database 62 is used as a dictionary when estimating emotional levels based on facial images extracted from the participant's image data.

The trained image analysis model storage unit 63 stores a trained image analysis model generated by machine learning using as training data a large number of combinations of training face images and corresponding correct emotion types, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise." At least one of the emotions, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," can be used as the emotion type. By inputting a face image extracted from the participant's image data into this trained image analysis model and performing a calculation, an estimated emotion level corresponding to the face image is output. For example, an estimated probability of each emotion, such as "joy," "sadness," or "anger," for the input face image is output.

The image analysis unit 61 extracts facial images from the image data of participants received by the communication unit 68, and analyzes the facial images using the eye shape, changes in eye shape during the video conference, eyebrow shape, changes in eyebrow shape during the conference, shape of the corners of the mouth, changes in shape of the corners of the mouth, shape of the cheeks, changes in cheek shape, frequency of teeth, changes in frequency of teeth, etc., to estimate the emotional level and obtain the image analysis results.

That is, under the control of the image and audio analysis server control unit 60, the image analysis unit 61 uses the image analysis dictionary database 62 to estimate an emotional level from a facial image extracted from image data. Using the image analysis dictionary database 62, the emotional level corresponding to the shape of each part of the facial image, such as the eyes, eyebrows, corners of the mouth, cheeks, and mouth, and changes in that shape, is determined, and the emotional level of each part is weighted and other processing is performed to estimate the emotional level of the entire facial image. Furthermore, the image analysis unit 61 estimates an emotional level from the extracted facial image using a trained image analysis model stored in the trained image analysis model storage unit 63. The emotional level estimated using the image analysis dictionary database 62 is then adjusted to the emotional level estimated using the trained image analysis model, thereby correcting it to an optimal emotional level. It is not necessary for the image analysis unit 61 to use both the image analysis dictionary database 62 and the trained image analysis model; it may use either one to estimate an emotional level from a facial image.

The emotion level estimated by analyzing the image data in this way becomes the image analysis result. The image and audio analysis server control unit 60 transmits this image analysis result from the communication unit 68 to the WEB/APP server 2.

The image analysis unit 61 may also be equipped with a configuration (not shown) that estimates the gender and age of participants based on facial images extracted from their image data.

The voice analysis dictionary database 65 stores a database of information on the volume of a voice, the pitch of a voice, speaking speed, the frequency of speaking over other speakers, and the emotional and harassment levels corresponding to changes in the volume, pitch, speaking speed, etc. This database 65 is used as a dictionary when estimating the emotional and harassment levels from the participants' voice data.

The trained voice analysis model storage unit 66 stores trained voice analysis models generated through machine learning using numerous combinations of training voice data and corresponding correct emotion data, and training voice data and corresponding correct harassment data, as training data. These trained voice analysis models are comprised of a trained voice analysis emotion generation model and a trained voice analysis harassment generation model. The trained voice analysis emotion generation model is generated using training voice data and corresponding correct emotion types, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," as training data. The trained voice analysis emotion generation model is input and calculated to determine emotion levels. For example, the estimated probability of each emotion, such as "joy" or "sadness," for the input voice data is output. Meanwhile, the trained voice analysis harassment generation model is generated using training voice data and corresponding correct harassment types, such as "power harassment," "sexual harassment," and "no harassment," as training data. The trained voice analysis harassment generation model is input and calculated to determine harassment levels. For example, the estimated probability of each type of harassment, such as "power harassment" or "sexual harassment," for the input voice data is output. By inputting the participant's voice data into this trained voice analysis model and performing calculations, the estimated emotional level and harassment level corresponding to the voice data is output.

The voice analysis unit 64 analyzes the participant's voice data received by the communication unit 68 using factors such as voice volume, changes in voice volume during the video conference, voice pitch, changes in voice pitch during the conference, speaking speed, changes in speaking speed, frequency of speaking over other participants' words, and changes in frequency of speaking over other participants' words, and estimates emotional level and harassment level to obtain voice analysis results.

That is, under the control of the image and audio analysis server control unit 60, the audio analysis unit 64 estimates the emotional level and harassment level of the audio data using the audio analysis dictionary database 65. Using the audio analysis dictionary database 65, the emotional level and harassment level corresponding to the volume of the voice, pitch, speaking rate, etc. and changes therein are determined, and the emotional level and harassment level for each of these judgment elements are weighted and other processing is performed to estimate the emotional level and harassment level for the audio data as a whole. The audio analysis unit 64 also estimates the emotional level and harassment level of the audio data using a trained audio analysis model stored in the trained audio analysis model storage unit 66. The emotional level and harassment level estimated using the audio analysis dictionary database 65 are then adjusted with the emotional level and harassment level estimated using the trained audio analysis model to correct them to the optimal emotional level and harassment level.

The voice analysis unit 64 does not need to use both the voice analysis dictionary database 65 and the trained voice analysis model; it may use either one to estimate the emotional level, etc. from the participant's voice data. The voice analysis unit 64 may also estimate either the emotional level or the harassment level.

In this way, the emotion level and harassment level estimated by analyzing the voice data become the voice analysis results. The image and voice analysis server control unit 60 transmits this voice analysis result from the communication unit 68 to the WEB/APP server 2.

<Client terminal>
The client terminal 7 is connected to the Internet 8 by software called a browser, and is connected to a server that operates the video conference, such as the WEB/APP server 2. The video conference is carried out by transmitting images and audio of the video conference participants from the client terminal 7, and receiving images and audio of other participants and viewing them on the terminal 7. The client terminal 7 may be an information terminal such as a personal computer or a smartphone.

As shown in the schematic block diagram of Figure 7, the client terminal 7 is configured to include a client terminal control unit 70, a display unit 71, a camera 72, a microphone 73, a character input unit 74, a speaker 75, a communication unit 78, and a memory unit 79.

The client terminal control unit 70 is configured to include a CPU (not shown) that controls each element that makes up the client terminal 7. The memory unit 79 is configured from an auxiliary storage device (not shown) and RAM (not shown).

The communication unit 78 is connected to the Internet 8 and transmits and receives data to and from servers that operate video conferences, such as the WEB/APP server 2. The data transmitted and received by this communication unit 78 includes image data, audio data, and the like.

The display unit 71 is composed of a display device such as an LCD display, and displays the video conference screens 200, 210, etc., created by the screen composition unit 21 of the WEB/APP server 2.

The camera 72 is composed of a solid-state imaging device such as a CCD image sensor or CMOS image sensor, and captures images of conference participants. The microphone 73 converts the speech of participants into electrical signals to obtain audio data. The speaker 75 reproduces the audio data of speech by other participants as sound.

The character input unit 74 consists of an input device such as a keyboard, and is used by participants to input characters.

Image data of participants captured by camera 72, audio data of participants' speech captured by microphone 73, and text data constituting sentences entered by participants via text input unit 74 are transmitted from communication unit 78 to the server running the video conference. Video conference display screen data and conference audio data are also received by communication unit 78 and displayed on display unit 71, with the audio output from speaker 75.

<Video conference screen>
8 and 9 show examples of video conference screens displayed on the display unit 71 of the client terminal 7. The video conference display screen configured by the screen configuration unit 21 of the WEB/APP server 2 is transmitted from the screen providing unit 26 to the client terminal 7, and the client terminal 7 receives this display screen and displays it on its display unit 71.

The video conference screen 200 shown in Figure 8 displays seven people participating in the conference. The area displaying each participant displays a photograph of that participant 201, an emotion level indicator 202 in the upper left corner of the area, a gender indicator 203 in the center left corner, and an estimated age indicator 204 in the lower left corner.

The emotion level display 202 is composed of emojis that express types of emotions such as "joy," "sadness," "anger," "disgust," "fear," and "surprise," and a percentage display of the strength of those emotions. Alternatively, the estimated probability of emotions such as "joy" or "sadness" may be displayed as a percentage, or the emotional state may be displayed as a percentage, with the worst emotional state being 0% and the best emotional state being 100%. Figure 10(a) shows an example of an emoji that displays the emotional level when the emotional state is displayed as a percentage, with the worst emotional state being 0% and the best emotional state being 100%. Five types of emojis are provided, corresponding to 20% increments of the emotion level.

The harassment level display may also display emojis that represent the type of harassment, such as "power harassment" or "sexual harassment," along with a percentage indication of the intensity of that harassment. Additionally, the estimated probability of harassment, such as "power harassment" or "sexual harassment," may also be displayed as a percentage. A percentage indication may also be displayed, where 0% represents an appropriate state with no harassment and 100% represents an inappropriate state with harassment. Figure 10(b) shows an example of emojis that display the harassment level as a percentage, with 100% representing a state with harassment and 0% representing a state without harassment. Five types of emojis are available, corresponding to 20% increments of the harassment level.

As shown in the video conference screen 210 in Figure 9, a participant who is speaking may be displayed enlarged on the display unit 71 of the client terminal 7. On this video conference screen 210, an image 201 of the participant is displayed, with a text display 205 located below it. This text display 205 displays a text message indicating the content of the participant's remarks. If translation is enabled, a translation of the text message may be displayed in this area.

Furthermore, an emotion level display 206 is located below and to the left of the text display 205, and a harassment level display 207 is located to the right of that display 206.

The emotion level display 206 and harassment level display 207 may not only be composed of emoticons and percentages, but may also display detailed analysis results, as shown in Figure 11. Figure 11 is an example of the emotion level display 208, which displays the estimated probability of items that make up emotions, such as "joy," "sadness," "anger," "disgust," "fear," and "surprise."

The emotion levels displayed in emotion level displays 202, 206, and 208 and the harassment level displayed in harassment level display 207 may be emotion levels or harassment levels obtained as a result of utterance analysis by analyzing text indicating the content of participants' utterances in the text analysis unit 41 of the text analysis server 4. Alternatively, the emotion levels may be obtained as an image analysis result by analyzing image data of participants in the image analysis unit 61 of the image and audio analysis server 6. Alternatively, the emotion levels or harassment levels may be obtained as an audio analysis result by analyzing audio data of participants in the audio analysis unit 64. Alternatively, the overall emotion levels or overall harassment levels may be obtained as a comprehensive judgment result by comprehensively evaluating the utterance analysis results, image analysis results, and audio analysis results in the judgment unit 24 of the WEB/APP server 2.

<Operation of the video conference analysis system>
Next, a description will be given of the operation of the video conference system 100 including the video conference analysis system 1 according to the present embodiment 1. The following describes the operation at the start of a video conference and the operation when a video conference participant speaks.

<Operations when starting a video conference>
FIG. 12 shows a schematic flowchart of the video conference analysis system 1 at the start of a video conference.

First, prior to the start of a video conference, one of the conference participants uses his or her client terminal 7 to set up the conference, including the conference participants, the participant's language, whether to convert the participants' speech into text and display it on the screen, and whether to use translation (step S100). The conference settings also include settings for the analysis details of the video conference, such as whether to analyze text indicating the content of participants' speech to estimate emotional levels and harassment levels, whether to analyze images of participants to estimate emotional levels, whether to analyze participants' speech data to estimate emotional levels and harassment levels, and whether to calculate overall emotional levels and overall harassment levels. This conference setting is sent from the client terminal 7 to the WEB/APP server 2, received by the server 2, and stored in the conference setting unit 22 (step S110).

Once the conference setup is complete, the participants input a conference start command into the client terminal 7 (step S101). The conference start command is sent from the client terminal 7 to the web/app server 2 and is accepted by the server 2 (step S111).

Image data of the participants is sent from the client terminal 7 to the WEB/APP server 2. In the WEB/APP server 2, the image and audio receiving unit 25 receives the image data of the participants (step S112). To analyze the acquired images of the participants, the WEB/APP server control unit 20 of the WEB/APP server 2 sends the participant image data to the image and audio analysis server 6 (step S113).

The image analysis unit 61 of the image and audio analysis server 6 analyzes the received participant image data, estimates the emotional level, and obtains the image analysis results (step S130). The image analysis unit 61 also estimates the gender and age based on the participant image data.

The analysis results of the participant image data by the image and audio analysis server 6 are sent to the WEB/APP server 2 and acquired by that server 2 (step S114).

Image data of other conference participants is also sent from the client terminals 7 to the WEB/APP server 2, which accepts the data (step S115). The WEB/APP server 2 similarly sends the participant image data to the image and audio analysis server 6 (step S116), which analyzes the data (step S131), and obtains the analysis results (step S117).

After receiving and analyzing participant image data from the client terminals 7 of all conference participants, the WEB/APP server control unit 20 of the WEB/APP server 2 causes the screen composition unit 21 to compose a video conference screen 200, for example, as shown in FIG. 8. The WEB/APP server control unit 20 then transmits the composed video conference screen 200 from the screen providing unit 26 to all client terminals 7 participating in the conference (step S118).

By doing this, a screen like the video conference screen 200 is displayed on the display unit 71 of the client terminal 7 (steps S102 and S140), and the video conference begins.

<What happens when a video conference participant speaks>
FIG. 13 shows a schematic flowchart of the video conference analysis system 1 when a video conference participant makes a statement.

When a participant speaks during a video conference (step S200), the participant's voice data is sent from the participant's client terminal 7 to the WEB/APP server 2. In addition, the participant's image data is constantly sent from the client terminal 7 to the WEB/APP server 2.

When the WEB/APP server 2 receives the participants' voice data and image data (step S210), the WEB/APP server control unit 20 of this server 2 transmits the received voice data to the voice-to-text conversion server 3 to convert the voice data into text.

The speech-to-text conversion unit 31 of the speech-to-text conversion server 3 converts the received voice data into text (step S220). The text converted from the voice data of the utterance forms a text sentence indicating the content of the utterance. The text data obtained by converting the voice data in this way is transmitted to the WEB/APP server 2.

When the WEB/APP server 2 receives character data constituting a text sentence from the speech-to-text conversion server 3 (step S211), the WEB/APP server control unit 20 transmits the received character data to the text analysis server 4 to analyze the received text sentence.

The text analysis unit 41 of the text analysis server 4 analyzes the text composed of the received text data to estimate the emotional level and harassment level (step S230). The result of this analysis of the text indicating the content of the participants' comments becomes the comment analysis result. The text analysis result is sent to the WEB/APP server 2 and accepted by that server 2 (step S212).

If the setting is to translate the text indicating the content of the statement, the WEB/APP server control unit 20 transmits the character data constituting the text, the language of the text, the translation language, etc. to the translation server 5 in order to perform the translation.

The translation unit 51 of the translation server 5 translates the received text into the translation language (step S240). The generated translation of the text is sent to the WEB/APP server 2 and accepted by that server 2 (step S213).

Next, in order to analyze the participant's image data and audio data, the WEB/APP server control unit 20 transmits the participant image data and audio data received from the client terminal 7 to the image and audio analysis server 6.

The image analysis unit 61 of the image and audio analysis server 6 analyzes the received participant image data to estimate the emotional level (step S250). The result of analyzing the participant image data in this way becomes the image analysis result. The audio analysis unit 64 analyzes the received participant audio data to estimate the emotional level and harassment level (step S250). The result of analyzing the participant audio data in this way becomes the audio analysis result.

The image analysis results and audio analysis results obtained by the image and audio analysis server 6 are sent to the web/app server 2 and accepted by that server 2 (step S214).

The WEB/APP server control unit 20 causes the screen composition unit 21 to compose a video conference screen 210, for example, as shown in FIG. 9, based on the received text, the analysis results of the text, the translation of the text, the image analysis results, and the audio analysis results. The WEB/APP server control unit 20 then transmits the composed video conference screen 210 from the screen providing unit 26 to all client terminals 7 participating in the conference (step S215).

By doing this, a screen like the video conference screen 210 is displayed on the display unit 71 of the client terminal 7 (steps S201 and S260), and the video conference proceeds.

<Effects of First Embodiment>
According to the first embodiment, audio data of speeches made by video conference participants is converted into text, and a text sentence indicating the content of the speech is constructed. Furthermore, an analysis is performed based on the constructed text sentence, and the emotional level and harassment level are estimated to obtain a speech analysis result. A display screen is then constructed that displays the obtained text sentence indicating the content of the participants' speeches and the speech analysis result, and this display screen is transmitted to the video conference participants. In this way, the emotional level and harassment level can be analyzed based on the participants' speeches, and the speech analysis result can be shared among the video conference participants. Therefore, if a participant becomes emotional, for example, the participant can recognize the state of mind and control themselves, and other participants can also recognize the state of mind and encourage them to calm down.

In addition, text indicating the content of each participant's remarks is sent to video conference participants and can be viewed on the screen, so even if the connection is poor and it is difficult to hear the audio, the conference can continue without interruption. Participants with hearing impairments can also participate in the conference.

Furthermore, according to this embodiment 1, the speech-to-text conversion unit 31 of the speech-to-text conversion server 3 converts speech data into text using a trained speech-to-text model generated by machine learning, which is artificial intelligence. This allows for highly accurate and stable conversion of speech data into text. Furthermore, the text analysis unit 41 determines the emotion level and harassment level using a trained text analysis model generated by machine learning, which is artificial intelligence. This allows for accurate and stable estimation of the emotion level and harassment level.

Furthermore, according to this first embodiment, text indicating the content of statements made by conference participants is translated into a specified translation language, allowing participants who speak different languages to refer to the translation to facilitate smooth communication.

Furthermore, according to this embodiment 1, the translation unit 51 of the translation server 5 translates the source text using a trained translation model generated by machine learning. This allows for highly accurate and reliable translation.

Furthermore, according to this first embodiment, the image analysis unit 61 of the image and audio analysis server 6 estimates the emotion level from the image data using a trained image analysis model generated by machine learning. This makes it possible to stably determine the emotion level from the image data with high accuracy.

Furthermore, according to this first embodiment, emotional levels are estimated based on image data of the conference participants, and image analysis results are obtained. An image analysis display screen that displays these image analysis results is then constructed and transmitted to the video conference participants. In this way, emotional levels are estimated based on the image data of the participants, and the image analysis results are shared with the conference participants. Because the image analysis results are obtained from the image data of the participants, emotional levels can be analyzed using data other than text that indicates the content of the participants' remarks, and multifaceted analysis results can be obtained.

Furthermore, according to this embodiment 1, the audio analysis unit 64 of the image and audio analysis server 6 estimates the emotion level and harassment level from the audio data using a trained audio analysis model generated by machine learning. This makes it possible to accurately and reliably detect the emotion level and harassment level from the audio data.

Furthermore, according to this first embodiment, emotional and harassment levels are estimated based on the audio data of the speech of the conference participants, and audio analysis results are obtained. A audio analysis display screen displaying these audio analysis results is then constructed and transmitted to the video conference participants. In this way, emotional and harassment levels are estimated based on the audio data of the participants' speech, and the audio analysis results are shared with the conference participants. Because the audio analysis results are obtained from the participants' audio data, the emotional and harassment levels can be analyzed using the participants' audio data itself, as opposed to text that indicates the content of the participants' speech, and analysis results can be obtained from multiple perspectives.

Furthermore, according to this first embodiment, a comprehensive assessment result is obtained by comprehensively evaluating the utterance analysis result obtained based on the text indicating the content of the participants' utterances, the image analysis result obtained based on the image data of the participants, and the audio analysis result obtained based on the audio data of the participants' utterances. Then, a comprehensive assessment display screen is constructed to display this comprehensive assessment result, and is transmitted to the participants in the video conference. In this way, the utterance analysis result, image analysis result, and audio analysis result are combined, thereby obtaining a more multifaceted analysis result.

Furthermore, according to this first embodiment, the text indicating the content of participants' remarks is checked for the inclusion of inappropriate words, and if inappropriate words are detected, an inappropriate word blocking measure is taken. This inappropriate word blocking measure prevents inappropriate words that may offend others from being communicated to conference participants, allowing participants to participate in the conference with peace of mind.

Furthermore, according to this first embodiment, if the detection frequency of inappropriate words and phrases exceeds a predetermined threshold, a warning display screen is sent to the participants. This warning display screen allows meeting participants to objectively recognize that inappropriate remarks are increasing, and they can take measures such as taking a break.

Second Embodiment
Next, a second embodiment of the present invention will be described with reference to Figures 14 to 17. However, elements that are the same as or correspond to those in the first embodiment described above are given the same reference numerals, and duplicated descriptions will be omitted.

Figure 14 is a block diagram showing a schematic configuration of a videoconference system 100A including a videoconference analysis system 1A according to the second embodiment. This videoconference system 100A is configured by adding a videoconference server 9 having the function of operating videoconferences to the videoconference system 100 according to the first embodiment described above. Furthermore, the WEB/APP server 2A included in this videoconference analysis system 1A does not operate videoconferences like the WEB/APP server 2 in the first embodiment, but instead provides a program for analyzing videoconferences as an API. Otherwise, the configuration is substantially the same as that of the videoconference system 100 according to the first embodiment.

In this videoconference analysis system 1A, a videoconference server 9 is connected to the client terminals 7 of the conference participants via the Internet 8, enabling the videoconference to operate. When analyzing the remarks, images, and audio of the conference participants to estimate their emotional and harassment levels, the videoconference server 9 obtains the analysis results using a videoconference analysis API provided by the web/app server 2A.

<WEB/APP server>
The WEB/APP server 2A provides a video conference analysis program in the form of a video conference analysis API that analyzes images and sounds of participants to estimate their emotional levels and harassment levels.

As shown in the schematic block diagram of Figure 15, the WEB/APP server 2A has a videoconference analysis API provider 27, but otherwise has a configuration similar to that of the WEB/APP server 2 in embodiment 1 (see Figure 2).

Figure 16 shows an example of the specifications of the video conference analysis API provided by the video conference analysis API provider 27. The arguments input to this video conference analysis API include analysis content settings, participant voice data, participant image data, and text data entered by participants via a keyboard or the like. The return values output from this video conference analysis API include the text after speech conversion, translated text, emotion level, emotion analysis details, harassment level, and harassment analysis details. Here, emotion analysis details refer to the proportions of each item such as "happiness" and "sadness" that make up the emotions shown in Figure 11. Similarly, harassment analysis details refer to the proportions of each item that makes up harassment.

By setting the analysis content, which is an argument to the video conference analysis API shown in Figure 16, it is possible to select from 12 patterns of analysis content.

In setting (1), audio data is converted into text (P1 processing). In this setting, audio data input as an argument to the videoconference analysis API is accepted by the image and audio accepting unit 25 of the WEB/APP server 2A (image and audio accepting processing), and the WEB/APP server control unit 20 of this server 2A sends the accepted audio data to the speech-to-text conversion server 3. The speech-to-text conversion unit 31 of the speech-to-text conversion server 3 performs processing to convert the accepted audio data into text (speech-to-text conversion processing). The text data converted from the audio data in this way is sent from the speech-to-text conversion server 3 to the WEB/APP server 2A, and is accepted by that server 2A.

In setting (2), text is analyzed and emotion and harassment levels are estimated (P2 processing). In this setting, text data input as arguments to the videoconference analysis API is received by the communication unit 28 of the WEB/APP server 2A, and the WEB/APP server control unit 20 transmits the received text data to the text analysis server 4. The text analysis unit 41 of the text analysis server 4 analyzes the text composed of the received text data, estimates the emotion and harassment levels, and obtains the text analysis results (text analysis processing). The text analysis results obtained in this way are transmitted from the speech-to-text conversion server 3 to the WEB/APP server 2A, where they are received.

In setting (3), the original text is translated and a translation is obtained (P3 processing). In this setting, character data input as an argument to the videoconference analysis API is accepted by the communication unit 28 of the WEB/APP server 2A, and the WEB/APP server control unit 20 sends the accepted character data to the translation server 5. The translation unit 51 of the translation server 5 translates the original text composed of the accepted character data and generates a translation (translation processing). The translation obtained in this way is sent from the translation server 5 to the WEB/APP server 2A, and is accepted by that server 2A.

In setting (4), images and audio are analyzed to estimate emotional and harassment levels (P4 processing). In this setting, image data and audio data input as arguments to the videoconference analysis API are accepted by the image and audio acceptance unit 25 of the WEB/APP server 2A (image and audio acceptance processing), and the WEB/APP server control unit 20 transmits the accepted image data and audio data to the image and audio analysis server 6. The image analysis unit 61 of the image and audio analysis server 6 analyzes the accepted image data, estimates the emotional level, and obtains image analysis results (image analysis processing). The audio analysis unit 64 analyzes the accepted audio data, estimates the emotional and harassment levels, and obtains audio analysis results (audio analysis processing). The image and audio analysis results obtained in this way are transmitted from the image and audio analysis server 6 to the WEB/APP server 2A, where they are accepted.

In setting (5), the audio data input as an argument to this videoconference analysis API is converted to text (speech-to-text conversion process), the text sentences made up of these converted characters are analyzed to obtain analysis results such as emotional levels (text sentence analysis process), and then this text sentences are translated to obtain translations (translation process) (P5 process).

In setting (6), the voice data input as an argument to this API is converted into text (voice-to-text conversion process), and the text composed of the converted characters is analyzed to obtain analysis results such as emotional level (text analysis process) (P6 process).

In setting (7), the voice data input as an argument to this API is converted into text (voice-to-text conversion process), and the text sentence composed of the converted characters is translated to obtain a translated sentence (translation process) (P7 process).

In setting (8), the text sentences composed of text data input as arguments to this video conference analysis API are analyzed to obtain analysis results such as emotional levels (text sentence analysis process), and these text sentences are translated to obtain translations (translation process) (P8 process).

Setting (9) is a combination of settings (5) and (4). Audio data input as an argument to this video conference analysis API is converted to text (speech-to-text conversion processing), and the text composed of the converted characters is analyzed to obtain analysis results such as emotional level (text analysis processing). This text is then translated to obtain a translation (translation processing). Image data input as an argument to this API is then analyzed to estimate emotional level and obtain image analysis results (image analysis processing), and audio data input to this API is analyzed to estimate emotional level and harassment level and obtain audio analysis results (audio analysis processing).

Setting (10) is a combination of settings (6) and (4), setting (11) is a combination of settings (7) and (4), and setting (12) is a combination of settings (8) and (4).

<Video conference server>
The video conference server 9 receives image and audio data of the conference participants transmitted from each client terminal 7, composes a video conference screen and audio of the conference based on this image and audio data, and transmits it to each client terminal 7. By performing this operation, the conference participants can view the conference screen and audio via each client terminal 7, and the video conference can proceed. In this way, the video conference server 9 has the function of realizing a video conference.

As shown in the schematic block diagram of Figure 17, the video conference server 9 is configured to include a video conference server control unit 90, a screen configuration unit 91, a conference setting unit 92, a minutes recording unit 93, an image and audio receiving unit 95, a screen providing unit 96, a video conference analysis API calling unit 97, a communication unit 98, and a memory unit 99.

The videoconference server control unit 90 is configured to include a CPU (not shown) that controls each element that makes up the videoconference server 9. The memory unit 99 is configured from an auxiliary storage device (not shown) and RAM (not shown).

The communication unit 98 is connected to the Internet 8 and transmits and receives data between each server and each client terminal 7 that make up this videoconference analysis system 1A.

The image and audio receiving unit 95 performs substantially the same functions as the image and audio receiving unit 25 of the WEB/APP server 2 in embodiment 1. Based on the control of the video conference server control unit 90, this image and audio receiving unit 95 receives captured images of participants and audio data of participants' speech transmitted via the Internet 8 from each client terminal 7 of the conference participants.

The screen composition unit 91 performs substantially the same functions as the screen composition unit 21 of the WEB/APP server 2 according to the first embodiment. The screen composition unit 91 composes the video conference screen to be displayed on each client terminal 7 under the control of the video conference server control unit 90.
The screen providing unit 96 performs substantially the same function as the screen providing unit 26 of the WEB/APP server 2 according to the first embodiment. Based on the control of the video conference server control unit 90, the screen providing unit 96 transmits and provides the screen created by the screen creating unit 91 to each client terminal 7 of the participants.

The conference setting unit 92 performs substantially the same functions as the conference setting unit 22 of the WEB/APP server 2 in embodiment 1. This conference setting unit 92 is used to set conference participants, conference analysis content, and other settings prior to the conference.

The minutes recording unit 93 performs substantially the same functions as the minutes recording unit 23 of the WEB/APP server 2 in embodiment 1. This minutes recording unit 93 records the audio data of participants' remarks received by the image and audio receiving unit 95, and records text data obtained by converting the participants' audio data into text. Furthermore, it may also be configured to record the results of analysis based on the participants' images and audio data.

When analyzing participant image data and voice data, the video conference analysis API call unit 97 calls the video conference analysis API provided by the WEB/APP server 2A. This video conference analysis API call unit 97 inputs analysis content settings, participant voice data, participant image data, etc. into the video conference analysis API, and obtains analysis results, etc. from the video conference analysis API.

<Effects of the Second Embodiment>
According to the second embodiment, substantially the same effects as those of the first embodiment can be obtained.

According to the second embodiment, the audio data of speeches made by video conference participants is converted into text, and a text sentence indicating the content of the speech is constructed. Analysis is then performed based on the constructed text sentence, and the emotional level and harassment level are estimated to obtain a speech analysis result. In this way, the emotional level and harassment level can be analyzed based on the speeches made by video conference participants.

In addition, the video conferencing analysis program can be provided as an API, which means that the API can be used in a variety of video conferencing systems, making it versatile.

Furthermore, according to this second embodiment, the text indicating the content of the statements made by the conference participants is translated, so participants who speak different languages can refer to the translations to facilitate smooth communication.

Furthermore, according to this second embodiment, emotional levels are estimated based on image data of conference participants, and image analysis results are obtained. This makes it possible to analyze emotional levels using data other than text that indicates the content of participants' remarks, and obtain multifaceted analysis results.

Furthermore, according to this second embodiment, emotional and harassment levels are estimated based on the audio data of the speech of the conference participants, and audio analysis results are obtained. Therefore, unlike text that indicates the content of the participants' speech, the emotional and harassment levels can be analyzed using the participants' audio data itself, and multifaceted analysis results can be obtained.

[Other embodiments of the invention]
Although the videoconference analysis systems 1 and 1A according to the first and second embodiments are configured with multiple servers installed for different functions, all functions may be integrated into a single server.

Furthermore, the "network" is not limited to the Internet 8, but may be composed of other networks such as a local area network or a wide area network (WAN).

100, 100A...Video conference system, 1...Video conference analysis system, 2, 2A...WEB/APP server, 3...Speech-to-text conversion server, 4...Text analysis server, 5...Translation server, 6...Image and speech analysis server, 7 ₁ , 7 ₂ , 7 ₃ , ..., 7 _n , 7...Client terminal, 8...Internet (network), 9...Video conference server, 20...WEB/APP server control unit (control unit), 21...Screen composition unit, 22...Conference setting unit, 23...Proceedings recording unit, 24...Determination unit, 25...Image and audio receiving unit, 26...Screen providing unit, 27...Video conference analysis API providing unit, 31...Speech to text conversion unit, 34...Inappropriate phrase storage unit, 35...Inappropriate phrase detection unit, 41...Character and sentence analysis unit, 51...Translation unit, 61...Image analysis unit, 64...Speech analysis unit, 71...Display unit, 72...Camera, 73...Microphone, 74...Character input unit, 75...Speaker, 97...Video conference analysis API calling unit, 200, 210...Video conference screen, 201...Participant image, 202, 206, 208...Emotion level display, 205...Character and sentence display, 207...Harassment level display

Claims (16)

A video conference analysis system for analyzing statements made by participants in a video conference held over a network, comprising:
an image and audio receiving unit that receives audio data of the participants' speeches via the network;
a voice-to-text converter for converting the voice data received by the voice image receiver into text;
a character sentence analysis unit that analyzes character sentences that indicate the contents of the participants' statements, which are composed of characters converted from the voice data by the voice-to-text conversion unit, and obtains a statement analysis result;
a screen configuration unit that configures a display screen that displays the text and the utterance analysis result obtained by the text analysis unit;
a screen providing unit that transmits the display screen configured by the screen configuration unit to the participants participating in the video conference via the network;
a control unit that converts the voice data received by the image and voice receiving unit into text using the voice-to-text conversion unit to obtain the text sentence, analyzes the text sentence using the text sentence analysis unit to obtain the utterance analysis result, configures the display screen using the screen configuration unit, and controls operations to transmit the display screen from the screen providing unit,
The character and sentence analysis unit
A video conference analysis system characterized by performing an analysis based on the text indicating the content of the participants' remarks, estimating an emotional level indicating their emotional state and a harassment level indicating the degree of harassment, and obtaining the remark analysis results. The speech-to-text conversion unit
a trained speech-to-text conversion model is generated by machine learning using a combination of training speech data and correct character data corresponding to the training speech data as training data, and the speech data received by the image/speech receiving unit is input and converted into characters by calculation to obtain the character sentence;
The character and sentence analysis unit
The videoconference analysis system of claim 1, characterized in that the acquired text is input into a trained text analysis model generated by machine learning using combinations of training texts and correct emotion data corresponding to the training texts, and combinations of the training texts and correct harassment data corresponding to the training texts, and the acquired text is calculated to estimate the emotion level and the harassment level and obtain the utterance analysis results. a translation unit that translates an original text written in a source language into a predetermined translation language;
The control unit
Controlling an operation of translating the text indicating the content of the speech of the participant by the translation unit to obtain a translated text;
The screen configuration unit
a translation display screen that displays the translation of the character sentence translated by the translation unit;
The screen providing unit
3. The video conference analysis system according to claim 1, wherein the translation display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. The translation unit
The videoconferencing analysis system according to claim 3, characterized in that a combination of a training original sentence and a correct translation sentence that is the correct translation of the training original sentence is used as training data, and the original sentence is input into a trained translation model generated by machine learning, and a translation of the training original sentence is obtained by performing calculations on the trained translation model. an image analysis unit that performs analysis based on image data of the participants and estimates the emotion level to obtain an image analysis result;
The image and sound receiving unit
receiving the image data of the participants via the network;
The control unit
Controlling an operation of causing the image analysis unit to analyze the image data received by the image and audio receiving unit;
The image analysis unit
obtaining the image analysis results by inputting the image data into a trained image analysis model generated by machine learning using a combination of a training face image and a correct emotion type corresponding to the training face image as training data, and performing a calculation;
The screen configuration unit
configuring an image analysis display screen that displays the image analysis results obtained by the image analysis unit;
The screen providing unit
The video conference analysis system according to any one of claims 1 to 4, characterized in that the image analysis display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. an image analysis unit that performs analysis based on image data of the participants and estimates the emotion level to obtain an image analysis result;
The image and sound receiving unit
receiving the image data of the participants via the network;
The control unit
Controlling an operation of causing the image analysis unit to analyze the image data received by the image and audio receiving unit;
The image analysis unit
extracting a face image from the image data, and performing analysis using at least one of the following: eye shape, changes in the eye shape, eyebrow shape, changes in the eyebrow shape, shapes of the corners of the mouth which are the sides of the lips, changes in the shape of the corners of the mouth, cheek shape, changes in the cheek shape, frequency of appearance of teeth, and changes in the frequency of appearance of teeth; and obtaining the image analysis results;
The screen configuration unit
configuring an image analysis display screen that displays the image analysis results obtained by the image analysis unit;
The screen providing unit
The video conference analysis system according to any one of claims 1 to 4, characterized in that the image analysis display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotion level and the harassment level, and obtains a voice analysis result;
The control unit
Controlling an operation to cause the voice analysis unit to analyze the voice data;
The voice analysis unit
a trained voice analysis model is generated by machine learning using a combination of training voice data and correct emotion data corresponding to the training voice data, and a combination of the training voice data and correct harassment data corresponding to the training voice data, and the received voice data is inputted into the trained voice analysis model, and calculations are performed to estimate the emotion level and the harassment level, thereby obtaining the voice analysis result;
The screen configuration unit
configuring a voice analysis display screen that displays the voice analysis result obtained by the voice analysis unit;
The screen providing unit
The video conference analysis system according to any one of claims 1 to 4, characterized in that the voice analysis display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotion level and the harassment level, and obtains a voice analysis result;
The control unit
Controlling an operation to cause the voice analysis unit to analyze the voice data;
The voice analysis unit
performing an analysis using at least one of the following: voice volume, changes in voice volume, voice pitch, changes in voice pitch, speaking rate, changes in speaking rate, frequency of speaking over the words of other participants, and changes in the frequency of speaking over the words of the other participants; and obtaining the voice analysis result;
The screen configuration unit
configuring a voice analysis display screen that displays the voice analysis result obtained by the voice analysis unit;
The screen providing unit
The video conference analysis system according to any one of claims 1 to 4, characterized in that the voice analysis display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. an image analysis unit that performs analysis based on image data of the participants and estimates the emotion level to obtain an image analysis result;
a voice analysis unit that performs analysis based on the voice data received by the image and voice receiving unit, estimates the emotion level and the harassment level, and obtains a voice analysis result;
a judgment unit that comprehensively evaluates the statement analysis result analyzed by the character and sentence analysis unit, the image analysis result obtained by the image analysis unit, and the voice analysis result obtained by the voice analysis unit, and estimates at least one of a comprehensive emotion level and a comprehensive harassment level to obtain a comprehensive judgment result,
The image and sound receiving unit
receiving the image data of the participants via the network;
The control unit
The image data is analyzed by the image analysis unit, the voice data is analyzed by the voice analysis unit, and the judgment unit comprehensively evaluates the utterance analysis result analyzed by the character and sentence analysis unit, the image analysis result obtained by the image analysis unit, and the voice analysis result obtained by the voice analysis unit, thereby controlling the operation of obtaining the comprehensive judgment result;
The screen configuration unit
configuring a comprehensive judgment display screen that displays the comprehensive judgment result obtained by the judgment unit;
The screen providing unit
2. The video conference analysis system according to claim 1, wherein the comprehensive judgment display screen constructed by the screen construction unit is transmitted to the participants participating in the video conference via the network. an inappropriate phrase detection unit that detects whether the text contains inappropriate phrases that have been registered in advance;
When the inappropriate phrase is detected by the inappropriate phrase detection unit,
The control unit
The video conference analysis system of any one of claims 1 to 9, characterized in that it takes measures to block inappropriate phrases, including at least one of stopping transmission of the audio data corresponding to the inappropriate phrases sent via the network to the participants participating in the video conference, deleting the inappropriate phrases included in the text, and replacing the inappropriate phrases included in the text with appropriate phrases corresponding to the inappropriate phrases. When the detection frequency of the inappropriate phrase detected by the inappropriate phrase detection unit exceeds a predetermined threshold,
The control unit
11. The video conference analysis system of claim 10, wherein the screen composition unit composes a warning display screen, and the screen providing unit controls an operation of transmitting the warning display screen composed by the screen composition unit to the participants participating in the video conference via the network. A video conference analysis program for analyzing statements made by participants in a video conference conducted via a network,
an image and voice receiving process for receiving voice data of the participants via the network;
a voice-to-text conversion process for converting the voice data received in the voice and image receiving process into text;
a text analysis process for analyzing text indicating the content of the speech of the participant, which is composed of characters converted from the voice data in the voice-to-text conversion process, to obtain a speech analysis result;
The character sentence analysis process includes:
A video conference analysis program characterized by performing an analysis based on the text indicating the content of the participants' remarks, estimating an emotional level indicating their emotional state and a harassment level indicating the degree of harassment, and obtaining the remark analysis results. A translation process is provided for translating an original text written in a source language into a predetermined translation language,
The translation process includes:
13. The video conference analysis program according to claim 12, wherein the translated sentence is generated by translating the text indicating the content of the speech of the participant. an image analysis process for performing an analysis based on image data of the participants, estimating the emotion level, and obtaining an image analysis result;
The image and sound reception process includes:
receiving the image data of the participants via the network;
The image analysis process includes:
13. The video conference analysis program according to claim 12, wherein a facial image is extracted from the image data, and an analysis is performed using at least one of the following: eye shape, changes in the eye shape, eyebrow shape, changes in the eyebrow shape, shapes of the corners of the mouth (the parts on both sides of the lips), changes in the shape of the corners of the mouth, cheek shape, changes in the cheek shape, frequency of appearance of teeth, and changes in the frequency of appearance of teeth, to obtain the image analysis result. a voice analysis process for performing an analysis based on the voice data received in the image and voice receiving process, and estimating the emotion level and the harassment level to obtain a voice analysis result;
The voice analysis process includes:
13. The video conference analysis program according to claim 12, wherein the voice analysis result is obtained by performing analysis using at least one of the following: voice volume, changes in voice volume, voice pitch, changes in voice pitch, speaking rate, changes in speaking rate, frequency of speaking over the speech of other participants, and changes in frequency of speaking over the speech of the other participants. an image analysis process for performing an analysis based on image data of the participants, estimating the emotion level, and obtaining an image analysis result;
a voice analysis process for performing an analysis based on the voice data received in the image and voice receiving process, and estimating the emotion level and the harassment level to obtain a voice analysis result;
a judgment process for comprehensively evaluating the statement analysis result obtained by the character and sentence analysis process, the image analysis result obtained by the image analysis process, and the voice analysis result obtained by the voice analysis process, and estimating at least one of a comprehensive emotion level and a comprehensive harassment level to obtain a comprehensive judgment result;
The image and sound reception process includes:
receiving the image data of the participants via the network;
the image analysis process analyzes the image data to obtain the image analysis result;
the voice analysis process analyzes the voice data to obtain the voice analysis result;
13. The video conference analysis program according to claim 12, wherein the judgment process determines the overall judgment result by comprehensively evaluating the utterance analysis result obtained by the character sentence analysis process, the image analysis result obtained by the image analysis process, and the audio analysis result obtained by the audio analysis process.