JP7653641B2 - Online conference system - Google Patents

Description

The present invention relates to an online conference system that is connected to multiple terminals via a network and enables online conferences between the terminals.

In an online conference system, multiple terminals are connected via the Internet, and online conferences are realized by distributing audio and image data between the terminals. Images of other participants are displayed on the terminals of each participant in an online conference, and audio can be heard. Using such a system, multiple people can hold conferences and conversations with each other.

Various ideas have been proposed to improve the convenience of online conference systems. For example, Patent Document 1 discloses a technology that introduces artificial intelligence into online conferences to perform voice recognition, report creation, emotion analysis, and the like. Patent Document 2 discloses a technology in online conferences that adjusts the left and right volumes of stereo sound to output sound from a direction corresponding to the position of a participant's icon on the screen. Patent Document 3 discloses a technology in a web conference system that prompts users to input text when the noise volume exceeds a certain level.

JP 2008-63699 A Japanese Patent Application Publication No. 6-175942 JP 2019-117998 A

However, there is still room for improvement in various aspects of the online conference system.
The present invention aims to further improve the convenience of online conference systems.

The present invention provides, as a first configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a distribution control unit for distributing audio data and image data between the plurality of terminals;
The distribution control unit is
If, during the online conference, one of the terminals receives a limited conversation instruction to designate one of the multiple terminals, including that terminal, as a specific terminal and to conduct a conversation only between the specific terminals, the online conference system can perform limited distribution that prohibits the audio data exchanged between the specific terminal from being distributed in a state that can be played on other terminals.

In conventional online conference systems, when a large number of people are participating in an online conference, the only options are to mute the speaker so that no one can hear what you say, or to make sure that everyone else in the conference can hear what you say. The host of the online conference also has the ability to divide the conference room into several rooms and hold conversations in each room, but the same rule applies to comments in each conference room.
However, in real-life meetings, it is not always the case that participants speak to everyone. For example, participants may discuss things in a low voice with the person next to them. In particular, in meetings between companies, participants may discuss things only with the participants from the same company before speaking. Conventional online meeting systems could not meet such demands.
Recently, some social gatherings are held using online conference systems, but in real social gatherings that are not held using online conference systems, several groups of participants may be formed and several conversations may be held together. Participants can freely enjoy conversations with various participants by joining one conversation and then another at their own will. Conventional online conference systems did not allow such conversations with only a portion of the participants to take place at the will of each participant.
In contrast, according to the present invention, during an online conference, if any of the participants issues a limited conversation instruction from their own terminal, the voice data exchanged between the specific terminal will not be distributed in a state that allows playback on other terminals. In other words, conversation can be established only between specific terminals. Therefore, for example, during an online conference, it becomes possible for only some of the participants to hold a discussion. Also, at a social gathering, participants can split into multiple groups and talk freely.
Furthermore, in real-life meetings and social gatherings, conversations tend to be limited to those who are close to you, but the online conference system has the advantage that conversations can be held without such location restrictions. In other words, the first configuration can provide an advantage that real-life meetings and social gatherings do not have, in that location restrictions can be avoided.
Prohibiting distribution in a state where it can be played on other terminals includes a method of prohibiting the distribution of audio data itself, and a method of distributing audio data with a flag or the like that prohibits playback on the prohibited terminal. In addition, when image data and audio data are integrated, image data with audio and image data without audio may be generated and used separately.

In the first configuration, for example, the limited distribution may be established when a terminal transmits a limited conversation instruction to another terminal. Also, after transmitting the limited conversation instruction, the limited distribution may be established when consent is obtained from the destination terminal.

In the first configuration, image data may be distributed in a limited manner in the same way as audio data, thereby preventing other participants from seeing the state of the conversation through limited distribution.
As another embodiment, the image data may be distributed to all participants as usual. In this way, the image data will not disappear from the screen distributed to other participants even during limited distribution, so that conversations can be held through limited distribution without causing discomfort to other participants.

In the first configuration, the specific terminal can be specified in various ways. For example, the participants in the online conference, i.e., each terminal used by these participants, can be specified individually.

In addition, the specification of a specific terminal is
a conference database for storing group data in which a part of the terminals is grouped in advance;
The distribution control unit may, when receiving a limited conversation instruction targeted at the group, treat a terminal belonging to the group as the specific terminal.

In this way, limited conversation instructions can be easily given on a group basis.
In the above aspect, the limited distribution may be established within the group at the same time as transmitting the limited conversation instruction, or the limited distribution may be established only when consent is obtained from each participant in the group. In this case, the limited distribution may be established when consent is obtained from all participants in the group, or may be established only with some of the participants for whom consent has been obtained.
In the above aspect, a plurality of types of groups may be set in advance, which makes it possible to select a group for each online conference scene and perform limited distribution.

In the first configuration,
Even during the limited distribution, audio data from terminals other than the specific terminal may be distributed to at least some of the specific terminals.

This allows users to listen to audio from other terminals even during limited distribution, making it possible to grasp the overall progress of the online conference even during limited distribution.
The volume of the audio data in the above embodiment may be the same as that between the specific terminals, or may be lower than that. Also, the volume may be adjustable on the receiving terminal side independently of the specific terminal.

In the first configuration,
The distribution control unit is
During the limited distribution, when a new limited conversation instruction is received from a terminal other than the specific terminal by specifying a part of the specific terminal,
An inquiry as to whether or not to approve the limited conversation instruction may be made to all of the specific terminals, regardless of whether or not they are designated in the limited conversation instruction.

When a limited distribution is in progress, a new participant may appear who wishes to join. In such a case, if the participation is accepted carelessly, some of the participants in the limited distribution may continue the conversation without recognizing the new participant, and the contents of the conversation may be leaked to unintended persons.
In contrast, in the above embodiment, when a terminal performing limited distribution receives a limited conversation instruction from a new participant, all participants in the limited distribution are queried as to whether or not they approve the participation, so that the new participation can be approved with the knowledge of all participants, thereby avoiding the above-mentioned problem.
In the above embodiment, new participation may be permitted when approval is obtained from all inquired terminals, or when approval is partially obtained. In addition, when approval is only partially obtained, the specified terminal that did not approve may cancel the designation of the specified terminal.

In the first configuration,
The image generating unit may generate a screen on which an instruction icon for designating the terminal as the specific terminal is superimposed on image data from the terminal.

By doing this, on each person's terminal, the image of the other participants sent from the other terminals is displayed together with the instruction icon within the image. Therefore, by operating the instruction icon, it becomes easy to select individual participants and send limited conversation instructions.
The pointing icon may be, for example, a selection button, or the entire image may be made clickable and used as a pointing icon.

In the first configuration,
The device may have a screen generating unit that generates a screen displayed on the terminal when the limited distribution is being performed, which associates images from the specific terminal with each other and displays a release icon that can be used to give an instruction to release the limited distribution.

By doing this, images of other participants sent from other terminals are displayed on each person's terminal, and images such as lines linking the specific terminals that are the subject of limited distribution are displayed, so that it is easy to recognize which participant a conversation is being held with via limited distribution.
In addition, an image such as a line also displays a release icon for releasing the specific distribution, so that the limited distribution between specific participants can be released easily and without error by this operation, improving operability.
The image that associates the specific terminals with each other may be, for example, a line connecting them, or a frame surrounding them.

The present invention provides, as a second configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a distribution control unit that distributes audio data and image data between the plurality of terminals;
The online conference system may include a screen generating unit that displays a volume icon for individually specifying the volume of the terminal by superimposing it on the image data of the terminal.

In conventional online conference systems, when a large number of people are participating in an online conference, the voices of the other participants can only be heard at a uniform volume.
However, online conference systems can be used for various purposes, and nowadays, social gatherings are sometimes held using online conference systems. In real social gatherings, several groups of several people are formed among the participants, and conversations on various topics are often mixed. Also, while participating in one topic, a participant may also join in another topic if he or she becomes interested in that topic. In conventional online conference systems, the conversations of other participants are heard at a uniform volume, making it difficult to hold a conversation in such a situation.
In contrast, according to the second configuration, it is possible to freely adjust and listen to what each participant in an online conference is saying, making it possible to participate in a conversation without hindrance even in a situation where various conversations are being held together.
In addition, the second configuration has the advantage of being simple since it is only necessary to adjust the volume from other terminals on one's own terminal, and there is no need to obtain approval from other terminals.

The second configuration can be realized in a variety of ways.
For example, when the audio data from each terminal is distributed as independent data without being integrated in an online conference system, each terminal may independently play the audio data from each terminal. This makes it possible to easily adjust the volume of each audio data according to the instruction of the volume icon. When a smartphone, tablet, computer, or the like is used as a terminal, a function for playing multiple audio data individually and adjusting the volume may be provided by the so-called OS (Operating System). In such a case, as a second configuration, the volume icon may have a function for giving an instruction of the volume to the OS based on the operation result.
Alternatively, the audio data from each terminal may be synthesized at a volume in accordance with the volume icon and then played back. This synthesis may be performed by the online conference system or by the receiving terminal.

The present invention provides, as a third configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a distribution control unit that distributes audio data and image data between the plurality of terminals;
The online conference system may include a screen generating unit that receives a reaction from any one of the terminals addressed to any other terminal, and displays the reaction on the other terminal.

In real-world meetings, participants can guess whether other participants agree or disagree with their comments based on their facial expressions and behavior. The same is true in online meeting systems, where participants can guess whether other participants agree or disagree with their comments based on the facial expressions displayed on their own devices.
However, in an online conference system, there is a limit to what can be understood from the images of other participants displayed on one's own terminal. Conventionally, in an online conference system, little consideration has been given to a method for effectively understanding the approval or disapproval of other participants regarding one's own remarks.
In contrast, according to the third configuration, each participant in an online conference can individually send a "reaction" from one terminal to the other terminals. In this way, the participant's terminal displays the reactions of other participants to a statement while the participant is speaking. Therefore, the speaker can easily and objectively grasp the reactions of other participants to the speaker's statement.
Conversely, the third configuration can be said to be advantageous for participants other than the speaker. That is, in a real conference or a conventional online conference system, it is not possible to deliver one's "reaction" to a speaker who is speaking, but with the third configuration, this becomes possible. For example, if one thinks a comment is good, one can show the speaker a reaction of "agreement" without interrupting the comment. In this respect, the third configuration can be said to provide an advantage unique to online conference systems.

In the third configuration, the "reaction" can take various forms,
The reaction may be information represented by a mark or text prepared in advance.

The prepared marks may be, for example, a mark expressing agreement with an opinion, a mark expressing disagreement, a mark expressing a reaction such as surprise, etc. Various stamps used on SNS etc. may also be used. By using these marks, a reaction can be delivered with a simple operation by simply selecting one of them.
Also, by making it possible to send text, it becomes possible to deliver specific opinions to the person making the comment.

In the third configuration, the interface for sending marks and text can take various forms. For example, a list of prepared marks may be displayed on the speaker's screen displayed on his or her own device, and the mark may be sent by selecting one of the marks. Alternatively, a list of marks may be displayed outside the area where the image of each device is displayed, and one of the marks may be selected and dragged onto the speaker's screen.

In a third configuration,
a sentiment analysis unit that performs sentiment analysis of a user of the terminal based on image data from the terminal;
The reactions may be a result of the sentiment analysis.

This allows each participant to get a reaction without having to do anything.
The emotion analysis method can apply various techniques for analyzing emotions from facial expressions.
The result of the sentiment analysis may be expressed by a prepared mark such as "agree" or "disagree", text, or a score. For example, it may be expressed by the brightness or color of the screen.

The content of the emotion analysis and the method of using it may take various forms. For example, when analyzing emotions from facial expressions, attention may be paid to the gaze. It is possible to evaluate whether the user's gaze has left the screen, and judge the favorability of the user's comment depending on the number and frequency of such actions. It is preferable to feed back the favorability of the comment to the speaker in the various forms described above.
The line of sight can be determined, for example, from the position of the pupils in an image of the user's face. When the pupils enter a specified range around the eyes, it is recognized that they have moved off the screen. In this case, since the camera of the user's terminal is not necessarily positioned at the center of the screen, the position of the pupils when looking at the center of the screen may be stored as a reference position at the start of an online conference. A method can be used in which it is determined that the pupils have moved off the screen when they move a specified distance away from this initial position.
It may also be possible to determine whether the user is dozing off based on the degree to which the user's eyes are open, the shaking of the head, etc. In this case, the comment may be treated as an indication that the comment is boring, and feedback may be given to the commenter, or a warning may be issued to the user to wake up.
Furthermore, these emotion analyses can be performed on all participants in an online conference. For example, if there is an increase in the percentage of participants who are judged to be lacking concentration, such as dozing off, a message may be displayed encouraging the organizer or all participants to take a break.

In a third configuration,
The screen generating unit may display reactions from other terminals superimposed on image data of the user of the terminal.

This allows users at each terminal to know the reactions of the other participants to themselves.
During an online conference, not only the participant's image but also the images of the other participants are displayed on the participant's terminal, but the participant's reaction to each participant may also be displayed on the image of the other participants. This makes it possible to compare the reaction of each participant to the other participants with the reaction of each participant to the participant himself/herself.

In the third configuration,
The screen generation unit may display a reaction from another terminal to the terminal in a superimposed manner on an image of the other terminal that is the sender of the reaction.

The image of each participant is displayed on the participant's own device. In the above embodiment, when a participant sends a reaction to the participant, the reaction is displayed on the image of that participant. This makes it easy to know who sent the reaction to the participant.

The present invention provides, as a fourth configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
an analysis processing unit that collects one or both of the times when the users of the terminals are speaking and the times when they are silent;
The online conference system may include a screen generation unit that displays the tally results on the terminal.

In a meeting, the comments made by participants may be biased towards a few participants.
However, in conventional online conference systems, although participants could mutually distribute voice and images and hold a conference, it was not possible to objectively grasp the results of the conference.
In contrast, the fourth configuration makes it possible to count either or both of the speech time and the silence time for each terminal. Therefore, it is possible to grasp how much each participant is speaking. As a result, it is possible to contribute to improving the quality of the conference.

In the fourth configuration, only one's own tally results may be displayed, or other people's tally results may be displayed together.
The tabulated results may be displayed during the online conference, or as a review after the online conference.
The content of the count can be, for example, the time for each speech, the average speech time, the total speech time, etc. The same applies to the time of silence.

In the fourth configuration,
The analysis processing unit may present a speech duration per one speech by the user of the terminal.

By doing this, each participant can judge the tendency of the length of their own speech. As a result, the content and quality of their speech can be improved. Also, for example, if a speech that is too long can be shortened, it is possible to give other speakers a chance to speak, which contributes to improving the quality of the entire meeting.
In the fourth configuration, the presentation time may be presented during the online conference, which makes it possible to improve the quality of the speech and the quality of the online conference at any time.
In the fourth configuration, the speech time per speech may be presented in a form that allows comparison with other participants.
The data may also be presented in the form of a distribution of speech duration and number of speeches, which makes it possible to grasp speech trends in more detail, such as a tendency to make short speeches many times, or a tendency to make long speeches relatively few times.

In the fourth configuration,
The analysis processing unit may obtain a proportion of speaking time from each terminal in the online conference.

This allows participants in online conferences to grasp the balance of what is being said, which contributes to improving the quality of the online conference as a whole.
The percentage of speech time can be calculated in various ways. For example, the percentage of speech time relative to the total time of the online conference may be calculated. Alternatively, the percentage of speech time relative to the total sum of speech times may be calculated. Furthermore, in order to exclude simple replies, etc., only speeches of a certain length or more may be included in the calculation of speech time.

The present invention provides, as a fifth configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a voice data input unit for inputting voice data from the terminal;
a keyword extraction unit that extracts keywords contained in the voice data;
The online conference system may include a screen generating unit that displays the keyword on the terminal in association with the voice data.

Conventionally, in online conference systems, it has been possible to record statements made during a conference as audio data.
However, if one wanted to check these statements later, it would be necessary to listen to the entire content again, which was a huge burden.
In contrast, according to the fifth configuration, keywords can be extracted based on the speech data of the remarks and displayed in association with the speech data. This allows the content of each remark to be inferred later from the keywords, improving the efficiency of creating minutes and checking speech data.
The extracted keywords may be stored in association with the voice data.

In the fifth configuration, the keywords can be extracted in various ways, for example using artificial intelligence.
As another method, speech data may be recognized and converted into text data, words may be extracted by morphological analysis, and the extracted words may be ranked according to a predetermined rule to extract keywords.

The extraction of keywords can be performed at various times.
For example, extracting keywords during an online meeting and presenting them to all participants can help them understand what is being said.
Furthermore, when extracting keywords after an online conference, the processing can take more time, making it possible to perform extraction with higher accuracy.

The present invention provides, as a sixth configuration,
An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a conference database that stores the start and end times of each statement made from each terminal in the online conference as conference data;
The online conference system may include a screen generation unit that displays a Gantt chart that depicts the period during which the statements were made, divided for each terminal along a timeline based on the conference data.

Improving the efficiency and quality of meetings is always a challenge. One of the important aspects is to understand the contributions of each participant in the meeting.
However, in conventional online conference systems, little consideration was given to understanding the history of speech during a conference.
In contrast, according to the sixth configuration, the start and end times of each utterance are stored for each utterance, and can be displayed in a Gantt chart by dividing them by terminal, i.e., by participant using the terminal. This makes it possible to visually grasp which participant spoke during the conference, how frequently, and for what time. As a result, participants who speak at length can be prompted to summarize their utterances, and participants who have not spoken at all can be prompted to speak, improving the quality of the conference as a whole.
In the sixth configuration, the Gantt chart described above may be displayed during the meeting or after the meeting.

In a sixth configuration,
The screen generating unit may be configured, when a statement is made from one terminal during a period in which a statement is made from another terminal, to highlight a portion of the statement from the other terminal that overlaps in time with the statement from the one terminal.

By doing this, it is possible to visually and objectively grasp the point at which someone else's remarks have begun before they have finished. Making remarks in this manner not only robs others of the opportunity to speak, but also hinders the smooth and fair progress of the meeting. Therefore, by visualizing such inappropriate remarks, it is possible to improve the efficiency and quality of the meeting.
The highlighting can be done in various ways, such as highlighting with color, hatching or other patterns, surrounding with a thick line, blinking, and the like.

In a sixth configuration,
The conference database stores reactions from other terminals to the speech in association with each other,
The screen generating unit may display the reaction in association with the comment.

This allows you to visually and objectively grasp the reactions of other participants to each comment, which has the advantage of making it easier to determine whether each comment was agreed upon or not, making it easier to extract important comments when reviewing online meetings.
As described above, various aspects of the reaction can be used, such as marks, text, and the results of sentiment analysis.

In a sixth configuration,
The screen generating unit may display a keyword for each utterance identified based on the conference database in association with the utterance.

This has the advantage that the content of each comment can be inferred from the keywords, making it easier to extract important comments, such as comments whose audio data should be checked, when reviewing online meetings.
The extraction of keywords can be performed using the various methods described above. The extraction of keywords may be performed when the Gantt chart is drawn, or may be performed before the Gantt chart is drawn and the results are stored.

The present invention does not necessarily have to include all of the various features described above, and some of them may be omitted or combined as appropriate.
In addition to being configured as the above-mentioned online conference system, the present invention can also be configured as an online conference method for implementing an online conference by a computer. The present invention can also be configured as a computer program for causing a computer to implement the functions for implementing an online conference, and a computer-readable recording medium on which such a computer program is recorded.

FIG. 1 is an explanatory diagram showing a configuration of an online conference system. FIG. 2 is an explanatory diagram showing a database structure. 13 is a flowchart of a distribution control process. 13 is a flowchart of a distribution data management process. FIG. 1 is an explanatory diagram showing the concept of distribution control. FIG. 11 is an explanatory diagram showing an example of a direct talk screen in the online conference system. 13 is a flowchart of a participant screen generating process. FIG. 1 is an explanatory diagram showing an example of a screen in an online conference system. 13 is a flowchart of a meeting review process. 13 is a flowchart of a keyword extraction process. FIG. 11 is an explanatory diagram showing a display example of a Gantt chart. FIG. 13 is an explanatory diagram showing a display example of an analysis report.

An online conference system according to an embodiment of the present invention will be described in the following order.
A. System Configuration:
B. Direct Talk Function:
C. Meeting Facilitation Function:
D. Meeting Review Function:
E. Effects:

A. System Configuration:
FIG. 1 is an explanatory diagram showing the configuration of an online conference system. The online conference system 100 of the embodiment is configured by a server connected to the Internet, and can realize an online conference, an online social gathering (hereinafter, both of which are referred to as an online conference) and the like by distributing images and audio between multiple terminals 10 also connected to the Internet. As the terminals 10, tablets, smartphones, mobile phones, personal computers, notebook computers, and other devices that have a function of connecting to the Internet can be used. Any number of terminals can participate in an online conference.

The terminal 10 is a computer having an internal CPU and memory, and the functions shown in the figure are implemented as software by computer programs, although some or all of the functions shown in the figure may be implemented as hardware.
The transmitting/receiving unit 11 has a function of transmitting and receiving information via the Internet. In this embodiment, the information includes image data, audio data, and the like in an online conference.
The voice input unit 12 utilizes a microphone or the like provided in the terminal 10 to input the voice of the user, that is, the participant of the online conference.
The image capturing unit 13 utilizes a camera or the like provided in the terminal 10 to input images of the users, that is, the participants in the online conference.
The command input unit 16 inputs various commands from the touch panel, keyboard, etc. of the terminal 10. The commands include the display mode of the image of the online conference, volume adjustment, etc., as well as commands related to functions unique to this embodiment.
The display control unit 17 controls screen display on the display of the terminal 10. In this embodiment, images and the like received from other terminals 10 in an online conference are displayed on the display.
The audio output control unit 18 controls audio output from the terminal 10. In this embodiment, audio and the like received from other terminals 10 in an online conference are output.

The terminal 10 is provided with a browser 14, and the browser 14 executes the screen and other data for the online conference generated by the online conference system 100, thereby providing various functions for the online conference.
The browser 14 incorporates a volume adjustment unit 15. The volume adjustment unit 15 provides a function of individually adjusting the volume of the sound received from the other terminals 10 during an online conference, using an interface displayed on the browser 14. That is, when a user operates the interface for volume adjustment displayed on the browser 14, the volume adjustment unit 15 performs a function of outputting a volume adjustment instruction to the audio output control unit 18 based on the result of the operation.
Instead of the browser 14 and the volume control unit 15, a dedicated application for online conferences may be installed on the terminal 10.

The online conference system 100 is configured with a server equipped with a CPU, memory, etc., and each of the functions shown in the figure is constructed as software by a computer program. Some or all of the functions in the figure may be constructed as hardware. In addition, each function of the online conference system 100 is not limited to being provided by a single server, but may be provided by a distributed system using multiple servers.
Each function will be explained below.

The online conference system 100 is equipped with three databases. The structure of each database will be described in detail later.
The user database 170 records information such as the names of users, etc. It also records the participation history of online conferences.
In the conference database 171, a record is created for each online conference, and information on the participants and other information is recorded.
The image database 172 records image data transmitted from each terminal 10 during an online conference. In this embodiment, a format in which image data and audio data are integrated is used, so audio is also recorded in the image database 172. The image data and audio data may be configured as separate files, in which case the image data and audio data are recorded in association with each other in the image database 172.

The transmitting/receiving unit 110 has a function of transmitting and receiving information via the Internet. In this embodiment, the information includes image data, audio data, and the like in an online conference.
The image input unit 140 inputs image data from the terminal 10. In this embodiment, audio data is also input.

The distribution control unit 120 distributes data related to the online conference, such as image data and audio data, between the terminals 10 participating in the online conference. The distribution control unit 120 performs distribution control in two senses. In the embodiment, multiple online conferences may be held in parallel, and multiple terminals 10 participating in each conference are connected. As a first control process, the distribution control unit 120 distinguishes between these online conferences and controls distribution only between the terminals 10 participating in each online conference. The second control process is distribution control within each online conference. That is, basically, data is distributed so that all the multiple terminals 10 participating in the online conference can send and receive data to each other, but in a function characteristic of this embodiment described later, distribution between some terminals is prohibited. Such control is executed as distribution control within the online conference.

The stamp processing unit 150 has a function of transmitting and receiving stamps between the terminals 10. A stamp is a figure or the like for expressing opinions and impressions such as for or against comments made by other participants during an online conference. A stamp may also include a text message.
The screen generation unit 130 performs the function of displaying a screen on the terminal 10. Specifically, it generates data in a format such as HTML for displaying a screen on each terminal 10. This data is transmitted to the terminal 10 and displayed on the browser 14.

The analysis processing unit 160 performs a function of analyzing the remarks and reactions of participants in an online conference. The analysis may be performed in real time during the online conference, or may be reviewed after the online conference ends. Examples of the analysis content include the remark time and the remark ratio of each participant.
The analysis processing unit 160 also includes a sentiment analysis unit 161 and a keyword extraction unit 162 .
The emotion analysis unit 161 analyzes reactions to comments based on the facial expressions of the online conference participants obtained from the terminal 10. Various well-known techniques, such as the use of artificial intelligence, can be applied to the analysis of emotions based on facial expressions. In this sense, the emotion analysis unit 161 may be realized, for example, by using a service provided by the cloud.
The keyword extraction unit 162 analyzes comments made in online conferences to extract keywords. Various well-known techniques, such as the use of artificial intelligence, can be applied to the extraction of keywords. Therefore, like the sentiment analysis unit 161, the keyword extraction unit 162 may also be realized by using a service provided by the cloud, for example.

The functions of the online conference system 100 and the terminal 10 shown in FIG. 1 are merely examples, and various other configurations are possible. Some of the functions shown in the figure may be omitted or integrated. In addition, functions other than those shown in the figure may be provided.

FIG. 2 is an explanatory diagram showing a database structure.
The data structure of the user database 170 is shown in the upper right. In the user database 170, a record is created for each user with a user ID as unique identification information. The record records the user's name, email and other contact information, and the history of online conferences in which the user participated. In this embodiment, the history records the conference ID, which is identification information assigned to each conference in the conference database 171.

The data structure of image database 172 is shown in the lower right. In image database 172, image data is recorded with an image ID, which is unique identification information. In addition, the data may be recorded in association with a user ID, date and time, etc.

The left side shows the data structure of the conference database 171. In the conference database 171, a record is created for each online conference, with a conference ID being unique identification information.
Each record contains information such as the “title” of the online conference, the date, the start time, the end time, etc. The user who hosts the online conference can register this information from the terminal 10.

In addition, the user IDs of the user database 170 are recorded as participants in the online conference. Although it is possible for multiple people to use one terminal 10 in an online conference, in this embodiment, only users who have signed in to the online conference system 100 from each terminal 10 are recorded. In the following explanation, to avoid complicating the explanation, it will be explained as if one person is using one terminal 10.

A "group" is information that groups users who participate in an online conference. This information can be registered at the discretion of the participants in the online conference. In the example shown in the figure, two groups, group 1 and group 2, are registered, and the user IDs of the participants in each group are recorded. It is acceptable for one user to be registered in multiple groups. As will be described later, this embodiment is provided with a direct talk function that allows only a portion of the participants in an online conference to talk, and groups can be used to specify conversation partners using this function.

The "speech log" is a record of comments made by each participant during an online conference. The time of the comment, the user ID of the participant who made the comment, and the image data are recorded in association with each other. For image data, the image ID from the image database 172 is recorded. By referring to the speech log, it is possible to play back comments made during an online conference.

The "stamp log" is a record of stamps exchanged between participants during an online conference. The time the stamp was sent, the sender, the recipient, and the contents of the stamp are recorded in association with each other. The user ID of the sender and the recipient is recorded. The contents of the stamp may be recorded as an ID representing the stamp. If text messages can be sent and received, the contents of the text may also be recorded.

The "distribution data log" is a record of the results of distribution control during an online conference. As described below, in this embodiment, when image data is sent and received between multiple terminals 10 participating in an online conference, it is possible to distribute image data not only between all terminals, but also between only some of the terminals. In this way, in this embodiment, distribution data is used as data for controlling which terminals the image data is distributed between. In the distribution data log, this distribution data is recorded in association with the time. In this way, it is possible to reproduce the distribution status of image data at each point in time during an online conference.

The data structure shown in FIG. 2 is merely an example, and various other structures are possible. Some of the data shown in FIG. 2 may be omitted, and other data may be recorded in each database.

B. Direct Talk Function:
A direct talk function implemented in the online conference system of the embodiment will be described. The direct talk function is a limited distribution function that allows conversations to take place only between some participants during an online conference. For example, when direct talk is taking place between participants A and B, the conversation between A and B is not distributed to other participants. However, the conversations of the other participants are not restricted, and the conversations between the other participants can take place as usual.
For example, consider a social gathering using an online conference system. In real social gatherings, conversations are often held in parallel among some of the participants, but in online conference systems, one person's remarks are usually broadcast to everyone, making it difficult for multiple conversations to take place in parallel. In such cases, if the direct talk function is used, conversations can be held only among some of the participants, making it possible to have smoother conversations at the social gathering.
Consider also the case where an online conference system is used for discussion between companies. In such a conference, it may be necessary to hold a discussion within the company only during the conference and then provide a response to the other party. Even if it is not as big as a discussion, there may be a situation where you want to confirm any questions among your colleagues while the conference is in progress without interrupting the progress. In a normal online conference system, such discussions among colleagues cannot be held, but in this embodiment, it is possible to realize this by using the direct talk function.
The control process for implementing the direct talk function will now be described.

Figure 3 is a flowchart of the distribution control process. This process is mainly executed by the distribution control unit 120 (see Figure 1), and in terms of hardware, it is executed by the server that constitutes the online conference system 100. This process is executed repeatedly during the online conference.

When the process starts, the server inputs image data including audio from each terminal 10 (step S10).
Then, the distribution destination is specified based on the distribution data (step S11). An example of distribution data is shown in the figure. Distribution data is data that defines which terminal data received from is to be distributed to which terminal. In the example shown in the figure, each row represents a calling terminal, and each column represents a receiving terminal. In the row of calling terminal A, only receiving terminal D has a "◯" mark, so image data received from terminal A will be distributed only to terminal D. Conversely, image data received from terminal D will be distributed only to terminal A. In other words, a direct talk state is taking place between terminals A and D.
While direct talk is taking place, conversations on other terminals B and C are not restricted. Therefore, image data received from terminal B is distributed to terminal C. Also, image data received from terminal C is distributed to terminals B and D. Since direct talk is taking place between terminals AD, conversations between AD are not distributed to terminals B and C, but terminals A and D can listen to conversations on other terminals depending on the settings. Terminal D is set up so that it can also listen to conversations on terminal C, so image data from terminal C is distributed to terminal D even during direct talk.
In this way, depending on the settings of the distribution data, it is possible to control whether normal conversations can be conducted between all terminals or whether direct talk can be conducted between some terminals.

The server distributes image data to each terminal according to the result of step S11 (step S12), stores the distributed data in the conference database, and updates the contents of the database (step S13).
In the process of the embodiment, the distribution of image data including voice is controlled, so that during direct talk, not only voice but also image distribution is restricted. In other words, the images of other participants are not displayed on the terminals of participants during direct talk.
Alternatively, direct talk may be applied only to voice data. In other words, image data not including voice may be always mutually distributed between all terminals regardless of the distribution data, and only voice data may be distributed based on the distribution data. When image data and voice data are treated as separate data, it is sufficient to control the distribution of each of them by the distribution data. On the other hand, when data is integrated as image data including voice, two methods are possible. One is a method in which voice data and image data are separated and then the distribution of each is controlled individually. The other is a method in which both are distributed as an integrated data, but for terminals for which the distribution of voice is prohibited in the distribution data, a flag prohibiting the reproduction of voice is attached to the image data before distribution. The terminal receives image data including voice, but because the flag is attached, the voice is erased and only the image data is reproduced. By doing this, it is possible to obtain the same effect as if voice was not distributed.

Figure 4 is a flowchart of the distribution data management process. This process determines the content of the distribution data for controlling the distribution destination according to the operation of the terminal 10 of the participant of the online conference. This process is mainly executed by the distribution control unit 120 (see Figure 1), and in terms of hardware, it is executed by the server that constitutes the online conference system 100. This process is executed repeatedly during the online conference.

When the process begins, the server detects whether or not the participant has operated the screen displayed on their terminal 10. If no operation has been performed (step S20), there is no need to change the distribution data, so the server ends the distribution data management process without performing any processing. If an operation has been performed (step S20), the server executes processing according to the content of each operation depending on the content of the operation.

If the operation content is a direct talk request (step S20), that is, if any participant designates another participant and requests to hold a direct talk, the following process is performed.
The server identifies the request source and the request destination based on the operation content. In this embodiment, two types of request methods can be used: a designated request and a group request.
A designated request is a request made by individually designating another participant. In this case, the request destination is the terminal 10 used by the designated participant.
A group request is a request made by specifying a group to which a participant belongs. Participants in an online conference can arbitrarily form groups and register them in advance (see "Group" in the conference database 171 in FIG. 2). In a group request, one of these groups is specified to make a direct talk request. For example, in a conference between companies, if participants from the same company are grouped, it becomes possible to easily send a direct talk request to participants in the group without having to specify them individually.

Next, the server adds the other party in the direct talk to the list of request destinations (step S22). For example, consider the case where participant C issues a direct talk request to participant A while participants A and B are in direct talk. In this case, participant C must also get approval from participant B in order to participate in the direct talk. From this perspective, if the direct talk request destination (participant A in the above example) is already in direct talk, the server also adds that other party (participant B in the above example) to the list of request destinations.

The server manages the results of direct talk transmission and approval as request data, so the server updates the request data by adding the request to be newly transmitted based on the above process (step S23).
Then, an approval request button is displayed on the screen of each terminal to which the request is to be sent (step S24).

If the operation is to approve a request or adjust the volume (step S20), the following process is performed. Approval of a request means that a participant at a terminal displaying the "approval request button" described above in step S24 approves the request by clicking the button, etc. If the request is approved, the server needs to change the distribution data to establish direct talk.
Moreover, adjusting the volume refers to an operation in which a participant individually adjusts the volume of the sound from other participants. When a direct talk is set, the voice of the other participants is not distributed for a time. For example, when a direct talk is started between participants A and B, the voice of the other participants is set not to be distributed to participant A and B for a time. However, in this embodiment, it is permitted to listen to the conversation of other participants even during a direct talk. Therefore, when participant A or participant B adjusts the volume during a direct talk so that the conversation of participant C can be heard, the server needs to change the distribution data so that the voice of participant C is distributed.
The process for changing the distribution data in this way will be described below.

When direct talk is set, the voices of the other participants may continue to be distributed rather than being erased. The participant who started direct talk may, if necessary, lower the volume of the other participants on their terminal to mute them. In such a case, there is no need to change the distribution data in response to the volume adjustment, so the following process may be omitted.

When processing begins, the server updates the request data according to the operation (step S30). That is, if the direct talk is approved, the server searches for the corresponding request record in the request data and rewrites its status to "approved." When adjusting the volume, there is no need to update the request data, since this is unrelated to the approval of the request.

If the request has not yet been approved by all participants (step S31), direct talk should not be established between only the few participants for whom approval has been obtained, so the server ends the distribution data management process without performing any special processing. This process may be performed separately for the following cases. For example, if a request is sent to multiple participants, such as a group request, when direct talk has not yet been established, it is considered acceptable to establish partial direct talk once approval has been obtained from some of the participants. On the other hand, if a request is made by a new participant while direct talk is already in progress, it can be said that approval from all participants in the existing direct talk is required. Therefore, step S31 may be processed separately for cases where direct talk has already been established and cases where it has not.

When the request has been approved by all the participants (step S31), the server updates the distribution data so that the approved direct talk is established (step S32). When adjusting the volume, the same process is carried out assuming that all the participants have approved. In this way, audio is distributed according to the operation.

If the operation is to approve the cancellation or clearing of a direct talk (step S20), the following process is performed. Cancellation means ending an individual direct talk, and clear means ending all direct talks in which participant A is involved. For example, when participant A is in direct talk with participants B and C, participant A can "cancel" the direct talk with participant B or participant C individually, or can "clear" the direct talk with participants B and C to end them all at once.

When the server starts the process, it first updates the request data in response to the release or clear operation (step S40). In this embodiment, the request data corresponding to the released or cleared direct talk is deleted.
Next, the server updates the distribution data (step S41). In the case of canceling, the distribution data is updated so that the voice of the terminal related to the specified direct talk is distributed to other terminals as well. In the case of clearing, the same process is performed for all related direct talks. However, this is a process for the voice sent from the terminal that instructed to cancel or clear, and it is not intended that the direct talks between other terminals are also canceled.
This will cancel or clear Direct Talk, allowing you to have conversations including with other devices.

5 is an explanatory diagram showing the concept of distribution control, showing an example of the transition of distribution data and request data.
The distribution data D1 in the upper left shows the normal state, that is, the state in which voice is distributed between all terminals. In this state, suppose that a direct talk request is sent from terminal A to terminal D. The server then generates the corresponding request data R1. In the figure, A → D indicates that the request originates from terminal A and is sent to terminal D.

Next, assume that this request is approved by terminal D. In response to this operation, the server changes the request status to "approved," as shown in request data R2. The "○" in the diagram indicates approval.

Then, in response to the approved request, the distribution data D1 is changed to distribution data D2 so that direct talk can be realized between the terminals AD.
That is, as shown in data D2a, audio from terminal A is distributed only to terminal D. The "x" in the terminal B and terminal C columns indicates that distribution to these terminals is prohibited. Similarly, as shown in data D2d, audio from terminal D is distributed only to terminal A.
Furthermore, when direct talk is established between terminals A and D, the voices of terminals B and C are also temporarily set so as not to be distributed to terminals A and D. Therefore, as shown in data D2b, the state is such that no voice is distributed from terminal B to terminal D.
The distribution from terminal C to terminal D is also initially marked as "X." However, if the volume is adjusted by operating terminal D so that the voice from terminal C can be heard, the distribution data is set to "O," as shown in data D2c.

In this state, suppose that a direct talk request is sent from terminal B to terminal A. The server then generates request data R3 from B to A. Also, since direct talk has already been established between terminals AD, the server also adds a request from terminal B to the other end, terminal D.

If this request is approved by terminal A, the server sets the request from terminal B to terminal A to an "approved" state, as shown in request data R4. However, if approval has not been obtained from terminal D, approval has not been obtained from everyone (see step S31 in Figure 4), so distribution data D2 is not updated.

In this state, suppose that the direct talk between terminals A and D is released. Then, a request from terminal B to terminal A can be accepted, so the server changes the distribution data D2 to distribution data D3 accordingly. That is, direct talk is established between terminals A and B. Also, when direct talk between terminals A and D is established, the voices of terminals C and D are no longer distributed to terminals A and B, so as a result, the same state as direct talk is established between terminals C and C.
In this state, if the direct talk between terminals A and B is released, direct talk will no longer be established between any of the terminals, so that the distribution data D3 will be changed to the state of distribution data D1, and distribution will be performed between all terminals.

Figure 6 is an explanatory diagram showing an example of a direct talk screen in an online conference system. Figure 6(a) shows the state before direct talk is established, and Figure 6(b) shows the screen during direct talk.

As shown in Fig. 6(a), in this example, the participant's own image is displayed in the upper left, and the other areas display the images of participants A to C. Buttons 51 for specifying a direct talk request on a group basis are displayed on the participant's own image. A request to group 1 or group 2 can be made by clicking on either button.
In addition, a button 52 labeled "Direct Talk" is displayed on each image of participants A to C. When selecting an individual participant and requesting direct talk, the participant can click this button. In the example of Fig. 6(a), a request for direct talk has been sent to participant A, so only participant A's button 52 is displayed in inverted black and white.

A volume control 53 for adjusting the volume is displayed on the screen of each of the participants A to C. During an online conference, the volume of each participant can be adjusted individually by sliding the volume control 53. In the example shown in the figure, the volume of participant A is set to high, the volume of participant C is set to low, and participant B is muted.
In this embodiment, the terminal 10 receives image data including the voice of each participant and plays it to play the voice. Since the image data of the participants is distributed as individual data, the volume can also be adjusted individually. That is, the terminal 10 performs processing to play multiple voices in parallel. In the online conference system 100 of the embodiment, the volume instruction corresponding to the operation of the volume 53 displayed on the screen is configured to instruct the playback function of the terminal 10, thereby realizing individual volume adjustment.
The method of adjusting the volume is not limited to a method that depends on the function of the terminal 10, and may be realized on the server side. In this case, the server receives the adjustment state of the volume 53 for each participant from the terminal, and distributes the voices of each participant in a synthesized state at a volume corresponding to the received volume to the terminal.

When direct talk is established, a link 56 is displayed connecting the participants who are in direct talk with each other, as shown in Fig. 6(b). In the example shown in the figure, it is shown that the participant himself/herself and participant A are in direct talk. This link 56 is also displayed on the screen of participant A's terminal. It is not displayed on the screens displayed on the terminals of participant B and participant C. This makes it possible to prevent participants B and C from realizing that the participant himself/herself is in direct talk with participant A.
As a modified example, the direct talk link 56 may be displayed on the terminals of the unrelated participants B and C, and it may be possible to select whether or not to display the link.

A release button 55 is displayed at one end of the link 56. By clicking this, an established direct talk can be released. When multiple direct talks have been established, such as when the user is in direct talk with participants A and C, the link 56 and the release button 55 are displayed between participants A and C, respectively, so that the direct talks can be released individually using the release button 55.
Also, a "clear" button 54 is displayed on the user's screen. By operating the button 54, the user can clear all direct talks in which the user is involved.

When a direct talk request is made to oneself, the "Request" button 57 flashes, as shown on the screen of participant B in the figure. By clicking button 57, the request can be "approved." In the embodiment, no operation for rejecting a request is provided, but it may be possible to reject the request.

By the above-described processing and screen display, the online conference system 100 of this embodiment can realize direct talk among some of the participants.
The above-described processing and screen display are merely examples, and various modified examples can be configured.

C. Meeting Facilitation Function:
Next, the conference promotion function will be described. In this embodiment, the conference promotion function is provided to efficiently and effectively advance an online conference. The conference promotion function includes a function to encourage participants to speak by displaying the level of speech of each participant; a function to encourage speaking by displaying the reactions of other participants while speaking, and a function to provide an opportunity to reconsider thoughts. These functions will be described below.

Figure 7 is a flowchart of the participant screen generation process. This process is mainly executed by the screen generation unit 130, image input unit 140, stamp processing unit 150, and analysis processing unit 160 (see Figure 1), and in terms of hardware, it is executed by the server that constitutes the online conference system 100. This process is executed repeatedly while the online conference is being held.

When the process starts, the server inputs image data including voice from each terminal (step S50). Then, the server identifies the speaker in the online conference by determining from which terminal the image data including voice that can be considered a speech was sent (step S51).

In addition, stamp information from other participants to the speaker is input (step S52). As stamp information, marks expressing opinions and impressions such as approval, surprise, and opposition to the statement can be used. In the online conference system 100, these stamps are prepared in advance, and participants can select some of them to easily transmit their own opinions and impressions on the statement without interrupting the statement.
The stamp information may include text data from each participant, making it possible to convey specific opinions and impressions that cannot be expressed by stamps.
Stamp information may be sent to anyone, not just to the speaker. However, if stamps are sent in this manner, the purpose of each stamp may become unclear, and miscellaneous information may be exchanged during the online conference, which may hinder the progress of the conference. Therefore, in this embodiment, the speaker is identified in step S51, and only the stamp information sent to the speaker is effectively treated as being sent.

The server then performs emotion analysis processing on participants other than the speaker (step S53). In this embodiment, emotion analysis is performed based on the facial expressions of each participant. This processing can be performed using images of the participants received from each terminal, for example, by utilizing artificial intelligence. Since there are already well-known technologies for emotion analysis, these may be used.

As for the emotion analysis, for example, it is possible to evaluate whether or not the user's gaze has left the screen, and judge the favorability of the user's comment depending on the number and frequency of such actions.
It may also be possible to determine whether the user is dozing off based on the degree to which the user's eyes are open, the shaking of the head, etc. In this case, the comment may be treated as an indication that the comment is boring, and feedback may be given to the commenter, or a warning may be issued to the user to wake up.
Furthermore, these emotion analyses can be performed on all participants in an online conference. For example, if there is an increase in the percentage of participants who are judged to be lacking concentration, such as dozing off, a message may be displayed encouraging the organizer or all participants to take a break.

The server also calculates the speech time and silent time of each participant (step S54). The speech time is the length of a speech. The silent time is the time during which no speech is made, i.e., the time elapsed since the last speech.
The figure shows a schematic diagram of how to calculate the speech time and silent time. The hatched parts in the figure represent speech. After participant A makes speech a1, participant B makes speech b1, and then speaker A makes speech a2 again. In step S54, in order to show the current situation, the length of the current speech a2 becomes the speech time for participant A. As long as the speech continues, the speech time will gradually increase. If the speech time is displayed, participant A can recognize that he/she has been talking too long during the online conference, and can improve the efficiency of the conference by trying to organize his/her speech compactly.
For participant B, the time that has elapsed since finishing speech b1 becomes the silent time. If no speech is made, the silent time will gradually increase. By displaying the silent time, participant B can be encouraged to speak.

The server also calculates the speech rate of each participant (step S55). The speech rate is a ratio that indicates the percentage of the time each participant speaks in the online conference. In this embodiment, the speech rate is calculated as "cumulative time during which each participant speaks/time of the online conference". The time of the online conference includes time when everyone is silent. Alternatively, it may be "cumulative time during which each participant speaks/total time of speech by all participants". In this way, it is possible to grasp the proportion of each participant based only on the time when speech is being made. Also, since simple replies are included in speech, the speech rate may be calculated based only on speech that exceeds a certain length that is deemed to be a coherent opinion.

The server generates image data representing the results of the above processing, and transmits it to each terminal (step S56).
In this example, some of the processes performed in steps S52 to S55 may be omitted.

8 is an explanatory diagram showing an example of a screen in an online conference system. An image of the participant himself is displayed in the upper left corner, and images of participants A to C are displayed in the other areas.
An indicator 61 showing speaking time and silent time is displayed on the screen of each participant, including the participant himself/herself. When the participant is speaking, the speaking time gradually increases as shown in the figure. Conversely, when the participant is not speaking, the silent time increases as shown by the dashed line. The example in the figure shows a state in which the participant himself/herself is speaking, so the silent time is displayed for each of participants A to C.

A chart 62 showing the speech rate of each participant is displayed in the bottom right of the participant's screen. The hatched portion of the chart 62 represents the participant's own speech rate. The speech rate is displayed in common to all participants, so it is not displayed on the screens of other participants.

Stamps 60 are displayed on the left side of the participant's screen. A heart mark is a stamp that indicates approval of a comment, an ! stamp that indicates surprise, and an NG stamp that indicates disapproval. These are stamps sent to the participant by participants A to C while the participant was making a comment. Displaying stamps in this way has the advantage of allowing the participant to know the reaction to the participant's comment. The types of stamps are not limited to these. Also, text sent by other participants may be displayed in any area of the screen.
Conversely, a participant can also send stamps to other participants. Various types of interfaces for sending stamps are conceivable. For example, each stamp may be displayed on the screen of each participant, and the participant may click on the stamp to send it. Also, a stamp may be displayed outside the screen, and the participant may drag the stamp onto the screen to send it.

The results of the emotion analysis are displayed on the screen of each participant. In this embodiment, the emotion analysis determines whether the participant agrees with the statement, disagrees, is neutral, or does not know, and the results are displayed as facial expression marks 63. Participant A's screen displays mark 63 indicating that he or she agrees with the statement. Participant C's mark indicates that he or she is neutral. The marks may be displayed anywhere on the screen, or may be displayed superimposed on participant B's face, as in mark 65.

According to the online conference system of the embodiment, by performing the display shown in FIG. 8, it is possible to encourage each person to speak. Furthermore, in a real conference, it is difficult to send opinions and impressions to a speaker while he or she is speaking without interrupting the speaker, but in this embodiment, this can be achieved by effectively using stamps. Therefore, it is possible to liven up the discussion even more than in a real conference. Furthermore, there is also the advantage that the true feelings of each participant can be guessed by using sentiment analysis.
The screen display in FIG. 8 is merely an example, and some of it may be omitted, or various other displays may be made.

D. Meeting Review Function:
Next, the conference review function will be described. The conference review function is a function that analyzes and displays the state of each participant's remarks in a conference after the online conference has ended. By utilizing this function, each participant can evaluate whether they were passive in the conference, whether they have a habit of speaking too long, etc. Such evaluations can improve the quality of future online conferences. The processing and display screen of the conference review function will be described below.

Figure 9 is a flowchart of the conference review process. This process is used to analyze various pieces of information after an online conference has ended. This process is primarily performed by the screen generation unit 130 and the analysis processing unit 160 (see Figure 1), and in terms of hardware, is executed by the server that constitutes the online conference system 100. This process is started when any user starts it.

When the process starts, the server first inputs the user ID of the user who started the process (step S60). Then, a list of conferences that the user has participated in is displayed, and the server accepts the selection of the conference to be analyzed (step S61). As described in FIG. 1, the user database 170 records the participation history of online conferences, so the history corresponding to the user ID input in step S60 can be referenced.
The figure shows an example of a meeting list. Information to help select the meeting to review, such as the meeting date and time, title, and participants, is displayed. The user can select one of the meetings from the list by clicking it.

When the target conference is selected, the server reads the conference data (step S62). The conference data is recorded in the conference database 171.
Then, a keyword extraction process is executed (step S63). The details of this process will be described later, but the speech of each participant is subjected to speech recognition and keywords contained therein are extracted. The keywords are extracted in association with the speech.

The server then executes an analysis data generation process (step S64). In this embodiment, the server generates the following four types of data:
First, the distribution of speech duration and frequency is tallied as speech records. This may be done for all participants, or only for the participant who is executing this process. This talliation makes it possible to know how long a participant usually spends speaking, and to know habits such as whether or not one tends to speak for longer periods of time. If the tallies are done for all participants, one can know more objectively the tendency of one's own speaking duration by comparing with other participants.
Secondly, as a reaction analysis, the "number of stamps/number of comments" is calculated for each type of stamp. This allows you to know the trends in other participants' opinions and impressions of your comments, such as approval, surprise, and disapproval. In addition to this, reaction analysis can also perform various analyses to evaluate your own comments, such as the number of times you received the most "approval" or "disapproval" stamps.
Third, the speech ratio of each participant. This is calculated as the final performance after the end of the conference by the same process as that described above in step S55 of Fig. 7. By calculating the speech ratio, it is possible to evaluate how actively a participant participated in the online conference.
Fourth, there is a comprehensive evaluation. The comprehensive evaluation can be obtained in various ways. For example, the first to third analysis results may be scored according to their respective criteria, and the scores may be calculated by adding them up. The comprehensive evaluation may also be divided into elements such as proactiveness and speech length, and the first to third analysis results may be scored for each element. In addition, the comprehensive evaluation can be obtained in various ways. The comprehensive evaluation does not necessarily have to be a theoretically "legitimate" evaluation. If an objective comprehensive evaluation defined by a predetermined criterion is displayed for each online conference, it may motivate users to improve the quality of the conference.

The server generates image data representing the above analysis results and outputs it to the terminal (step S65). In this embodiment, two types of screens are prepared: a Gantt chart and an analysis report. Of course, the output screens are not limited to these.
These screens can be switched if the user instructs to switch the display (step S66). The above is the content of the conference review process.

10 is a flowchart of the keyword extraction process, which corresponds to step S63 of the meeting review process (FIG. 9).
When the process starts, the server inputs voice data (step S70). When image data including voice is used, only the voice portion can be extracted from the image data.
Then, speech recognition processing is performed based on the speech data (step S71). Well-known speech recognition technology is already available, so this technology may be utilized.
Next, the text data obtained by the speech recognition is subjected to morphological analysis to extract words (step S72). A well-known technique can be applied to the morphological analysis. In this way, candidate words to be keywords are extracted.

The server then removes non-candidate words (step S73). Non-candidate words are words that appear frequently during online conferences but cannot become keywords. Examples of non-candidate words include pronouns, demonstrative pronouns, and words that indicate dates and times, such as today, yesterday, and tomorrow. Non-candidate words can be registered in a database. In addition, the database of non-candidate words can be updated by having an operator check the results of keyword extraction.

The server ranks the extracted words (step S74). In this embodiment, the ranking is based on criteria such as frequency of use, proper nouns, number of searches on the Internet, and keyword databases.
The frequency of use is the number of times it is used during an online meeting. Since a keyword is considered to be more important when used by multiple people than when used by one person, it may be evaluated by the number of people using it, rather than simply the number of times it is used.
Proper nouns are highly ranked because they are often included in the themes of online meetings, such as project names and event names, and because proper nouns are easy to recall and identify the contents of a conversation. On the other hand, proper nouns, such as people's names and place names, may not be suitable as keywords, so for example, rankings may be assigned only when multiple people use a common proper noun.
Internet search counts are keywords that frequently appear in various Internet search engines. These can be obtained by accessing the search engine sites, etc.
The keyword database is a database of keywords that the user has registered in advance, making it possible to extract even unusual words without fail as keywords.
The extraction of keywords may be performed using other criteria than those described above. Furthermore, multiple criteria may be applied with a priority order.

Once the words have been ranked in this way, the server identifies and outputs a predetermined number of the top ranking words as keywords (step S75). These keywords are used for displaying information on the screen during the conference review process (Figure 9). The extracted keywords may be recorded in the conference database 171 in association with comments.

Next, we will explain examples of display screens in the meeting review process. In this example, two screens are provided: a Gantt chart and an analysis report. We will explain each in turn below.

11 is an explanatory diagram showing an example of a Gantt chart display. In the Gantt chart, the horizontal axis represents time, and comments from each participant in an online conference are displayed as a bar-shaped graph. In the example shown, comments from four participants, Participant A to Participant D, are displayed. Each square comment frame 70 indicates that a comment has been made.
Each comment frame 70 may be linked to audio data. For example, when a comment frame 70 is clicked, the comment is played back, or audio at a corresponding time is played back depending on the position of the comment frame 70 clicked. Image data may be made playable together with audio data.

During a conference, a participant may speak while another participant is speaking. In the example shown in the figure, participant B begins speaking while participant A, indicated by speech frame 70, is speaking. In such a case, in this embodiment, when the later participant begins speaking, the portion that overlaps with the other participant's speech is highlighted 72, as shown by the hatched portion in the figure. The other hatched portions similarly indicate portions where a speech was made that overlaps with another participant's speech. Since speaking over the speech of another participant is not a very desirable behavior during a conference, highlighting in this way has the effect of reminding the participant to refrain from making such a statement next time.

Each comment is accompanied by a stamp 71 sent during the comment. This allows participants to visually see how other participants responded to each comment, making it easy to identify comments that are considered important during online meetings.

In addition, below the area displaying each participant's comment, extracted keywords corresponding to each comment are displayed. This makes it easy to recall the theme on which each comment was based.

FIG. 12 is an explanatory diagram showing an example of a display of an analysis report.
The upper left corner displays the user's name, as well as information such as the date, time, topic, and participants of the meeting being analyzed.
The bottom left corner shows the speech record. In this embodiment, the horizontal axis shows the speech length, and the number of speeches corresponding to each speech length is shown as a bar graph. The speech time can be quantized in units of one minute, for example, and then tallied.
The speech history also shows the average length of your own speech. It may also be possible to show the overall average including other participants. By doing this, you can know whether your speech tends to be long or short, and you can improve the quality of your speech in the future.
Additionally, the effective speech rate was also shown for speech performance. Some speeches were simply replies or nods, and were not coherent opinions. Therefore, among all of a person's speeches, speeches that lasted a certain amount of time or more were considered effective, and the percentage of such speeches was calculated as the effective speech rate. The standard time for determining whether a speech was effective or not can be determined at will.

The bottom center shows the reaction analysis results. It shows the ratio of other participants' reactions to one's own comments. In the example, the figure shows that the heart mark, or stamp of approval, was obtained just under half of the time.
Reaction analysis can be performed in various ways. For example, if two out of four participants agree with one of the person's comments, one disagrees, and one has no reaction, then 50% agree, 25% disagree, and 25% have no reaction. After performing such an analysis for each comment, the average for all comments can be calculated.
Alternatively, the proportion of each opinion can be calculated by treating all stamps, including those for and against, as 100%. In this case, the proportion of "no reaction" stamps will not appear in the analysis results.

The bottom right shows the speaking percentage of each participant in the online conference. The hatched areas in the figure represent each participant's own speaking percentage. The calculation method is as explained in step S55 of the participant screen generation process (Figure 7).

The top right corner shows the overall evaluation of the online conference. First, a radar chart is displayed divided into the categories of proactiveness, speech length, effectiveness, persuasiveness, and cooperativeness. Proactiveness can be scored, for example, by multiplying the number of comments, effective speech rate, and speech ratio by a specified coefficient. Similarly, for other categories, the analysis results of the online conference can be scored by defining an arithmetic formula appropriate to the item. The scores obtained in this way can be plotted to create a radar chart.
In addition, the score is shown in the upper right corner as an overall evaluation. This score can also be calculated in various ways. For example, the area of the radar chart when each item is fully scored is set to 100 points, and the score can be determined based on the area of the radar chart that is actually obtained.

The analysis report may display not only the results of a particular online conference of a user, but also a comparison with the results of past online conferences of the user and a comparison with other participants.
It is also possible to compile comments from everyone who participated in an online meeting and display them as a consolidated analysis.

In this embodiment, the conference review function described above allows the user to review the remarks of each participant, their participation attitude, and their speaking habits after the online conference has been held. By taking such a review into account when attending the next online conference, it is possible to improve the efficiency of the conference.
In addition, in this embodiment, as shown in Figure 11, each comment is displayed in association with the reactions and keywords of other participants, making it easy to recall the content of comments made during an online conference and to easily find important comments.
In the embodiment, the conference review process is described as being performed after the online conference has ended, but this process may be performed at any time during the conference, making it possible to improve the quality of the conference while it is ongoing.

E. Effects:
The above describes an embodiment of the present invention. In the online conference system of the embodiment, the usefulness and convenience of online conferences can be further improved by functions such as a direct talk function, a conference promotion function, and a conference review function. In addition, it is possible to provide functions that cannot be realized in real conferences, and to improve the quality of the conference itself.
It is not necessary to provide all of the various features described in the embodiments, and some of them may be omitted or combined as appropriate. In addition to the embodiments, the present invention can also be constructed in various modified forms.

The present invention can be applied to an online conference system that is connected to multiple terminals via a network and enables online conferences between the terminals.

10 Terminal 11 Transmitting/receiving unit 12 Audio input unit 13 Image capturing unit 14 Browser 15 Volume adjustment unit 16 Command input unit 17 Display control unit 18 Audio output control unit 51 Button 52 Button 53 Volume 54 Button 55 Cancel button 56 Link 57 Button 60 Stamp 61 Indicator 62 Chart 63 Mark 65 Mark 70 Speech frame 71 Stamp 72 Highlighting 100 Online conference system 110 Transmitting/receiving unit 120 Distribution control unit 130 Screen generation unit 140 Image input unit 150 Stamp processing unit 160 Analysis processing unit 161 Sentiment analysis unit 162 Keyword extraction unit 170 User database 171 Conference database 172 Image database

Claims (2)

An online conference system that is connected to a plurality of terminals via a network and realizes an online conference between the terminals,
a distribution control unit that distributes audio data and image data between the plurality of terminals;
a screen generating unit that generates , for each of the plurality of terminals, a screen on which an instruction icon for designating the terminal as a specific terminal is superimposed on image data received from the terminal ;
The distribution control unit is
When a limited conversation instruction is received from any one of the terminals during the online conference to have a conversation only with any other terminal designated by the instruction icon , the designated terminal and the one terminal are designated as specific terminals, and a limited distribution is performed that prohibits the distribution of audio data exchanged between the specific terminals in a playable state to terminals other than the specific terminal ;
The screen generation unit, when the limited distribution is being performed, generates a screen displayed on the one terminal, which displays a link display that links images from the specific terminal by connecting them with a line or surrounding them with a frame, and a cancellation icon that allows an instruction to cancel the limited distribution, superimposed on the link display . The online conference system according to claim 1 ,
The distribution control unit distributes the link display and the release icon only to terminals that are performing the limited distribution.

Images