JP7501575B2 - Videoconferencing system, videoconferencing method, and program - Google Patents

Description

The present invention relates to a video conferencing system, a video conferencing method, and a program.

One method for holding a conference with people in different locations is to use a video conferencing system. With a video conferencing system, people in different locations can hold a conference by exchanging video footage taken at each location.

Technologies related to the above-mentioned video conferencing system are disclosed, for example, in the following Patent Documents 1 to 3.

The following Patent Document 1 discloses a technology for easily identifying participants in two-way communication when two-way communication is performed between two or more points. Specifically, the technology discloses (1) authenticating participants at each point using identification information stored in a database, (2) detecting the position of the authenticated participant, and (3) adding data that visually displays the attribute information of the participant stored in the database to the portion of the video captured at each point that corresponds to the detected position of the participant.

In addition, the following Patent Document 2 discloses a technology that makes it possible to know information about meeting participants even if the participants are new to the person. Specifically, the technology discloses a technology that (1) accepts input of participant information about meeting participants, (2) has each participant perform a predetermined action (such as responding verbally or raising their hand), (3) identifies the person who performed the action on a captured image, (4) associates the face recognition results of the identified person with participant information, and (5) superimposes the participant information on a captured image in which each participant appears according to the association.

In addition, the following Patent Document 3 discloses a technology that automatically adjusts the range of the image displayed to the other party to include the image of a person who has been authenticated as a conference participant.

JP 2004-129071 A JP 2010-028715 A JP 2015-177418 A

In meetings, including those held using videoconferencing systems, it should be clear who is participating, and it is not advisable to leave individuals unidentified.

The present invention has been made in consideration of the above problems. One of the objectives of the present invention is to provide a technology that prevents an unidentified person from remaining at a conference in a video conference system.

The video conference system of the present invention comprises:
An image acquisition means for acquiring an image showing participants of a conference;
a person area detection means for detecting an area recognized as a person from within the image;
A person identification means for executing a person identification process for identifying a person included in the area;
a display control means for causing a display device to display first information indicating that the person has not been identified in the person identification process;
Equipped with.

The video conferencing method of the present invention comprises:
The computer
Acquire images of the people attending the meeting,
Detecting an area recognized as a person from within the image;
execute a person identification process to identify a person included in the area;
displaying, on a display device, first information indicating that the person could not be identified in the person identification process;
This includes:

The program of the present invention causes a computer to execute the above-mentioned video conferencing method.

According to the present invention, it is possible to prevent a person who has not been identified as an individual from remaining at the conference in a video conference system.

FIG. 1 is a diagram illustrating an example of the configuration of a video conference system according to a first embodiment. FIG. 1 is a block diagram illustrating a hardware configuration of a video conference system. 4 is a flowchart illustrating a flow of a process executed by the video conference system of the first embodiment. FIG. 11 is a diagram showing an example of first information displayed by a display control unit. 13 is a diagram showing an example of second information displayed by a person area detection unit. FIG. FIG. 11 is a diagram illustrating an example of a display by a display control unit. FIG. 11 is a diagram showing an example of displaying second information in a list format. FIG. 13 is a diagram illustrating an example of the configuration of a video conference system according to a third embodiment. 13 is a flowchart illustrating a flow of a process executed by the video conference system according to the third embodiment. FIG. 13 is a diagram illustrating an example of the configuration of a video conference system according to a fourth embodiment. 13 is a flowchart illustrating a flow of a process executed by a video conference system according to a fourth embodiment.

The following describes an embodiment of the present invention with reference to the drawings. In all drawings, similar components are given similar reference numerals and descriptions are omitted where appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration.

[First embodiment]
[System configuration example]
FIG. 1 is a diagram showing an example of the configuration of a video conference system 1 in the first embodiment. In the video conference system 1 shown in FIG. 1, a server device 10 is connected to a communication terminal 20A and a communication terminal 20B. The communication terminal 20A and the communication terminal 20B are terminals provided at points A and B, which are located at locations distant from each other, respectively. A video conference is held using these terminals. An imaging device 30A and a display device 40A are connected to the communication terminal 20A. Also, an imaging device 30B and a display device 40B are connected to the communication terminal 20B. The imaging devices 30A and 30B are used to capture participants in a conference at each point. An image of the participant at point A generated by the imaging device 30A is displayed on the display device 40B at point B via the server device 10. Also, an image of the participant at point B generated by the imaging device 30B is displayed on the display device 40A at point A via the server device 10. An image of the participant at point A may be displayed on display device 40A at point A so that the participant at point A can check the photographed image. Similarly, an image of the participant at point B may be displayed on display device 40B at point B so that the participant at point B can check the photographed image.

As shown in FIG. 1, the video conference system 1 includes an image acquisition unit 110, a person area detection unit 120, a person identification unit 130, and a display control unit 140. In the example of FIG. 1, these processing units are provided in one server device 10, but the configuration of the video conference system 1 is not limited to the example of FIG. 1. Although not shown, all or part of these processing units may be distributed or overlapped among multiple server devices.

The image acquisition unit 110 acquires images showing the people participating in the conference. In the example of FIG. 1, the image acquisition unit 110 can acquire images showing the people participating in the conference from communication terminals 20A and 20B that are connected via a network.

The person area detection unit 120 detects an area recognized as a person from the image acquired by the image acquisition unit 110. The person area detection unit 120 can detect an area recognized (classified) as a "person" by using a known general object detection algorithm. The person area detection unit 120 may also detect, for example, an area of a moving object as a person area. The person area detection unit 120 can determine a "moving object" based on, for example, the amount of movement of a feature point between multiple images arranged in chronological order. Specifically, the person area detection unit 120 can estimate, as a person area, an area of an object that includes a feature point that has moved by more than a reference value between multiple images arranged in chronological order.

The person identification unit 130 executes a process (person identification process) for identifying a person included in an area recognized as a "person" by the person area detection unit 120. In other words, the person identification unit 130 identifies (authenticates) each participant appearing in an image acquired by the image acquisition unit 110. The person identification unit 130 can identify who is included in each area based on the result of comparing the feature amount extracted from the area detected by the person area detection unit 120 with the feature amount of the participant registered in advance. The feature amount of the participant of the meeting is associated with the information of the participant (such as name and affiliation) and is stored in advance in the storage device of the server device 10. Here, if the person identification unit 130 is unable to identify a person included in an area detected by the person area detection unit 120, it associates information (identification failure information) indicating that the person included in the area could not be identified with the area. Note that "if a person cannot be identified" refers to, for example, a case where there is no person whose score calculated as a result of the comparison is equal to or greater than a reference value.

The display control unit 140 transmits the images generated by the imaging devices 30 provided at each point to the communication terminals 20 provided at other points. The display control unit 140 may also transmit the images generated by the imaging devices 30 provided at each point to the communication terminals 20 at the points where the imaging devices 30 are provided. The communication terminals 20 at each point display the received images on the display devices 40 connected to the communication terminals 20. In the example of FIG. 1, the display control unit 140 transmits the image of point B acquired via the communication terminal 20B to the communication terminal 20A. The display control unit 140 also transmits the image of point A acquired via the communication terminal 20A to the communication terminal 20B. If there is a person who could not be identified by the person identification process of the person identification unit 130, the display control unit 140 displays information (first information) indicating that the person could not be identified by the person identification process on a display device provided at least at one of the points. The display control unit 140 can determine whether there is a person who could not be identified by the person identification process, for example, based on whether there is an area associated with the identification failure information. The first information displayed by the display control unit 140 can be said to be information indicating that the video conference system 1 recognizes the presence of a certain person, but is unable to identify who that person is.

[Hardware configuration example]
Each functional component of the video conference system 1 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further description will be given of the case where each functional component of the video conference system 1 is realized by a combination of hardware and software in the server device 10.

FIG. 2 is a block diagram illustrating an example of the hardware configuration of the video conference system 1. In the example of FIG. 2, the server device 10 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input/output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, input/output interface 1050, and network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processor 1020 and other components to each other is not limited to a bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

Memory 1030 is a main storage device realized by RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the videoconferencing system 1 (such as the image acquisition unit 110, the person area detection unit 120, the person identification unit 130, and the display control unit 140). The processor 1020 loads each of these program modules into the memory 1030 and executes them to realize each function corresponding to the program module.

The input/output interface 1050 is an interface for connecting the server device 10 to various input/output devices. The input/output interface 1050 can be connected to input devices (not shown) such as a keyboard or a mouse, or output devices (not shown) such as a display or a speaker.

The network interface 1060 is an interface for connecting the server device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may be connected to the network wirelessly or by wire. As shown in the figure, the server device 10 is communicably connected to the communication terminals 20A and 20B via the network interface 1060. Each communication terminal 20 is connected to an imaging device 30 for photographing participants in the conference, a display device 40 for displaying images generated by each imaging device 30, and a sound collection device 50 for picking up sounds during the conference. In addition, an audio output device (not shown) for outputting the sound of the conference is further connected to the communication terminal 20.

The image acquisition unit 110 can acquire images of participants in the conference from each communication terminal 20 via the network interface 1060. The display control unit 140 can also transmit images of the other participants to each communication terminal 20 via the network interface 1060. The display control unit 140 can also transmit images of participants at the location where each communication terminal 20 is installed to each communication terminal 20 via the network interface 1060.

[Processing flow]
The flow of processing executed by the video conference system 1 of the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the flow of processing executed by the video conference system 1 of the first embodiment.

The image acquisition unit 110 acquires an image showing the people participating in the conference from the communication terminal 20A or the communication terminal 20B (S102). Then, the person area detection unit 120 detects an area recognized as a person from the image acquired in the process of S102 (S104).

The person identification unit 130 executes person identification processing for the area detected in the processing of S104 (S106). If the person cannot be identified (S108: NO), the person identification unit 130 associates information indicating that the person contained in the area was not identified (identification failure information) with the area that was the subject of the person identification processing (S110). On the other hand, if the person cannot be identified (S108: NO), the processing of associating the above-mentioned identification failure information is not executed. Note that the processing from S106 to S110 is repeated until all areas of the person detected in the processing of S104 have been processed (S112: NO).

After all of the areas of the people detected in the process of S104 have been processed (S112: YES), the display control unit 140 causes the images of the participants taken at each location to be displayed on the display devices provided at the locations where the participants are located (S114). Here, if there is an area to which identification failure information is associated in the process of S108, the display control unit 140 causes, for example, the first information to be superimposed on the position of the person in that area so that it is clear that the person has not been identified (e.g., FIG. 4).

4 is a diagram showing an example of the first information displayed by the display control unit 140. In the example of FIG. 4, the first information "Unknown" is superimposed on two people whose backs are shown in the image. The first information may be any specific mark indicating that the person could not be identified, and is not limited to the display "Unknown". In this figure, the first information "Unknown" is displayed as follows. First, the person area detection unit 120 detects the areas of four people from the image of FIG. 4. Since two of the detected four people are facing away from the viewer (a situation in which features used to identify individuals, such as eyes, nose, and mouth, cannot be extracted), the person identification unit 130 fails to identify these people in the person identification process. The person identification unit 130 associates identification failure information with the areas of these two people based on the result of the person identification process. The display control unit 140 superimposes the first information on the image based on the identification failure information associated with the area of each person, and outputs the information to the display device. A display device that receives this data can display an image, such as the one shown in Figure 4, on its display surface.

In this embodiment, when an unidentified person is present among the participants of a conference held using the video conference system 1, first information is displayed on the display device used in the conference. This first information is information indicating that the person has not been identified (for example, a display such as "Unknown"). In other words, the participants of the conference can at a glance know from this first information that there is a person who has not been identified by the video conference system 1. In addition, the first information is displayed according to the position of the area of the person who has not been identified in the image. This allows the participants of the conference to recognize which person has not been identified. The participants of the conference can then take appropriate measures to enable the video conference system 1 to identify that person. For example, the participants of the conference can urge the unidentified person to change the direction or posture of their face so that their face (the area including the feature points used in identifying the person) is clearly reflected in the image.

Here, the display control unit 140 may display the first information on a display device seen by the person participating in the meeting who is shown in the image. That is, if there is a person who could not be identified by the person identification process, the display control unit 140 may display the first information on a display device provided at the location where the person is located. For example, it is assumed that a person participating in the meeting at point A could not be identified by the person identification process. In this case, the display control unit 140 transmits data in which the first information is superimposed on an image generated by the imaging device 30A to the communication terminal 20A. The communication terminal 20A displays the image generated by the imaging device 30A and the first information superimposed on the image on the display device 40A. In this way, it is possible for the person participating in the meeting at the side of point A to easily grasp that there is a person who has not been identified (not authenticated). As a result, for example, the person who has not been identified can take measures such as voluntarily changing the direction or posture of the face so that his or her face (an area including feature points used in person identification) is clearly reflected in the image.

[Second embodiment]
This embodiment has the same configuration as the above-described first embodiment, except for the following points.

In the present embodiment, when a participant of a meeting is identified by the person identification process of the person identification unit 130, the display control unit 140 further displays information (second information) including the name of the person on the display device (example: FIG. 5). FIG. 5 is a diagram showing an example of the second information displayed by the person area detection unit 120. In the example of FIG. 5, it is assumed that the two people sitting at the back of the image have clear feature points such as eyes, nose, and mouth, and the person identification unit 130 has been able to identify these two people. In this case, the person identification unit 130 acquires second information including the name of the identified person and associates it with the area of the person. The second information is acquired in association with the feature amount of the participant before the meeting is held, and is registered in advance in the storage device 1040 or the like. In addition to the person's name, the second information may further include the name of the group (company, department, etc.) to which the person belongs, the person's job title, etc. Then, based on the result of the person identification process by the person identification unit 130, the display control unit 140 superimposes and displays the second information according to the position of the person in the area to which the second information is associated. As a result, for example, an image as shown in FIG. 5 is displayed on the display device 40 to be viewed by the participants in the conference held using the video conference system 1. This allows information about the participants in the conference, such as their names and affiliations, to be grasped at a glance.

When the display control unit 140 displays the first information and the second information together, the display control unit 140 may make the display mode of the first information different from the display mode of the second information. In other words, the display control unit 140 may visually make the first information different from the second information. As an example, the display control unit 140 may make the first information more prominent than the second information. Specifically, the display control unit 140 can make the first information more prominent than the second information by making at least one of the outer shape, size, color, language, and font of the first information different from that of the second information. For example, the display control unit 140 can make the first information more prominent than the second information by displaying the first information larger than the second information, setting the color of the first information to a prominent color, or setting the font of the first information to a special font different from the standard font. FIG. 6 is a diagram showing an example of a display by the display control unit 140. In the example of FIG. 6, a state in which the first information is made more prominent than the second information by changing the background color of the first display is depicted. This makes it easier for conference participants to notice the presence of people who have not been identified by the video conference system 1.

In addition, the display control unit 140 may display the second information in a list format when the number of people participating in the meeting identified by the person identification process is equal to or greater than a predetermined threshold (e.g., FIG. 7). FIG. 7 is a diagram showing an example of displaying the second information in a list format. As shown in FIG. 7, the display control unit 140 can identify an empty area in the image (an area other than the area of people) and display the second information in a list format in that area. The display control unit 140 may determine the size of the list according to the size of the empty area. In addition, the display control unit 140 may display the list together with a scroll bar to save the area for displaying the list. In this way, it is possible to suppress a decrease in visibility of the faces of people participating in the meeting (the main information of the image) due to a large amount of second information being displayed.

[Modification of the second embodiment]
The display control unit 140 may make the first information less noticeable than the second information. In particular, when transmitting an image of a participant at a certain location to the location of the other party, it is preferable that the display control unit 140 makes the first information less noticeable than the second information. For example, the display control unit 140 can make the first information less noticeable than the second information by making the first information smaller, fading the color of the first information, or hiding the first information in the image to be transmitted to the other party. By making the first information less noticeable when transmitting an image of a participant at a certain location to the location of the other party, it is possible to suppress a decrease in visibility of information that is more important to the other party (information that is originally necessary in the conference, such as a person's face or a person's movement).

[Third embodiment]
The video conferencing system 1 of this embodiment has the same configuration as each of the above-described embodiments, except that it further has a configuration for identifying a person if there is a person who could not be identified by the video conferencing system 1.

〔System configuration〕
Fig. 8 is a diagram showing an example of the configuration of a video conference system 1 according to the third embodiment. In the video conference system 1 shown in Fig. 8, mobile imaging devices 32 (32A, 32B) are further provided at each location in addition to the above-mentioned imaging devices 30 (30A, 30B). The mobile imaging devices 32 are, for example, devices owned by individuals, such as a mobile terminal with a camera function (such as a smartphone or a tablet terminal) or a notebook PC (Personal Computer) with a camera.

In this embodiment, when there is a participant in the meeting who was not identified in the person identification process based on the image acquired via the imaging device 30, the image acquisition unit 110 acquires an additional image showing the participant from the mobile imaging device 32A or imaging device 32B. Then, the image acquisition unit 110 identifies the participant who was not identified based on the additional image acquired by the image acquisition unit 110.

[Processing flow]
The flow of processing executed by the video conference system 1 of the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart illustrating the flow of processing executed by the video conference system 1 of the third embodiment.

First, an additional image of a person who was not identified in the person identification process of S106 in FIG. 3 is generated by the mobile imaging device 32 (S202). The participants in the conference can know whether or not they have been identified as an individual in the video conference system 1 based on the first information displayed on the display device 40. A person who recognizes that he or she has not been identified from the first information displayed on the display device 40 launches a dedicated application installed on, for example, a smartphone or a notebook PC to capture an additional image. The captured additional image is transmitted to the server device 10 via the network and acquired by the image acquisition unit 110 (S204). Here, there may be multiple unidentified persons in the image generated by the imaging device 30. In this case, information indicating which unidentified person the additional image corresponds to is required. Therefore, when multiple unidentified persons are present in the image generated by the imaging device 30, an operation to specify the unidentified person corresponding to the additional image may be further accepted in an application launched on a smartphone or a notebook PC, for example. For example, an application launched on a smartphone or a notebook PC may display the screens exemplified in FIGS. 4 to 6 on the display surface of the smartphone or notebook PC before or after capturing an additional image, and accept an operation to select an unidentified person from the image. In this case, the image acquisition unit 110 can acquire the additional image and information indicating the unidentified person corresponding to the additional image.

The person identification unit 130 executes a person identification process using an additional image generated by a mobile imaging device 32 such as a smartphone or a notebook PC (S206). Specifically, as in the person identification process of S106 in FIG. 3, the person in the additional image can be identified by matching the feature amount extracted from the additional image with the feature amount of the participant registered in advance. Here, if the additional image is unclear and the person cannot be identified, the person identification unit 130 may be configured to output a message to the device that sent the additional image, prompting the person to retake the image. In addition, if the person identification unit 130 has not been able to identify an unidentified person despite retaking the additional image a predetermined number of times, the person identification unit 130 may determine that the unidentified person is an outsider (a person other than a person registered in advance as a participant of the meeting). In this case, the person identification unit 130 may execute a process of notifying the presence of an outsider using the display device 40 or a speaker (not shown). In this way, the participants in the meeting can recognize the presence of an outsider who has infiltrated the meeting.

As described above, according to this embodiment, it is possible to obtain additional images from the mobile imaging device 32 to identify a person who has not been identified by the video conference system 1. By using the mobile imaging device 32, it is possible to obtain images that more clearly show the characteristic points of the person. As a result, it is possible to accurately identify a person who has not been identified by the video conference system 1.

Once the person identification unit 130 has identified a participant in a meeting, it can maintain the state in which the person has been identified even if the person's features cannot be extracted from an image acquired thereafter (for example, if the person looks down and facial features become invisible). For example, the person identification unit 130 can maintain the state in which the person has been identified by tracking the area of the identified person across multiple images based on the position in the image, etc.

[Modification of the third embodiment]
In this embodiment, instead of using the additional image generated by the mobile imaging device 32, one of the participants in the conference may directly input information for identifying the person who could not be identified. As an example, the following operation may be executed. First, the participant in the conference checks the person with whom the first information such as "Unknown" is associated on the display device 40, and then inputs information such as the name of the person through an application launched on a portable terminal (such as a smartphone or a notebook PC). If there are multiple people who have not been identified, an input is further executed to select a target person from among the multiple people. Then, the person identification unit 130 updates the identification failure information associated with the area of the person who could not be identified using the input information. As a result, the person who could not be identified is identified (authenticated) by the video conference system 1. As a result, the display of the first information such as "Unknown" is replaced with the name of the person who was input.

[Fourth embodiment]
This embodiment has the same configuration as each of the above-described embodiments, except that it further has a function of automatically creating minutes.

Fig. 10 is a diagram showing an example of the configuration of a video conference system 1 in the fourth embodiment. The video conference system 1 shown in Fig. 10 further includes a list creation unit 150, a voice acquisition unit 160, a speaker identification unit 170, and a minutes creation unit 180.

The list creation unit 150 creates a list of people identified by the person identification process of the person identification unit 130. The list creation unit 150 operates, for example, as follows. First, when a person is identified by the person identification process of the person identification unit 130, the list creation unit 150 acquires the result from the person identification unit 130. Then, the list creation unit 150 adds the person identification result acquired from the person identification unit 130 to a list stored in the memory 1030 or the like. This makes it possible to automatically generate a list of participants in a conference held using the videoconferencing system 1.

The voice acquisition unit 160 acquires voice data of a conversation during a conference, generated by a microphone (not shown). The speaker identification unit 170 identifies the speaker of the voice data acquired by the voice acquisition unit 160. As an example, the speaker identification unit 170 can identify the speaker of the voice data acquired by the voice acquisition unit 160 by comparing the voice data with voiceprint data of each participant registered in advance in the storage device 1040 before the conference is held. As another example, the speaker identification unit 170 can identify the speaker of the voice data acquired by the voice acquisition unit 160 by analyzing an image (an image generated by the imaging device 30) acquired in synchronization with the voice data. Specifically, the speaker identification unit 170 identifies the area of a person whose mouth is moving as a result of analyzing the image acquired in synchronization with the voice data. Then, the speaker can be identified from the result of the person identification process for the area of the person whose mouth is moving. The minutes-taking unit 180 generates minutes-taking data by associating the speaker identification result by the speaker identification unit 170 with text data generated based on the voice data acquired by the voice acquisition unit 160. The minutes-taking unit 180 can also add the list of people generated by the list creation unit 150 to the minutes-taking data as participants of the meeting.

[Hardware configuration example]
The video conference system 1 of this embodiment has the same hardware configuration (e.g., FIG. 2) as that of the first embodiment. The storage device 1040 of this embodiment further stores program modules for realizing the functions of the list creation unit 150, the voice acquisition unit 160, the speaker identification unit 170, and the minutes creation unit 180 described above. The processor 1020 reads these program modules onto the memory 1030 and executes them to realize each of the functions of this embodiment described above.

[Processing flow]
The flow of processing executed by the video conference system 1 of the fourth embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart illustrating the flow of processing executed by the video conference system 1 of the fourth embodiment.

First, the voice acquisition unit 160 acquires voice data of the conference (S302). The voice data of the conference is generated by a sound collection device 50 provided at each location. The sound collection device 50 is connected to the communication terminal 20. The voice acquisition unit 160 communicates with the communication terminal 20 at each location via the network interface 1060, and can acquire the voice data generated by the sound collection device 50 at that location.

Then, the speaker identification unit 170 identifies the speaker of the voice data acquired by the voice acquisition unit 160 (S304). As an example, the speaker identification unit 170 can identify the speaker of the voice data acquired by the voice acquisition unit 160 as follows. First, the speaker identification unit 170 compares the voice data with the voiceprint data of each participant registered in advance in the storage device 1040 or the like, and identifies voiceprint data whose degree of match with the voiceprint of the voice data meets a criterion. Then, the speaker identification unit 170 can identify the speaker of the voice data acquired by the voice acquisition unit 160 by acquiring identification information of the participant associated with the identified voiceprint data (such as the name of the person or an ID assigned to each person). As another example, the speaker identification unit 170 can identify the speaker of the voice data acquired by the voice acquisition unit 160 as follows. First, the speaker identification unit 170 analyzes the image acquired by the image acquisition unit 110 in synchronization with the voice data. Specifically, the speaker identification unit 170 detects the mouth area of a person from within an image, and determines whether or not that area (i.e., the mouth) is moving between multiple images arranged in time series. The speaker identification unit 170 can then identify the speaker of the voice data acquired by the voice acquisition unit 160 by acquiring the results of the person identification process by the person identification unit 130 for the area of the person whose mouth area is determined to be moving. Note that it is assumed here that all people have been identified using, for example, the configuration described in the third embodiment.

The minutes-taking unit 180 generates minutes-taking data based on the processing results of the voice acquisition unit 160 and the speaker identification unit 170 (S306). Specifically, the minutes-taking unit 180 converts the voice data acquired by the voice acquisition unit 160 into text data using an API (Application Programming Interface) that converts voice data into text. The minutes-taking unit 180 also acquires information on the speaker of the voice data identified by the speaker identification unit 170 (e.g., the speaker's name, etc.). The minutes-taking unit 180 then associates the text data generated from the voice data acquired by the voice acquisition unit 160 with the information on the person identified as the speaker of the voice data, and adds the text data to the minutes-taking data. The minutes-taking unit 180 may also read the list of participants in the meeting generated by the list creation unit 150, and add the information on the participants to the minutes-taking data.

As described above, according to the configuration of this embodiment, minutes of a conference held using the videoconferencing system 1 can be automatically created. This reduces the effort required for conference participants to create minutes.

The above describes the embodiments of the present invention with reference to the drawings, but these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

In addition, in the multiple flow charts used in the above explanation, multiple steps (processing) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the illustrated steps can be changed to the extent that does not cause any problems in terms of content. In addition, each of the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

A part or all of the above-described embodiments may be described as, but is not limited to, the following supplementary notes.
1.
An image acquisition means for acquiring an image showing participants of a conference;
a person area detection means for detecting an area recognized as a person from within the image;
A person identification means for executing a person identification process for identifying a person included in the area;
a display control means for causing a display device to display first information indicating that the person has not been identified in the person identification process;
A video conferencing system comprising:
2.
The display control means further causes the display device to display second information including names of the participants of the conference identified in the person identification process.
1. The video conference system according to claim 1.
3.
The display control means causes a display mode of the first information to differ from a display mode of the second information.
2. The video conference system according to claim 1.
4.
The display control means makes the first information more prominent than the second information.
3. The video conference system according to claim 1.
5.
the display control means makes the first information stand out by making at least one of an outer shape, a size, a color, a language, and a font of the first information different from that of the second information;
4. The video conference system according to claim 1.
6.
the display control means displays the second information in a list format when the number of participants of the meeting identified by the person identification process is equal to or greater than a predetermined threshold.
2. The video conference system according to any one of 2. to 5.
7.
the display control means causes the first information to be displayed on a display device viewed by the participants of the conference who are shown in the image;
1. The video conference system according to any one of claims 1 to 6.
8.
the image acquisition means acquires an additional image, in which a participant not identified in the person identification process based on the image, from a mobile imaging device different from the imaging device that generated the image;
the person identification means identifies a participant who has not been identified in the person identification process based on the image, based on the additional image;
8. The video conference system according to any one of 1 to 7.
9.
The system further includes a list creation means for creating a list of the people identified by the person identification process.
8. The video conference system according to claim 1.
10.
A voice acquisition means for acquiring voice data;
a speaker identification means for identifying a speaker of the voice data by analyzing the voice data or an image acquired in synchronization with the voice data;
and a minutes creating means for creating minutes data by associating the speaker identification result with text data created based on the voice data.
8. The video conference system according to claim 9.
11.
The display control means makes the first information less noticeable than the second information.
3. The video conference system according to claim 1.
12.
The computer
Acquire images of the people attending the meeting,
Detecting an area recognized as a person from within the image;
execute a person identification process to identify a person included in the area;
displaying, on a display device, first information indicating that the person could not be identified in the person identification process;
23. A video conferencing method comprising:
13.
The computer,
and displaying, on the display device, second information including names of the participants of the conference identified in the person identification process.
13. The video conferencing method according to claim 12, comprising:
14.
The computer,
executing a process of making a display mode of the first information different from a display mode of the second information;
13. The video conferencing method according to claim 12, further comprising:
15.
The computer executes a process for making the first information more prominent than the second information.
14. The video conferencing method according to claim 13, further comprising:
16.
The computer,
At least one of the outer shape, size, color, language, and font of the first information is made different from that of the second information, thereby making the first information stand out.
15. The video conferencing method according to claim 15, comprising:
17.
The computer,
When the number of participants in the conference identified by the person identification process is equal to or greater than a predetermined threshold, the second information is displayed in a list format.
17. The video conferencing method according to any one of claims 13 to 16, comprising:
18.
The computer,
displaying the first information on a display device viewed by a participant of the conference who is shown in the image;
18. The video conferencing method according to any one of claims 12 to 17, comprising:
19.
The computer,
acquiring an additional image, in which a participant who was not identified in the person identification process based on the image is captured, from a mobile imaging device different from the imaging device that generated the image;
identifying a participant who was not identified in the person identification process based on the image based on the additional image;
19. The video conferencing method according to any one of claims 12 to 18, comprising:
20.
The computer,
Creating a list of people identified by the person identification process;
19. The video conferencing method according to claim 19, comprising:
21.
The computer,
Acquire audio data,
Identifying a speaker of the voice data by analyzing the voice data or an image acquired in synchronism with the voice data;
generating minutes data by associating the speaker identification result with text data generated based on the voice data;
21. The video conferencing method according to claim 19 or 20, comprising:
22.
The computer,
performing a process to make the first information less noticeable than the second information;
14. The video conferencing method according to claim 13, further comprising:
23.
A program for causing a computer to execute the video conference method according to any one of 12. to 22.

1 Video conference system 10 Server device 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input/output interface 1060 Network interface 110 Image acquisition unit 120 Person area detection unit 130 Person identification unit 140 Display control unit 150 List creation unit 160 Audio acquisition unit 170 Speaker identification unit 180 Minutes creation unit 20 Communication terminal 30 Imaging device 32 Imaging device 40 Display device 50 Sound collection device

Claims (11)

An image acquisition means for acquiring an image showing participants of a conference;
a person area detection means for detecting an area recognized as a person from within the image;
A person identification means for executing a person identification process for identifying a person included in the area;
a display control means for causing a display device to display first information indicating that the person could not be identified by the person identification process and second information including the names of the participants of the conference identified by the person identification process;
Equipped with
The display control means makes the first information more prominent than the second information ,
The display control means displays the second information in a list format when the number of participants in the conference identified by the person identification process is equal to or greater than a predetermined threshold . the display control means makes the first information stand out by making at least one of an outer shape, a size, a color, a language, and a font of the first information different from that of the second information;
The video conferencing system of claim 1 . The display control means displays the second information in an area other than the specified area.
3. A video conference system according to claim 1 or 2. the display control means changes a size of the second information in accordance with a size of a free area in the image.
A video conference system according to any one of claims 1 to 3 . the display control means causes the first information to be displayed on a display device viewed by the participants of the conference who are shown in the image;
A video conferencing system according to any one of claims 1 to 4 . the image acquisition means acquires an additional image, in which a participant not identified in the person identification process based on the image, from a mobile imaging device different from the imaging device that generated the image;
the person identification means identifies a participant who has not been identified in the person identification process based on the image, based on the additional image;
A video conferencing system according to any one of claims 1 to 5 . The system further includes a list creation means for creating a list of the people identified by the person identification process.
The video conferencing system of claim 6 . A voice acquisition means for acquiring voice data;
a speaker identification means for identifying a speaker of the voice data by analyzing the voice data or an image acquired in synchronization with the voice data;
and a minutes creating means for creating minutes data by associating the speaker identification result with text data created based on the voice data.
8. A video conference system according to claim 6 or 7 . An image acquisition means for acquiring an image showing participants of a conference;
a person area detection means for detecting an area recognized as a person from within the image;
A person identification means for executing a person identification process for identifying a person included in the area;
a display control means for causing a display device to display first information indicating that the person could not be identified by the person identification process and second information including the names of the participants of the conference identified by the person identification process;
Equipped with
the display control means makes the first information less noticeable than the second information ,
The display control means displays the second information in a list format when the number of participants in the conference identified by the person identification process is equal to or greater than a predetermined threshold . The computer
Acquire images of the people attending the meeting,
Detecting an area recognized as a person from within the image;
execute a person identification process to identify a person included in the area;
displaying, on a display device, first information indicating that the person could not be identified by the person identification process and second information including the names of the participants of the conference identified by the person identification process;
Making the first information more prominent than the second information;
When the number of participants in the conference identified by the person identification process is equal to or greater than a predetermined threshold, the second information is displayed in a list format.
23. A video conferencing method comprising: A program for causing a computer to execute the video conferencing method according to claim 10 .

Images