JP6550951B2 - Terminal, video conference system, and program - Google Patents

Description

  The present invention relates to a terminal, a video conference system, and a program.

  2. Description of the Related Art Conventionally, a video conference system in which two video conference terminals are connected via a network such as the Internet is known. According to the video conference system, video and audio data are bidirectionally transmitted and received in real time, and real-time communication can be realized even between remote persons. It has also been already known that a plurality of users at multiple locations can participate in the same conference at the same time by using a plurality of video conference terminals.

  By the way, in a meeting where people gather, it is possible to recognize without knowing who the meeting attendee the speaker is talking to. On the other hand, in a video conference system, a meeting attendee does not always keep an eye on a camera or the like, and it is difficult to identify which speaker is speaking to whom.

  Therefore, for example, in the video conference system according to Patent Document 1, the user of the speaker side terminal 21 focuses on the video of the user of the audience side terminal 22 among the images displayed by the display means. The terminal 22 receives, from the terminal 21, first attention information indicating that the terminal 22 is focused.

  Then, when receiving the first attention information, the terminal 22 determines whether or not the terminal corresponding to the video that the user is paying attention matches with the terminal 21. The terminal 22 generates and transmits the second attention information when it is determined that the two match, and when the terminal 21 receives the second attention information, it corresponds to the terminal 22 among the images displayed by the display unit. Emphasize the image and display it.

  For example, according to Patent Document 1, when oneself becomes a speaker, it can be understood whether or not the other party to which he / she is focusing is also focusing on himself / herself. However, in the process of determining the speaker, both terminal users have to call both names, which is troublesome.

  Further, in Patent Document 1, when the images of both users are focused, the process of highlighting the other party's image on the own terminal is performed only after the attention information is mutually transmitted and received. For this reason, for example, when a problem occurs in transmission / reception of attention information due to a transmission delay, if the above processing is delayed, the real-time property of the video conference system is impaired, and smooth communication is hindered.

  The present invention has been made in view of such circumstances, and its object is to realize smooth communication in a video conference system.

  In order to achieve the above-mentioned object, according to the present invention, imaging information obtained by imaging another user who uses the other terminal from the other terminal connected via the network and voice information emitted in the vicinity of the other terminal Whether the receiving means for receiving the attention information indicating that the other user is focusing on the own terminal and the other terminal of the transmission source of the voice information and the other terminal of the transmission source of the attention information coincide with each other The apparatus is characterized by comprising: an utterance passive judging unit to judge, and an utterance passive notifying unit to notify an utterance from another user who uses the other terminal to the own terminal when it is judged by the utterance passive judging unit that the other terminal matches. I assume.

  According to the present invention, it is possible to realize smooth communication in a video conferencing system.

It is the schematic of the video conference system in embodiment of this invention. It is a hardware block diagram of the terminal in embodiment of this invention. It is a functional block diagram of the terminal in the embodiment of the present invention. It is a flowchart which shows the outline of the process sequence in embodiment of this invention. It is a flowchart which shows the production | generation procedure of the gaze terminal ID signal in embodiment of this invention. It is a schematic diagram which shows an example of the gaze determination process in the embodiment of the present invention. It is a flowchart which shows the utterance passive notification process procedure in embodiment of this invention. It is a time chart which shows the flow of the utterance passive notification in embodiment of this invention.

  The terminal and the video conference system according to the embodiment of the present invention will be described below with reference to the drawings, but the present invention is not limited to this embodiment as long as the purpose of the present invention is not exceeded. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and the redundant description will be appropriately simplified or omitted. In addition, the embodiments described below are the best modes of the present invention, and do not limit the scope of the claims of the present invention.

  First, an outline of the video conference system 1 according to the present embodiment will be described with reference to FIG. This figure shows a system in which four terminals constituting this system are respectively arranged at four bases A to D and each terminal is connected via a network 14. Here, although a system composed of four bases where four users exist respectively is illustrated, a system composed of three or less bases is composed of five or more bases. It may be a system.

  A schematic configuration example of the terminal and peripheral device group arranged at each site will be described using the configuration arranged at the site A. At the site A, a video conference terminal 4 as a terminal, an imaging device 5 such as a gaze tracking device that tracks the line of sight of the user A, a camera 6 that captures the user A, and a microphone that acquires the sound emitted by the user A Etc., an audio output device 8, and an image output device 9 for displaying an image of the user A and an image of each user received from each terminal arranged at each site. Since the other bases other than the base A have the same configuration, the description thereof is omitted.

  The line-of-sight tracking device as the imaging device 5 is a device that is installed in front of the user A and acquires line-of-sight data of the user A by measuring the eye movement of the user A. The camera 6 is used to photograph the user A, perform predetermined image processing from the photographed image, and detect the position of the line of sight, its change, the angle, and the like. In addition, as the gaze tracking device, a glasses-type device may be used. In the present invention, the imaging device 5 and the camera 6 are collectively referred to as an imaging device.

  The video conference terminal 4 receives the imaging of each user from each site and causes the video output device 9 to display it, analyzes the imaging of the user A, transmits it to the video conference terminals at each site, and this implementation to be described later. For example, an information processing apparatus such as a personal computer that performs various types of information processing according to the form. Detailed description of the video conference terminal 4 will be described later. Note that data transmission / reception such as imaging may be performed via a mediator such as a server.

  In the base A, the voice input device 7 is a microphone or the like which receives voices emitted by the user A and voices emitted in the vicinity of the video conference terminal 4. The voice output device 8 is a speaker or the like that outputs voice and the like of each user received from each site. Note that these may be incorporated in the video conference terminal 4 or may be separately connected to the video conference terminal 4 as an external microphone, an external speaker, or the like.

  In addition, the video output device 9 at the site A includes the image of the user B at the site B at the upper left of the screen, the image of the user C at the site C at the upper right of the screen, and the image of the user D at the site D at the lower left of the screen. It is needless to say that this display mode is an example, although the imaging of the user A is displayed on the screen.

  In the present embodiment, when the gaze target of the user using the video conference system is detected and uttered while gazing, it is assumed that the utterance is directed to the gaze target, and the user of the utterance target utters this utterance. Notify that you are speaking to yourself. In the following, description will be made using an example in which the user A at the base A gazes at the image of the user D at the base D output to the video output device 9 and speaks.

  In this case, the video conference terminal of the base D determines that the speech is directed from the user A to the user D, and the base D displays a red frame surrounding imaging of the user A at the base A. Further, in this case, a voice such as a beep is output at the start timing and the end timing of the speech, and the user D is notified of the sound.

  In the present embodiment, the imaging of each site is displayed by dividing the screen into four, but in the case of, for example, a one-to-one system, one screen shows, for example, the appearance of the entire conference room as imaging of the other party. Such an image display may be possible. In addition, although one user may be at one site, a plurality of users may be at one site, but in this case, it is preferable to be able to acquire line-of-sight data of a plurality of people.

  Next, the hardware configuration of the terminal in the video conference system of this embodiment will be described with reference to FIG. Although the configuration of the base A will be described as an example, the other bases B to D have the same configuration, so the description will be omitted.

  The video conference terminal 4 includes an input unit 28, a memory 29, a CPU 30, and a network interface 31 (hereinafter “network I / F”). As other hardware, for example, an HDD or various external or built-in media drives may be provided.

  The input unit 28 is an operation button for performing various operations such as power ON / OFF, volume change, and the like. The memory 29 is a program for executing each processing in the present embodiment, various control programs, input / output video / audio data, sight line data, a gaze terminal ID signal described later, a speaker ID signal, an utterance passive ID signal, etc. It is ROM, RAM, etc. which are saved.

  The CPU 30 controls the operation of the video conference terminal 4 and performs encoding and decoding processing of video data. The network I / F 31 transfers various data using a communication network.

  Next, functional blocks of the video conference terminal 4 in the present embodiment will be described with reference to FIG. Here, in accordance with the example of FIG. 1, description will be made using a configuration example in which four bases A to D are connected by the network 14. The functional blocks of the video conference terminal at sites B to D have the same configuration as that of the video conference terminal 4 and thus are shown as a simplified diagram in this figure, and redundant description with the configuration of site A will be omitted.

  In the drawing, the imaging device 5, the audio input device 7, the audio output device 8, and the video output device 9 are collectively referred to as an input / output device group 10. In addition, the video conference terminal 4 'and the input / output device group 10' are in the site B, the video conference terminal 4 "and the input / output device group 10" are in the site C, and the video conference terminal 4 "'is in the site D. An output device group 10 "'is provided.

  The video conference terminal 4 includes a video input unit 15, a data transmission unit 16, an attention information generation unit 17, a voice acquisition unit 18, a data reception unit 21, a speaker determination unit 22, and a speaker identification signal generation unit. The configuration includes an utterance passive determination unit 24 and an output unit 25. The output unit 25 includes an utterance passive notification unit 251.

  The video input unit 15 is an imaging information acquisition unit that acquires a video captured by the imaging device 5. The acquired video is separately compressed / encoded by the video compression unit.

  The data transmission unit 16 is a transmission unit that transmits the video data acquired by the video input unit 15 to the video conference terminal at each site via the network 14. Moreover, the data transmission part 16 transmits the below-mentioned gaze terminal ID signal which is attention information to the video conference terminal of each base.

  The attention information generation unit 17 analyzes the imaging information of the other user who uses the other terminal acquired by the video input unit 15, and uses the own terminal displayed on the video output device 9 by the output unit 25 which is a display unit. It is attention information generation means for generating attention information indicating that another user who is using another terminal is focusing on the imaging information of the own user. Note that the names of the own terminal and the other terminal and the names of the own user and the other user in the present embodiment are for convenience, and do not define superiority or inferiority as to which is the main or sub.

  The attention information is, for example, a gaze terminal ID signal generated when the line-of-sight data acquired by the line-of-sight tracking apparatus described with reference to FIG. 1 is concentrated on the user video at a specific base. The gaze terminal ID signal includes the name of the gaze target base and the terminal identification signal at the gaze base. Details will be described later. Note that attention information is not limited to line-of-sight data, for example, analyzes the user's face orientation etc. from imaging of the user, and specifies whether the face orientation is directed to a specific target by known image analysis technology However, it may be generated based on the analysis result.

  The voice acquisition unit 18 is voice information acquisition means for acquiring voice information input to the voice input device 7 such as a microphone, for example. The audio input device 7 receives an input of audio information emitted near the video conference terminal 4 at the video conference terminal 4 at the base A. That is, in this case, in addition to the voice uttered by the user A, the voice input device 7 also receives an input such as a surrounding sound. When the acquired voice data is compressed and encoded, it is decoded by the voice decompression unit.

  The data receiving unit 21 includes imaging information obtained by imaging another user who uses the other terminal from another terminal connected via the network, voice information emitted in the vicinity of the other terminal, and the other user who is the own terminal. Receiving means for receiving attention information indicating that attention is being paid. Although imaging information of other users is compressed / encoded by the video compression unit in the video conference terminal at each site, this is decoded by the video decompression unit.

  The speaker determination unit 22 analyzes the voice information emitted in the vicinity of the plurality of other terminals received by the data reception unit 21, and among the plurality of other users who use the plurality of other terminals, any other user speaks It is determined whether the The analysis of the voice information may be performed by, for example, a method of determining based on the input gain of the acquired voice information, but the analysis is not limited to this and the analysis may be performed by a known method.

  The speaker identification signal generation unit 23 is a speaker identification signal for identifying a speaker for the own terminal based on an identification signal for identifying another terminal used by another user determined to be speaking by the speaker determination unit 22. Generate The speaker identification signal includes an identification signal for identifying the transmission source terminal of the voice information of another user who is determined to be speaking by the speaker determination unit 22.

  The utterance passive determination unit 24 determines whether or not the other terminal transmitting the voice information matches the other terminal transmitting the attention information.

  When the utterance passive determination unit 24 determines that the other terminal from which the voice information is transmitted matches the other terminal from which the attention information is transmitted, the utterance passive notification unit 251 receives the own terminal from the other user who uses the other terminal. This is an utterance passive notification means for notifying the utterance to the.

  Further, the utterance passive notification unit 251 utters, from the other user, the imaging information displayed on the video output device 9 by the output unit 25 of the other user who uses the other terminal determined to match by the utterance passive determination unit 24. It functions as a display control unit that controls the display on the video output device 9 by the output unit 25 so as to change the passive to a display mode that can be recognized by the user of the own terminal. Specifically, with the change of the display mode, as described with reference to FIG. 1, for example, a red frame is displayed so as to surround the imaging information displayed on the video output device 9, or the imaging information is The display mode is changed so that it is more conspicuous than the imaging information of other bases, such as enlargement of the imaging information.

  On the other hand, when the utterance passive determination unit 24 determines that the other terminal of the transmission source of the voice information and the other terminal of the transmission source of the attention information match, the own user uses the own terminal from other users who use the other terminal. An utterance passive identification signal may be generated that identifies that the utterance is being directed to In this case, the utterance passive notification unit 251 may notify the utterance passive based on the utterance passive identification signal generated by the utterance passive determination unit 24. Although details will be described later, by generating the utterance passive signal, the terminal can easily recognize that it is in the utterance passive state.

  In addition, when the utterance passive identification signal is received from the utterance passive determination unit 24, the utterance passive notification unit 251 may control the output unit 25 to output a predetermined voice. The predetermined voice is, for example, the above-described beep sound. This enables the user to recognize that the speech directed to him / her is started.

  Furthermore, the utterance passive notification unit 251 may perform control to output a predetermined voice when reception of the utterance passive identification signal from the utterance passive determination unit 24 is interrupted. This enables the user to recognize that the speech directed to him has ended.

  Furthermore, the utterance passive notification unit 251 may continue the utterance passive notification for a certain period of time after the reception of the utterance passive identification signal from the utterance passive determination unit 24 is discontinued. Although details will be described later, for example, when reception of an utterance passive identification signal is interrupted, a situation in which a user beeps immediately and a response is urgently avoided is avoided, and the user performs a response slowly. Can contribute to smooth communication.

  The utterance passive notification unit 251 increases the volume of a predetermined voice output in advance when the utterance passive identification signal is received, and decreases the volume increased when reception of the utterance passive identification signal is interrupted. It is good to do. Thereby, the user can recognize the timing of the speech passive start and end by the change of the sound volume.

  Furthermore, the utterance passive notification unit 251 may perform control to vibrate the video conference terminal when the utterance passive identification signal is received, and to vibrate the video conference terminal again when the reception of the utterance passive identification signal is stopped. . Thereby, the user can recognize each timing of the speech passive start and end by vibration.

  Further, even if the data reception unit 21 stops receiving the attention information, the utterance passive determination unit 24 continues generating the utterance passive signal until the data reception unit 21 stops receiving the audio information. It is also good. Although the details will be described later, even if the gaze terminal ID signal which is attention information is not received because of this, for example, the other user ceases to gaze at the displayed image, while the utterance by the other user is continued Because it is considered as being in the speech passive state, smooth communication is not impeded.

  Here, for example, it is assumed that the user D at the site D speaks while gazing at the user A displayed on the display which is the video output device. The utterance passive determination unit 24 of the video conference terminal 4 compares a gaze terminal ID signal, which is attention information received from each site, with a speaker ID signal. Then, the utterance passive determination unit 24 determines that the user A is in the utterance passive state, and generates the utterance passive ID signal, when the base names indicated by the two correspond as a result of the comparison. In this example, since the gaze terminal ID signal and the speaker ID signal both indicate the D base, the user A is in an utterance passive state by the user D.

  Next, a schematic procedure of processing in the present embodiment will be described with reference to FIGS. Here, as in the description using FIG. 1, it is assumed that the user D at the site D is watching the image of the user A at the site A.

  First, the video conference terminal 4 '' at the site D acquires an image of the user D via the imaging device, and acquires line-of-sight data of the user D from the image (step S1).

  Next, the attention information generation unit 17 of the video conference terminal 4 ′ ′ generates a gaze terminal ID that is attention information (step S2). Details of gaze terminal ID generation processing by the attention information generation unit 17 will be described later.

  The video conference terminal 4 ″ ′ at the site D transmits the imaging and gaze terminal ID of the user D to the video conference terminal 4 at the site A (step S3).

  The speaker determination unit 22 of the base A determines the speaker, and the speaker identification signal generation unit 23 generates a speaker ID signal which is a speaker identification signal (step S4). Details of the speaker ID signal generation will be described later.

  Then, the utterance passive determination processing is executed by the utterance passive determination unit 24 (step S5). Details of this process will be described later.

  The utterance passive notification unit 251 performs display control processing for imaging of the user who is focusing on himself (step S6).

  Details of the gaze terminal ID generation process shown in FIG. 4 will be described with reference to FIG. Here, the gaze determination processing procedure at the site A will be described as an example, assuming that the user A at the site A speaks to the user D at the site D.

  First, the video conference terminal 4 acquires line-of-sight data of the user A from the imaging device 5 (step S11). For example, the line-of-sight data is acquired as coordinate data represented by (x, y).

  Since the line-of-sight data normally changes with time, the video conference terminal 4 updates the line-of-sight data at a predetermined update frequency (step S12). For example, when the frame rate of the video data is 30 fps, the update frequency is preferably set at 33 msec intervals. At this time, the imaging device 5 acquires the amount of change when the eyeball is imaged as an angle.

  The video conference terminal 4 sequentially stores the line-of-sight data acquired in the memory 29 and refers to, for example, the past 10 data (step S13). Then, the video conference terminal 4 performs the stop determination based on the sight data for the past 10 data (step S14). In this case, it is preferable to use the determination condition as disclosed in, for example, Non-Patent Document 1 or the like on the basis of the feature of human eye movement, for example.

That is, in this embodiment, as the stop determination condition,
1: "The line-of-sight data before and after does not deviate by more than 2.1 degrees in the viewing angle"
2: “The maximum distance between two line-of-sight data in the past 10 data is not more than 2.1 degrees away from the viewing angle”
However, this is an example, and not limited to this, other known judgment conditions may be adopted.

  Note that human blinking is generally said to be 100 msec to 150 msec, which corresponds to 3-5 data as line-of-sight data. When blinking, data cannot be obtained because line-of-sight data based on the user's eyeball cannot be acquired. Therefore, in the present embodiment, data loss up to 5 consecutive times out of the past 10 data is excluded from the stop determination, and determination is made with the remaining data.

  When it is determined in step S14 that the line-of-sight data is stopped (YES in step S14), the video conference terminal 4 confirms at which base on the screen the image is stopped (step S15). At this time, it is assumed that the video conference terminal 4 obtains information on which region on the screen the video at each site is displayed. Then, the video conference terminal 4 is stopped at which base video depending on whether or not the seat data of the line-of-sight data for 10 data determined to be parked falls within the rectangular area where each base video is displayed. Determine if it exists.

  In addition, when line-of-sight data exists at the boundary between the base images, the base relating to the large number of images in which the line-of-sight data are contained among the past 10 data is taken as the fixed base. If the line-of-sight data has the same number, neither of them is regarded as the station base, and the station determination processing up to that point is reset.

  On the other hand, when it is determined in step S14 that the line-of-sight data is not stopped (step S14, NO), the stop determination process is reset (step S113).

  Next, the video conference terminal 4 determines whether or not the stop base confirmed in step S15 is the same stop base as the previous time (step S16). In the case of the stopping base different from the previous time (step S16, NO), the video conference terminal 4 resets the stopping determination (step S113).

  On the other hand, in the case of the same stop base as the last time, that is, the previous gaze determination (step S16, YES), the video conference terminal 4 determines whether or not the gaze terminal ID signal is being transmitted (step S17).

  When the gaze terminal ID signal is not being transmitted (step S17, NO), the video conference terminal 4 sets the stop count to “+1” (step S18). On the other hand, when the gaze terminal ID signal is being transmitted (step S17, YES), the video conference terminal 4 does not perform the gaze determination and continues to transmit the gaze terminal ID signal to the video conference terminal at the target base as it is. (Step S112).

  After step S18, the video conference terminal 4 determines whether or not the stop has been continued three times (step S19). In the present embodiment, “gazing” is determined when the stop continues three times. If it is determined that the number of times is less than three, the process returns to step S12. By this determination, it is possible to determine that it is also a gaze to follow the target of the line of sight slowly and smoothly even if the line-of-sight data does not stay at a fixed position 30 times.

  The video conference terminal 4 repeats the process of step S18, and when it is determined that the stop has continued for three consecutive times (step S19, YES), determines that it is "gaze" and changes the stay base to the gaze base (step S110). ).

  The video conference terminal 4 generates a gaze terminal ID signal including its own base ID and the video conference terminal ID of the gaze base (step S111). Thereafter, the video conference terminal 4 transmits a gaze terminal ID signal to the gaze target video conference terminal (step S112).

  When it is determined that the line of sight changes greatly and the stop and gaze are finished in the stay determination in step S14 while the stay and gaze are occurring (NO in step S14), those states are reset. (Step S113).

  Further, after step S113, the video conference terminal 4 determines whether or not it is currently in the state of transmitting the gaze terminal ID signal (step S114), and determines that it is transmitting the gaze terminal ID signal (step S114). In step S114, YES), transmission of the gaze terminal ID signal is stopped (step S115). Note that the gaze terminal ID signal in step S112 is always transmitted during this period.

  Next, an example of gaze determination processing in the present embodiment will be described with reference to FIG. This figure shows the staying state of the line-of-sight data when the user A at the site A is paying attention to the imaging of the user D at the site D displayed on the screen. In the figure, a small circle with a numeral, such as “E1”, represents the position indicated by the line-of-sight data on the screen in the video output device 9, and the numeral attached to the small circle represents the data acquisition order. And

  It is determined that the change from E1 to E2 is not a stop under condition 1 of the stop determination. Similarly, it is determined that E3, E4, E5, and E6 are not stationary, and changes of E7 and E8 correspond to Condition 1 of the above-described stationary determination. Similarly, changes from E9 to E10 to E16 correspond to the condition for the stop determination, and assuming that the ten data correspond to the condition 2 for the stop determination described above, it can be said that the line-of-sight data E7 to E16 is stopped. Further, since the 10 data at this time is in the base image area of the base D indicated by the video 12, the stop base is D.

  Next, in the line-of-sight data E17 to E26 and E27 to E36, the stop base is D as well. Here, since the stop has occurred three times in succession, it is considered that E37 to E44 are being watched, and in this case, the stop determination is not performed. At this time, the video conference terminal 4 at the site A transmits a gaze terminal ID signal in which the ID of the site D and the ID of its own site are added to the video conference terminal at the site D.

  The gaze data at E44 to E45 has changed by more than the threshold value of the parking lot condition, and it is assumed that the gaze has ended here. At that time, the video conference terminal 4 can notify the video conference terminal of the base D in real time of the watching timing and the length thereof by stopping the transmission of the gaze terminal ID signal.

  Next, an utterance passive notification processing procedure according to the present embodiment will be described with reference to FIG. Here, it is assumed that the user A at the site A is speaking toward the user D at the site D, and the passive speech notification procedure at the site D will be described as an example.

  First, the video conference terminal at the site D receives the gaze terminal ID signal and the like from the video conference terminal 4 at the site A in addition to the video information and the audio information (step S21).

  Next, the video conference terminal of the base D analyzes the base of the received voice data from the transmission source terminal ID attached to the received data (step S22).

  Then, the video conference terminal of the base D performs speaker determination as to whether or not there is a speaker (step S23). The video conference terminal at the base D determines that there is a speaker (step S23, YES), and when it is determined that the speaker is speaking, generates a speaker ID signal including the ID of the base where the speaker is present (Step S24).

  When the video conference terminal of the base D receives the gaze terminal ID signal (YES in step S25), whether the base of the transmission source matches the base of the ID attached to the speaker ID signal Is determined (step S26). When it is determined that the two match (step S26, YES), the video conference terminal at the site D generates an utterance passive ID signal (step S27).

  Then, the video conference terminal at site D starts utterance passive notification (step S28). Here, the video conference terminal at the site D outputs a beep sound and displays the frame of the site video in which the site A that is the subject of speech passive is displayed in red on the screen of the video conference terminal ( Step S29). Step S29 indicates that the utterance passive notification is continued. In this case, the base image frame of the base A is continuously displayed in red.

  Next, the video conference terminal at the base D returns to step S21 to perform the next speaker determination. If the speaker is still speaking (step S22⇒step S23, YES), the speaker ID signal is updated (step S24) and continues to be generated continuously.

  Further, the video conference terminal at the site D has a large change in the line of sight of the user at the site A, and as a result, the video conference terminal does not receive the gaze site ID signal from the site A (NO in step S25). The utterance being uttered is considered to be directed to the user at the site D. That is, since the video conference terminal of the base D still generates the utterance passive ID signal in step S210 (step S210, YES), the utterance passive notification is continued (step S29). Here, when the utterance passive ID signal is not generated (step S210, NO), the video conference terminal of the base D returns to step S21 again, and receives the next data (step S21).

  The video conference terminal at site D receives each data again and makes the next determination (step S21). The video conference terminal at site D determines that there is no speaker in the speaker determination (step S23, NO) when the silent state continues for a certain period of time in the audio data analysis (step S22).

  In this case, the video conference terminal at the site D stops the utterance passive ID signal (step S211), and ends the utterance passive notification (step S212). Also, at this time, the video conference terminal at the site D restores the screen display state to the original state, and naturally prompts the user D at the site D to answer the speech from the user at the site A by outputting a beep sound. be able to.

  Further, if the speaker ID and the gaze base ID do not match in step S26 (step S26, NO), the video conference terminal of the base D determines that it is not an utterance directed to itself, Without returning to step S21.

  Next, the flow of passive utterance notification in this embodiment will be described with reference to FIG. Here, assuming that the user A at the site A speaks to the user D at the site D, each determination performed at the video conference terminal at each site and the process of the ID signal will be described in time series. Here, latency due to processing of each function, transmission delay, and the like can occur, but it is assumed that synchronization between video / audio data to be transmitted and received and line-of-sight data is guaranteed.

  First, in the video conference terminal 4 of the base A, it is determined in the video conference terminal of the base D that the line-of-sight data of the user A is parked in the base D video (step S31, this is referred to as “station 1”). (Step S41). The same determination is made in step S32 and step S33.

  At the same time, when a user at the site A speaks, a speaker determination as to which user at the site is speaking is performed at the video conference terminal at the site D that has received the voice data (step S42). A speaker ID signal is generated at the video conference terminal at site D (step S35).

  When the stop at the base A continues three times, it is determined that the base video of the base D is being watched (step S43), and the video conference terminal 4 at the base A transmits a watch base ID signal to the base D at the same time ( Step 37). The video conference terminal 4 at the site D receives the gaze site ID signal (step 38). At this time, since the speaker ID signal has already been generated, the video conference terminal at site D performs passive speech determination (step S44).

  The video conference terminal at the site D generates a passive speech ID signal when the gaze site ID signal matches the site indicated by the speaker ID signal (step S39). In addition, the video conference terminal at the site D notifies the user D that the utterance passive has been started by outputting a beep sound (step S310).

  Thereafter, the video conference terminal of the base D displays the red frame surrounding the base video of the base A displayed on the screen at the base D, and notifies the user D that the speech passive is continuing ( Step S311).

  Here, when the line-of-sight data of the user A at the base A fluctuates greatly and looks at another base or out of the screen (step S36), the video conference terminal 4 at the base A ends the transmission of the gaze base ID signal (step S37). Accordingly, the reception of the gaze base ID signal at the base D is also ended (step S38). On the other hand, since the video conference terminal at site D continues to generate the speaker ID signal (step S39), the passive speech notification is not terminated (step S311).

  When the speech of the user A ends at the base A (step S34), and it is determined by the speaker determination unit at the base D that the silent state continues for a predetermined time, the video conference terminal at the base D generates the speaker ID signal (Step S35), and the generation of the utterance passive ID signal is also stopped (step S39).

  In addition, the video conference terminal of the base D stops the utterance passive notification (step S311) by the red frame display surrounding the base video of the base A for a set predetermined time (hereinafter referred to as "set time length"). Hold (step S313). When the set time length has elapsed, the video conference terminal at the site D stops the passive utterance notification (step S311) and outputs a beep sound to the user D that the utterance of the user A has ended. Notification is made (step S312).

  As a result, the user at the site D can visually understand who is speaking to him among the users at other sites, and can also know the end timing. Therefore, it becomes possible to reply to the other user's utterance at a natural timing.

  As described above, according to the video conference system of the present embodiment, it is determined that the user's line-of-sight information is always tracked and the user is speaking while gazing at the video of the user at the other site displayed on the screen. Depending on the determination result, only the area where the speaker is displayed is displayed differently on the receiving terminal, and the notification sound is output at the start timing and the end timing of the speech. This makes it possible to notify the user on the receiving terminal side who and when he / she is speaking.

  Further, in the present embodiment, it is sufficient to analyze only the line-of-sight data on the utterer side without measuring the line-of-sight data of the user at both bases, and it is excellent in real time required for smooth communication.

  Also, in order to take account of the difference between the utterance length and the gaze length, even if the user of the utterance target is turned away during the utterance, it is determined that the utterance is directed to the gaze object until the utterance is finished. The receiving user can continue to be notified that the utterance is directed toward him / herself.

  Each of the above-described embodiments is a preferred embodiment of the present invention, and various modifications can be made without departing from the scope of the present invention. For example, each process in the above-described information processing apparatus and video conference system of the present embodiment can be executed using hardware, software, or a combined configuration of both.

  In the case of executing processing using software, it is possible to install and execute a program in which a processing sequence is recorded in a memory in a computer incorporated in dedicated hardware. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

DESCRIPTION OF SYMBOLS 1 Video conference system 4 Video conference terminal 5 Imaging device 6 Camera 7 Audio | voice input device 8 Audio | voice output device 9 Video | video output device 10 Input / output device group 14 Network 15 Image | video input part 16 Data transmission part 17 Attention information generation part 18 Voice acquisition part 21 Data reception unit 22 Speaker determination unit 23 Speaker identification signal generation unit 24 Utterance passive determination unit 25 Output unit 251 Utterance passive notification unit

JP 2010-200150 A

Takaki Wakiyama, 2 others, “Creation of interest model based on attention detection and picture recommendation”, Journal of Information Processing Society of Japan, May 2007, Vol.48 No.3, p. 1048-1057

Claims (12)

Imaging information obtained by imaging another user using the other terminal from another terminal connected via a network, voice information emitted in the vicinity of the other terminal, and the other user focusing on the own terminal Receiving means for receiving attention information indicating that
Utterance passive determination means for determining whether or not the other terminal of the transmission source of the voice information matches the other terminal of the transmission source of the attention information;
A terminal comprising: utterance passive notification means for notifying an utterance from the other user who uses the other terminal to the own terminal when it is determined by the utterance passive determination means that the other terminal matches. A display unit configured to display imaging information of the other user received by the reception unit;
The said utterance passive notification means is the user of the terminal of the utterance passive from the other user, the imaging information displayed by the display means of the other user who uses the other terminal determined to match by the utterance passive determination means. The terminal according to claim 1, characterized in that the display means is controlled to change to a display mode that can be recognized.   The utterance passive determination means generates an utterance passive identification signal for identifying that the other user using the other terminal is speaking when it is determined that the other terminals match. 2. The terminal according to 2.   4. The terminal according to claim 3, wherein the utterance passive notification means notifies the utterance passive based on the utterance passive identification signal generated by the utterance passive determination means.   5. The terminal according to claim 3, wherein said utterance passive notification means performs control to output a predetermined voice when said utterance passive identification signal is received from said utterance passive determination means.   The said speech passive notification means performs control which outputs a predetermined | prescribed audio | voice, when reception of the said speech passive identification signal from the said speech passive determination means is stopped. The terminal described in.   7. The terminal according to claim 6, wherein the utterance passive notification means continues the utterance passive notification for a certain period of time after the reception of the utterance passive identification signal from the utterance passive determination means stops.   The utterance passive determination means is characterized in that generation of the utterance passive identification signal is continued until reception of voice information is interrupted by the reception means even when reception of the attention information is interrupted by the reception means. The terminal according to any one of claims 3 to 7. A video conference system in which an own terminal and two or more other terminals are connected via a network,
The other terminal is
Imaging information acquisition means for acquiring imaging information of another user who uses the other terminal imaged by the imaging device;
Voice information acquisition means for acquiring voice information emitted near the other terminal;
Displayed on the same screen together with imaging information obtained by imaging at least the other user other than the other terminal received from the other terminal other than the other terminal. Display means to
The imaging information of the other user using the other terminal acquired by the imaging information acquiring means is analyzed, and the other terminal is used for the imaging information of the own user using the own terminal displayed by the display means Attention information generation means for generating attention information indicating that another user is looking at;
Imaging information of another user using the other terminal acquired by the imaging information acquiring means, audio information emitted in the vicinity of the other terminal acquired by the audio information acquiring means, generated by the attention information generating means And sending means for sending attention information to the terminal.
The terminal itself
Receiving means for receiving, from the other terminal, imaging information obtained by imaging the other user who uses the other terminal, voice information emitted in the vicinity of the other terminal, and the attention information;
Utterance passive determination means for determining whether or not the other terminal of the transmission source of the voice information matches the other terminal of the transmission source of the attention information;
A video conference system comprising: utterance passive notification means for notifying an utterance from another user who uses the other terminal to the own terminal when it is determined by the utterance passive determination means that the other terminal matches.   The attention information generation means is the number of staying lines of the line of sight of the other user analyzed from the imaging information of the other user using the other terminal with respect to the imaging information of the own user using the own terminal displayed by the display means The video conference system according to claim 9, wherein attention information is generated when is equal to or greater than a predetermined threshold. From other terminals connected via the network, imaging information obtained by photographing other users who use the other terminals, audio information emitted in the vicinity of the other terminals, and the other users paying attention to the own terminal A process of receiving attention information indicating that information and storing it in the storage unit;
A process of determining whether the other terminal as the transmission source of the voice information and the other terminal as the transmission source of the attention information match;
A program for causing a computer to execute a process of notifying an utterance from another user using the other terminal to the own terminal when it is determined that the other terminals match. A computer readable program to be executed by a video conference system in which an own terminal and two or more other terminals are connected via a network,
The other terminal is
A process of acquiring imaging information of another user who uses the other terminal imaged by the imaging device and storing the acquired imaging information in the storage unit of the other terminal;
A process of acquiring voice information emitted in the vicinity of the other terminal and storing it in a storage unit of the other terminal;
The imaging information obtained by imaging at least the own user using the own terminal received from the own terminal is arranged on the same screen along with the imaging information obtained by imaging the other terminal other than the other terminal received from the other terminal other than the other terminal Processing to be displayed on the display unit of the other terminal;
The imaging information of the other user using the other terminal stored in the storage unit of the other terminal is analyzed, and the other terminal is used for the imaging information of the own user using the own terminal displayed on the display unit A process of generating attention information indicating that another user is paying attention and storing the attention information in the storage unit of the other terminal;
It includes processing of transmitting imaging information of another user who uses the other terminal stored in the storage unit of the other terminal, voice information emitted in the vicinity of the other terminal, and the attention information to the own terminal ,
The own terminal is
A process of receiving imaging information obtained by imaging another user using the other terminal from the other terminal, voice information emitted in the vicinity of the other terminal, and the attention information and storing the information in a storage unit of the own terminal ,
A process of determining whether the other terminal as the transmission source of the voice information and the other terminal as the transmission source of the attention information match;
A program including: a process of notifying an utterance from another user who uses the other terminal to the own terminal when it is determined that the other terminal matches.

Images