JP6417652B2 - Information processing apparatus, information processing system, information processing apparatus control method, information processing system control method, and program - Google Patents

Description

  The present invention relates to a technique for recovering audio delay due to network delay or the like in an application that transmits and receives audio over a network.

  In an application that transmits and receives audio over a network, for example, a queue for accumulating uplink audio from each client may be prepared on the server in consideration of network fluctuations. However, having a queue also causes a situation in which delay increases as data accumulates in the queue.

  It is possible to recover the delay by discarding silence from the voice of each client received by the client. However, when mixing by the server, the downlink voice received by the client is one voice data mixed by the server. It becomes. In that case, if any one of the clients is speaking, the mixed voice is not silent and is not discarded, and the delay cannot be recovered.

  For this reason, when downlink audio is mixed and transmitted to a client by a server, when the downlink communication between the client and the server fluctuates, the audio that has already been transmitted to other clients is transmitted to the client. Therefore, a state in which all the audio received thereafter is delayed is maintained.

  As described above, when the delay increases, specifically, the following problems occur. If a remote conference is performed at multiple locations, audio delays accumulate over time, which may hinder the progress of the conference. For example, when the other party thinks that he / she is not speaking and there is a possibility that the voice is duplicated and cannot be heard. In addition, there is a possibility that the conversation does not hold, for example, the question is asked again because the answer to the question is not received.

  The communication system in Patent Document 1 has a system configuration in which audio data is transmitted from a transmission device to a reception device, and the transmission side device generates silence state information based on the volume level of the audio data, and the transmission device A jitter absorption buffer that accumulates data and adjusts the delay time, and when the amount of audio data stored in the jitter absorption buffer exceeds the set allowable storage amount, the allowable storage amount is increased and the jitter absorption buffer A jitter buffer adjustment unit that discards audio data indicating a silent state and restores the permissible accumulated amount to the default value when the accumulated audio data amount is within the set allowable accumulated amount for a certain period of time. It is.

JP2012-124689

  However, the technique described in Patent Document 1 is limited to a case where the transmission terminal and the reception terminal are in a one-to-one relationship. For example, the relationship is a one-to-many relationship as in a conference system. Audio data is not directly sent from one client to another client. For example, it is not possible to transmit audio data to a client terminal after mixing audio data of another client terminal in a conference server.

An object of the present invention is a mechanism to send audio data, it is to provide a mechanism capable of restoring a delay by changing the voice data to be transmitted.

According to the present invention, a predetermined unit of audio data received from the client terminal that can communicate with a client terminal that transmits input audio data to an information processing apparatus in a predetermined unit An information processing apparatus including an accumulation storage unit that accumulates and stores as a pre-synthesis audio data group for each of the predetermined data acquired from the pre-synthesis audio data group for each of the client terminals accumulated in the accumulation storage unit Synthetic data generation means for generating synthetic data using unit voice data, and transmission means for transmitting the synthetic data generated by the synthetic data generation means to a client terminal that is a transmission destination among a plurality of client terminals when the audio data is not used for synthesis when receiving a notification from the client terminal, in the accumulation storage unit Comprising specifying means for specifying the pre-combination audio data group for each of the client terminal product, and the synthetic data generating means, when receiving a notification from the client terminal, said in the pre-combination voice data group The voice data not used for the synthesis specified by the specifying unit is skipped , the next voice data to be synthesized is acquired, and the synthesized data for the client terminal that has received the notification is generated.

According to the present invention, the mechanism for sending voice data, it is possible to provide a mechanism capable of restoring a delay by changing the voice data to be transmitted.

It is an example of the figure which shows the system configuration | structure which concerns on embodiment of this invention. It is an example of the figure which shows the structure of the hardware which concerns on embodiment of this invention. It is an example of the figure which shows the function structure of the software which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a reproduction delay and recovery processing in the client terminal which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows the condition where the audio | voice information is stored in a queue in the conference server which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a delay state based on the reception state from a client terminal, and a recovery process in the conference server which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a delay state based on the transmission state to a client terminal, and a recovery process in the conference server which concerns on embodiment of this invention. It is an example of the flowchart of the reproduction | regeneration delay recovery process in the client terminal which concerns on embodiment of this invention. It is an example of the flowchart of the delay recovery process based on the reception state from the client terminal in the conference server which concerns on embodiment of this invention. It is an example of the flowchart of the delay recovery process based on the transmission state to the client terminal which concerns on embodiment of this invention. FIG. 8 is an example of a diagram illustrating a delay recovery method when a delay occurs in FIG. 7 using a queue image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is an example of a diagram showing a system configuration according to an embodiment of the present invention. The client terminals 101 (multiple) and the conference server 102 can be connected via a network.

  In the conference system (FIG. 1), one of the users participating in the conference (for convenience, the organizer) accesses the conference server 102 from the client terminal 101 and reserves the conference room. The conference room is a possible conference space, and only “invited participants” described later can enter the room. In addition, as a free space, an unspecified user can participate or cannot speak, but only viewing is possible.

  The organizer determines the conference ID (or conference room ID, room number, etc.) for identifying the conference, the time to use the conference room, etc., and invites a specific participant to be registered in the conference server. Participant's notification destination (for example, e-mail address) may be used to call for participation. In order to allow only a specific user to participate, a password for entering the conference room can be described in the notification.

  Here, audio data from a microphone (not shown) of a user (actual conference participant) of each client terminal 101a to 101c or imaging data from a camera (not shown) is shown as an image in FIGS. Thus, it is once transmitted to the conference server 102. For example, audio is transmitted from each client terminal 101 to the conference server 102 as uplink audio for the number of client terminals 101.

  The above uplink voice is mixed by the conference server 102 with the uplink voice received from the plurality of client terminals 101 and distributed to the other client terminals 101 (downlink voice data). However, it is not necessary to mix the voice of the terminal itself with the downlink voice data sent back to the client terminal 101 that transmitted a certain uplink voice during mixing.

  Here, although the client terminal 101 and the conference server 102 are described as separate cases, one client terminal 101 may be configured to have the functions of the conference server 102 in the same case.

  In the embodiment of the present invention, the description will be given by taking the conference system as an example. However, the present invention is not necessarily limited to the conference system. For example, voice exchange is performed via a network, and packets (data including voice data are ) Provided as a usable technique when a delay occurs.

  FIG. 2 is an example of a diagram illustrating a hardware configuration according to the embodiment of the present invention. As shown in FIG. 2, the client terminal 101 and the conference server 102 are connected via a system bus 204 to a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, A configuration is adopted in which a video controller 206, a memory controller 207, a communication I / F controller 208, and the like are connected. The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  Further, the ROM 203 or the external memory 211 will be described later, which is necessary for realizing the functions executed by each server or each PC, such as BIOS (Basic Input / Output System) and OS (Operating System) which are control programs of the CPU 201. Various programs are stored. Further, information necessary for carrying out the present invention is stored. The external memory may be a database.

  The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for executing the processing from the ROM 203 or the external memory 211 to the RAM 202 and executing the loaded program.

  The input controller 205 controls input from a keyboard (KB) 209 or a pointing device such as a mouse (not shown).

  The video controller 206 controls display on a display device such as the display 210. The display device may be a display device such as a liquid crystal display. These are used by the administrator as needed.

  The memory controller 207 is an external storage device (hard disk (HD)), flexible disk (FD), or PCMCIA (Personal Computer) that stores a boot program, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a Compact Flash (registered trademark) memory connected to a Memory Card International Association (Card Memory) card slot via an adapter.

  The communication I / F controller 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

  Note that the CPU 201 can display on the display 210 by executing an outline font rasterization process on a display information area in the RAM 202, for example. Further, the CPU 201 enables a user instruction using a mouse cursor (not shown) on the display 210.

  Various programs to be described later for realizing the present invention are recorded in the external memory 211 and executed by the CPU 201 by being loaded into the RAM 202 as necessary. Furthermore, definition files and various information tables used when executing the program are also stored in the external memory 211, and a detailed description thereof will be described later.

  FIG. 3 is an example of a diagram showing a functional configuration of software according to the embodiment of the present invention. The software components of the client terminal 101 and the conference server 102, each storage unit, and data exchange between them (excluding the dotted arrows associated with the audio data 328 and the like from the terminal 3) are shown in the figure.

  First, an outline of the flow of audio data will be described. When the client terminal 101 receives voice input from the user, it is converted into voice data and transmitted to the conference server 102. The audio data received by the conference server from the plurality of client terminals 101 is stored in a queue prepared for each client terminal 101. Audio data transmitted to a certain client terminal 101 is obtained by mixing audio data one by one from a queue corresponding to another client terminal 101 except for the audio data of the client terminal 101 itself (corresponding to the queue). To the client terminal 101. The client terminal 101 that has received the mixed voice data reproduces the mixed voice data of the other client terminal 101 (the voice data of the client terminal 101 itself is not mixed), so that the user listens to the voice. I can do it.

  In the audio input unit 311, the client terminal 101 receives input of audio data through a connected device such as a microphone used by the user. The audio data that has received the input is transmitted from the audio transmission unit 312 to the conference server 102. Here, the voice data is divided into a certain size (for example, voice inputted for 10 milliseconds with respect to time). In the following description, the term “voice data” refers to input voice data divided into a certain size. This is merely an example, and it is assumed that the conference system standard such as the number of data bits is followed.

  The audio data (upstream audio data) transmitted from the client terminal 101 is received by the audio receiving unit 321 of the conference server 102. The received audio data is stored in the mixing audio storage unit 326. The mixing audio storage unit 326 prepares a queue corresponding to each client terminal 101 participating in the conference, and stores a certain number of audio data. In FIG. 3, in the mixing audio storage unit 326, each element of the queue corresponding to each client terminal 101 is arranged in a vertical direction (seven rectangles in the example) corresponding to the terminals 1 to 3. Show.

  At the time of audio reception, if the uplink check unit 322 determines that processing delay has occurred due to reception (acquisition) of uplink audio data, delay recovery processing is performed. Specifically, this processing is performed for each received client terminal 101, and the number of audio data stored in the corresponding queue exceeds a predetermined number based on the upstream delay threshold of the server threshold storage unit 327. If it is determined that there is, the silence data is deleted.

  Here, one unit of audio data stored in each queue corresponds to one rectangle, and more specifically, a pair of audio data 328 and transmitted terminal information 329 is stored. The audio data 328 is audio data received from the client terminal 101 described above. On the other hand, the received audio data is mixed with audio data received from other terminals and transmitted from the conference server 102 to each client terminal 101 (mixed audio transmission unit 323). At that time, a transmitted client terminal 101 and an untransmitted client terminal 101 are generated due to some state (for example, a difference in network line speed). Therefore, information on whether the client terminal 101 has already been transmitted or the client terminal 101 that has not been transmitted is described in “transmitted terminal information 329” so that a delay recovery process described later can be performed for each client terminal 101. Any data structure may be used as long as the transmitted / untransmitted client terminal 101 can be identified.

  When mixing audio data to be transmitted to a certain client terminal 101, the audio data originally received from the client terminal 101 is omitted from the object to be mixed. That is, the audio data sent from the client terminal 101 other than itself is mixed.

  The downlink check unit 324 performs a delay recovery process on the client terminal 101 to be mixed (that is, the client terminal 101 other than the client terminal 101 to be transmitted) when the mixing audio transmission unit 323 transmits the audio data to the client terminal 101. . Specifically, this processing is performed when mixing the silence data in each queue when the downstream delay flag (existing for each client terminal 101) of the server threshold value storage unit 327 is “ON”. Skip and mix the next audio data.

  After completion of the delay recovery process by the downlink check unit 324 (including a case where it is determined that there is no need for delay recovery), the mixed data is transmitted to a predetermined client terminal 101. In the process of the conference server 102, the audio reception unit 321 and the mixing audio transmission unit 323 do not need to be synchronized processes.

  In the mixing voice receiving unit 313 of the client terminal 101, the conference server 102 receives data obtained by mixing voice data other than that of the client terminal 101 and stores it in the received voice storage unit 316 (queue). At that time, if it is determined that the number exceeds a predetermined number based on the reproduction delay threshold value of the terminal threshold value storage unit 317, the silence data is deleted.

  The audio reproduction unit 315 takes out the mixed audio from the received audio storage unit 316 (queue) and reproduces it. In the processing of the client terminal 101, the mixing audio reception unit 313 and the audio reproduction unit 315 do not have to be synchronized processing.

  The audio delay recovery process is executed by both the client terminal 101 and the conference server 102. An outline of each delay recovery process will be described with reference to FIG. 4 (client terminal 101 side) and FIGS. 5 to 7 (conference server 102 side) using an image of a queue storing audio data.

  FIG. 4 is a diagram showing an example of an image showing the occurrence of reproduction delay and recovery processing in the client terminal according to the embodiment of the present invention. In FIG. 4, 1) a state where no delay has occurred, 2) a state where a delay has occurred, and 3) a delay recovery method will be described. Note that all the queues according to the embodiment of the present invention described with reference to FIGS. 4 to 7 are described as FIFOs. First, “a state in which no delay occurs” will be described with reference to 1).

(1) First, the client terminal 101 receives audio data (mixed audio data) from the conference server 102.
(2) The audio data is stored in the received audio storage unit 316 as data in a waiting state for reproduction.

  (3) In the example of FIG. 4, only the received “A1” is stored. However, it may be regarded as a delay if the number is within a desired range based on the reproduction delay threshold. The queue is a FIFO, and the previously stored audio data is first extracted and reproduced.

  (4) The voice data at the head of the queue is taken out and played back by the voice playback unit 315, so that the user of the client terminal 101 can hear it.

  In the following description of FIG. 4, the flow of (1) to (4) is the same although there is a difference in whether or not a delay occurs.

  Next, the “state in which delay occurs” will be described with reference to 2). For example, it is assumed that a plurality of packets (represented by the same symbols as the voice data for convenience. In the example, seven packets A1 to A7) are received at the client terminal 101 almost simultaneously due to network delay or the like. Seven audio data are accumulated in the queue (received audio storage unit 316) of the client terminal 101, and reproduction of the audio data is delayed.

  In the subsequent processing, since the audio data that is deleted from the queue for playback and the audio data that is received and stored in the queue are in principle the same timing, the delay speed is constant (naturally there is a delay). It will never recover).

  Therefore, if another delay occurs for some reason, the total number of the delayed audio data is gradually accumulated, and finally reaches a level at which the user feels uncomfortable with the conference call. That is, when images captured by the camera are transmitted / received separately, it is conceivable that the speech image and sound are significantly different from each other, and the context of speech content with other client terminals 101 is incorrect.

  Therefore, as described above, based on the reproduction delay threshold value, if the number is within a desired range, it is regarded as a delay. However, when the predetermined range is exceeded, it is necessary to perform a delay recovery process during reproduction.

  The “delay recovery method” will be described with reference to 3) at the end of FIG. That is, based on the reproduction delay threshold value, delay recovery processing is performed assuming that the number of desired ranges has been exceeded. The audio data in the queue stores the audio data of A1 to A7. Of these, the audio data to which “()” is given (A2, A4, A5, A6 are described as (A2), for example) ) Is “silent data”. Here, the silent data is “voice data that is regarded as not including voice” in the system. The determination as to whether or not the audio data is silent data is a well-known technique disclosed in “Japanese Patent Laid-Open No. 2000-31223” and will not be described.

  For audio data reproduction, audio data is acquired one by one from the head of the queue and reproduced. The number of audio data stored in the queue is counted at the time of acquisition, and this number is set as a reproduction delay threshold. Based on this, when the number of desired ranges is exceeded, delay recovery processing is started. In that case, all audio data stored in the queue is confirmed, and the silence data is deleted. The delay recovery process is performed at a time without being interrupted by other processes.

FIG. 5 is a diagram showing an example of an image showing a situation in which audio information is stored in a queue in the conference server according to the embodiment of the present invention. FIG. 5 shows a state when no delay occurs.
(1) First, the conference server 102 receives audio data for each client terminal 101.

  (2) Next, the received audio data is stored in a queue prepared for each client terminal 101 in the mixing audio storage unit 326. The information to be stored is not only the audio data 328 as described with reference to FIG. 3, but also a transmission that stores for each other client terminal 101 whether or not the other client terminal 101 has already mixed and transmitted the audio data. Stored terminal information 329 is also stored. Both (328, 329) are collectively referred to as audio information for convenience. In the initial state immediately after being placed in the queue, it is not transmitted to any other client terminal 101. The information indicating whether or not the transmission has been completed may be a flag prepared with an array of all other client terminals 101, or a list of terminals that have already been transmitted or a list of terminals that have not been transmitted. Any data structure may be used.

(3) Next, the audio data of each queue (audio data included in the audio information at the head of the queue) is extracted and mixed. As described above, the audio data of other queues are mixed except for the queue corresponding to the client terminal 101 to which the audio data is to be transmitted.
(4) The mixed audio data is transmitted to the client terminal 101.

  FIG. 6 is a diagram showing an example of an image showing the generation and recovery processing of the delay state based on the reception state from the client terminal in the conference server according to the embodiment of the present invention. In FIG. 6, first, “1) a situation in which delay occurs due to acquisition of uplink voice data (acquisition of voice data transmitted from the client terminal 101 to the conference server 102)” will be described.

  (1) First, it is assumed that audio data is received from the client terminal 101A without delay. On the other hand, seven audio data (B1 to B7) are received from the client terminal 101B due to a network problem or the like.

  (2) Accordingly, one audio information is stored in the queue corresponding to the client terminal 101A, but seven audio information is stored in the corresponding queue in the client terminal 101B.

  (3) Mixing for transmission to the client terminal 101C uses audio data extracted from the queues for the client terminal 101A and B. From the queue for the client terminal 101A, the voice data “A7” (voice having the same time as “B7”) is acquired and used, but from the queue for the client terminal 101B, the voice data “B1” (from “B7”) is used. (Sound of 6 unit time ago) remains as delay data. Accordingly, mixing audio data of “A7 + B1” with different time is generated.

  As in the case of reproduction in the client terminal 101 described above, the audio information that is deleted from the queue for mixing and the audio information that is received and stored in the queue are in principle the same number, so the delay speed is constant. (The delay will not recover naturally). Accordingly, when a delay occurs only for the audio data received from the client terminal 101B for some reason, the total number of the delayed audio data is gradually accumulated.

  (4) The audio data that is finally transmitted (mixed) to the client terminal 101 reaches a level that is determined to be a problem by the user. That is, the audio data of the client terminals 101 other than the client terminal 101 that has received the mixed audio data is being mixed, but there is a possibility that the conversation may not be established due to a time lag.

  Next, “2) Delay recovery method” for the above-described delay will be described. As an example, it is assumed that audio information exceeding a predetermined number of ranges is accumulated in the queue received from the client terminal 101B as in 1) (determination based on the uplink delay threshold). When it is determined that audio information exceeding a predetermined number of ranges has been accumulated, delay recovery processing is started. In the queue on the left side of the figure “2)”, there are four pieces of voice information (those assigned “()”) corresponding to silence data among B1 to B7, and these are deleted. Whether the data is silent data is determined by the client terminal 101 and added to the audio data to be transmitted. The conference server 102 does not analyze the inside of the actual voice data, but determines whether or not the data is silence based on the additional information of the voice data transmitted from the client terminal 101. However, the conference server 102 may determine whether the data is silence data. Note that the conference server may determine whether or not the audio data is silent even in the audio data downlink processing (audio data transmission to the client terminal 101). The rest is stored as three (B1, B3, B7) audio information as shown on the right queue. These are executed for every queue (prepared for each client terminal 101). The delay recovery process is performed at a time without being interrupted by other processes.

FIG. 7 is a diagram showing an example of an image showing the generation and recovery processing of the delay state based on the transmission state to the client terminal in the conference server according to the embodiment of the present invention. In FIG. 7, first, “3) a situation in which delay occurs due to downlink audio data acquisition (mixed audio data transmission from the conference server 102 to the client terminal 101)” will be described.
(1) The conference server 102 receives audio data from both the client terminals 101A and 101B.

  (2) Voice information (information in which voice data and transmitted terminal information are associated) is stored in each corresponding queue. In 3) of FIG. 7, seven pieces of audio information are stored.

  (3) Extract audio data from each queue for each client terminal 101. In order to mix the audio data to be sent to each client terminal 101, all audio data of the other client terminals 101 other than itself are extracted for each client terminal 101.

(4) Next, the mixed audio data is transmitted to the client terminal 101C. At that time, a delay may occur depending on the state of the network. On the other hand, it is assumed that the voice transmitted to the client terminal 101B (voice data obtained by mixing the voice data of the client terminal 101A and the client terminal 101C) can be transmitted without delay. In this case, the voice of the client terminal 101A that has been transmitted to the client terminal 101B but not transmitted to the client terminal 101C exists in the queue. If there is at least one untransmitted client terminal 101 in the transmitted terminal information 329 of the voice information, the voice information stored in the queue cannot be deleted. Even if the data is not deleted, the data is ignored in the mixing / transmission in the client terminal 101B, but cannot be ignored in the client terminal 101C. Therefore, the audio reproduced at the client terminal 101A and the audio reproduced at the client terminal 101C It is also assumed that time intervals have accumulated and the temporal context of the conversation gradually becomes unnatural.
Next, "4) Delay recovery method" for the above-described delay will be described with reference to FIG.

  FIG. 11 is an example of a diagram illustrating a delay recovery method in the case where a delay occurs in FIG. 7 using a queue image.

As an example, when a delay occurs in the voice transmitted to the client terminal 101C (determined by the downlink delay flag), the silence data is skipped when the voices of the client terminal 101A and the client terminal 101B are taken out from the queue and mixed. Delay is recovered by mixing non-silent audio data. Note that if there is no untransmitted terminal in the skipped silence data, it is deleted from the queue. In the queue of “4)”, there are four pieces of audio information (with “()” added) corresponding to silent data among B1 to B7, and these are skipped. These are executed for every queue (prepared for each client terminal 101). The delay recovery process is performed at a time without being interrupted by other processes.

  The flowcharts of FIGS. 8 to 10 explain the delay recovery processing for the delay in the client terminal 101 or the conference server 102.

FIG. 8 is an example of a flowchart of the reproduction delay recovery process in the client terminal according to the embodiment of the present invention. Each step (S801 to S813) in the flowchart of FIG. 8 is executed by the CPU 201 of the client terminal 101, and steps S814 to S815 are executed by the CPU 201 of the conference server 102. The flowchart of FIG. 8 corresponds to the image of the queue of FIG.
In step S801, mixed audio data transmitted from the conference server is received.

In S802, the received mixed voice data is stored in the queue of the received voice storage unit 316 of the client terminal 101 (enqueue).
In step S803, the number of audio data stored in the queue is counted.

  In step S804, it is determined whether audio data exceeding a predetermined number of ranges is accumulated in the queue (determination based on a reproduction delay threshold).

  In S805, the process branches based on the determination as to whether or not a predetermined number of out-of-range audio data has been accumulated. When the predetermined number is within the range, the process proceeds to NO and proceeds to S810. If the predetermined number is out of the range, the process proceeds to YES and proceeds to S806 in order to execute the <silent discard> routine.

The processes in S806 to S809 are performed at a time without being interrupted by other processes. Here, all of the audio data in the queue of the client terminal 101 is checked.
In S806, one audio data at the head of the queue is dequeued (taken out).
In S807, it is checked whether or not the extracted audio data is silence data.

  If it is determined in S808 that the checked audio data is silent, the process returns to S806 to check the next audio data as it is (without returning the audio data to the queue). That is, audio data determined to be silent is discarded. If it is determined that the checked audio data is not silent, the audio data (taken from the queue in S806) is enqueued (stored) again in S809.

  As described above, the queue is a FIFO, and the processing of S806 to S809 is performed once for all audio data without being interrupted by other processing, so that all the silent data is deleted. Even after a change from the left cue described in 4-3) to the right cue, the temporal order of the remaining audio data is ensured.

  As described above, when a delay occurs in the queue of the client terminal 101 that has received the mixed audio data in the received audio storage unit 316, the delay recovery process is performed.

  Next, in S810, it is counted again whether audio data exceeding a predetermined number of ranges is accumulated in the queue.

  In S811, it is determined whether audio data exceeding a predetermined number of ranges has been accumulated (may be the same value as the reproduction delay threshold value or a different value). That is, it is determined whether or not a sufficient effect has been obtained as a result of completing the delay recovery process.

  In S812, if audio data exceeding a predetermined number of ranges is accumulated (in the case of YES), the process proceeds to S813. If not (NO), the process returns to S801 (sound reception). Note that the audio reproduction is performed by an asynchronous process and is not related to the delay recovery process, and thus is not shown in the flowchart.

  In step S813, the conference server 102 is notified of delay recovery processing. That is, the method of deleting silent data cannot be further handled by the client terminal 101. Therefore, in this case, the conference server side is requested to support for delay recovery. Further, after notifying the conference server 102 of the delay recovery process, the process returns to S801. On the other hand, in S814 of the conference server 102, the notification from S813 of the client terminal 101 is received.

In step S815, the “downlink delay flag” is turned on to indicate that there is a request for support for recovery support processing from the client terminal 101. As a result, the “downbound voice delay recovery” process operating in the conference server 102 may be activated. However, “downlink voice delay recovery” may be activated by a determination in an original routine regardless of whether there is a request from the client terminal 101.
This completes the description of the flowchart relating to the delay recovery processing in the client terminal 101.

  FIG. 9 is an example of a flowchart of delay recovery processing based on the reception state from the client terminal in the conference server according to the embodiment of the present invention. Each step of the flowchart of FIG. 9 is executed by the CPU 201 of the conference server 102. The following processing is performed for each individual queue prepared for each client terminal 101 in the conference server 102. Further, since the processing described with reference to the flowchart of FIG. 9 is audio data received by the conference server 102 from the client terminal 101, the audio delay recovery processing relating to uplink audio data is described. The flowchart in FIG. 9 corresponds to the images in FIGS. 5 and 6.

  In step S <b> 901, voice data (more precisely, a communication packet including voice data) is received from the client terminal 101. Here, only one flowchart is described, but specifically, even if one receiving unit accepts the data and determines which of the plurality of client terminals 101 transmits the data. In the first place, when the connection between the conference server 102 and the client terminal 101 is established, a receiving unit may be generated in multiple threads for each “client terminal 101” and may be used as a receiving unit for a specific client terminal 101. However, in S902 and after, it is assumed that the client terminal 101 is identified by the receiving unit, and the processing corresponds to one identified client terminal 101.

  In step S902, the voice data received in step S901 is already stored in the queue corresponding to the identified client terminal 101 in the mixing voice storage unit 326 in the voice information (the voice data in the packet and the transmitted terminal described above). Check how many pieces of information 329) are stored.

  In S903, it is determined whether or not the number of the audio information exceeds a predetermined number according to the upstream delay threshold value of the server threshold value storage unit 327.

  In S904, when it is determined that the number of audio information in the queue exceeds the predetermined number according to the uplink delay threshold (in the case of YES), the process proceeds to S905. If it is determined that it has not exceeded (NO), the process proceeds to S909.

  In S909, it is determined that the number of audio information stored in the queue does not exceed the predetermined number according to the uplink delay threshold, that is, it is determined that the audio information can still be stored (without performing delay recovery processing). (Enqueue) and return to S901 for the next voice reception.

  When the processing proceeds to S905, the processing from S905 to S908 is repeated for all the audio information stored in the queue. In the meantime, the loop from S905 to S908 does not come off.

  If it is determined that the predetermined number is exceeded according to the upstream delay threshold, one piece of audio information is extracted (dequeued, taken out from the queue) from the queue in S905.

  In S906, it is determined whether or not the audio data included in the audio information extracted in S905 is “silence data”. In S907, when the audio data included in the audio information extracted in S905 is “silent data” (in the case of YES), the process returns to S905. That is, since the dequeued audio information is silent data, it is not returned (deleted) to the queue.

  In S908, since the dequeued audio information is not silence data, it is enqueued (returned to the end of the queue).

  By performing the processing from S905 to S908, the audio information is reduced by the number of silence data in the queue, so that an effect of reducing the audio delay can be obtained. This completes the description of the flowchart in FIG. 9.

  FIG. 10 is an example of a flowchart of delay recovery processing based on the transmission state to the client terminal according to the embodiment of the present invention. Each step of the flowchart of FIG. 10 is executed by the CPU 201 of the conference server 102. The flowchart of FIG. 10 is a process of mixing the audio information (audio data included) in the queues of the plurality of client terminals 101 in the mixing audio storage unit 326 of the conference server 102 and transmitting the audio information to the client terminal 101. Since it is transmitted from the conference server 102 to the client terminal 101, it is uplink voice data. This corresponds to the image of the queue in FIG.

The voice data of the terminal itself is not mixed with the mixed voice data transmitted to a certain client terminal 101. Accordingly, the queues of the mixing audio storage unit 326 processed in the flowchart of FIG. 10 are processed for each queue corresponding to a queue other than the client terminal 101 to be transmitted.
In S1001, one piece of audio information is extracted from the queue (dequeue).
In S1002, the downlink delay flag that is turned “ON” in S815 of FIG. 8 is checked.

  In S1003, when the downlink delay flag is “ON” (in the case of YES), the process proceeds to S1004. In other words, this is because a delay recovery process on the conference server 102 side is requested from the client terminal 101 side. If it is “OFF” (NO), the process proceeds to S1007.

  In S1002, the threshold may be used in the same manner as the check at the time of uplink (uplink delay threshold in the server threshold value storage unit 327) instead of checking the downlink delay flag. This threshold value is set as a downlink delay threshold value, and when the number of audio information in the queue reaches a predetermined number based on the downlink audio threshold value, the process proceeds to S1004 (YES direction) in S1003, and if not, the process proceeds to S1007. .

  In S1004, it is checked whether or not the audio data included in the audio information dequeued in S1001 is silence data.

  In S1005, when the audio data checked in S1004 is “silent data” (in the case of YES), the process proceeds to S1006, the next audio information is taken out from the queue, and the process returns to S1004 to repeat the process. By repeating this process, the process of ignoring “silent data” proceeds.

  On the other hand, if the sound data checked in S1004 is not “silent data” (NO), the process proceeds to S1007, and is transmitted to the client terminal 101 as mixed sound in S1008 as described above. In S1007, the audio data used for mixing is not returned to the queue as unnecessary data. However, in actuality, since it may be necessary to use it in another client terminal 101, “all client terminals (except for the client terminal that has transmitted this audio data) have not yet transmitted this audio data. In this case, it is necessary to enqueue again. Above, description of the flowchart of FIG. 10 is completed.

  It should be noted that the configuration and contents of the various data described above are not limited to this, and it goes without saying that the various data and configurations are configured according to the application and purpose.

  Although one embodiment has been described above, the present invention can take an embodiment as, for example, a system, apparatus, method, program, or recording medium, and specifically includes a plurality of devices. The present invention may be applied to a system including a single device.

  Further, the program according to the present invention is a program capable of executing the processing method of the flowcharts of FIGS. 8 to 10, and the storage medium of the present invention can execute the processing method of the flowcharts of FIGS. 8 to 10. The program is stored. The program in the present invention may be a program for each processing method of each device in the flowcharts of FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by executing the reading.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying a computer program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, an EEPROM, Silicon disks, solid state drives, etc. can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention. Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention.
In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 Client terminal 102 Conference server 103 Network 311 Audio input unit 312 Audio transmission unit 313 Mixing audio reception unit 314 Reception audio check unit 315 Audio reproduction unit 316 Reception audio storage unit 317 Terminal threshold storage unit 321 Audio reception unit 322 Uplink check unit 323 Mixing Audio transmission unit 324 Down check unit 325 Mixing unit 326 Mixing audio storage unit 327 Server threshold storage unit 328 Audio data 329 Transmitted terminal information

Claims (15)

The voice data of a predetermined unit received from the client terminal that can communicate with the client terminal that transmits the input voice data to the information processing device in a predetermined unit is synthesized for each client terminal that is the transmission source of the voice data. An information processing apparatus including an accumulation storage unit for accumulating and storing as a voice data group,
Synthesized data generating means for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
Transmission means for transmitting the composite data generated by the composite data generation means to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Specific means to
With
When the synthesized data generating unit receives a notification from the client terminal that there is a delay due to the transmission of the synthesized data, the synthesized data specified by the identifying unit in the pre-synthesized audio data group is not used for the synthesis An information processing apparatus that skips data, acquires next audio data to be synthesized, and generates synthesized data for the client terminal that has received the notification. The synthesized data generating means acquires voice data from the pre-synthesis voice data group for each destination client terminal that transmits the synthesized data, and generates the synthesized data for the destination client terminal,
The transmission means transmits the composite data generated for the destination client terminal to the target client terminal,
The specifying means specifies voice data not to be used for synthesis when there is a delay due to the transmission of the synthesized data by the transmission means from the pre-synthesis voice data group,
The synthesized data generation means skips voice data not used for the synthesis specified by the specifying means, acquires next voice data to be synthesized, and generates synthesized data for the client terminal. The information processing apparatus according to claim 1 . Wherein the voice data is not used in the synthesis, the information processing apparatus according to any one of claims 1 or 2, characterized in that a silence data that are considered contains no speech. The information processing apparatus according to claim 3 , wherein the silence data that is regarded as not including speech is speech data to which additional information indicating silence is added. The voice data of a predetermined unit received from the client terminal that can communicate with the client terminal that transmits the input voice data to the information processing device in a predetermined unit is synthesized for each client terminal that is the transmission source of the voice data. A method for controlling an information processing apparatus including an accumulation storage unit that accumulates and stores audio data as a group of audio data,
A synthesized data generating step for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
A transmitting step of transmitting the combined data generated by the combined data generating step to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Specific steps to
Including
In the synthesis data generation step, when a notification that there is a delay due to the transmission of the synthesis data is received from the client terminal, the voice that is not used for the synthesis specified in the specification step in the pre-synthesis voice data group A control method characterized by skipping data, acquiring next voice data to be synthesized, and generating synthesized data for the client terminal that has received the notification. The voice data of a predetermined unit received from the client terminal that can communicate with the client terminal that transmits the input voice data to the information processing device in a predetermined unit is synthesized for each client terminal that is the transmission source of the voice data. A program that can be executed by an information processing apparatus including an accumulation storage unit that accumulates and stores audio data as a group of audio data,
The information processing apparatus;
Synthesized data generating means for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
Transmission means for transmitting the composite data generated by the composite data generation means to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Function as a specific means to
If the synthesized data generating means receives a notification from the client terminal that there is a delay due to transmission of the synthesized data, the voice that is not used for the synthesis specified by the specifying means in the pre-synthesis voice data group A program for skipping data, acquiring the next voice data to be synthesized, and functioning as means for generating synthesized data for the client terminal that has received the notification. A client terminal that transmits input audio data to the information processing apparatus in a predetermined unit, and a predetermined unit of audio data received from the client terminal, the pre-synthesis audio data group for each client terminal that is the source of the audio data An information processing system including an information processing apparatus including an accumulation storage unit that accumulates and stores as
Synthesized data generating means for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
Transmission means for transmitting the composite data generated by the composite data generation means to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Specific means to
With
When the synthesized data generating unit receives a notification from the client terminal that there is a delay due to the transmission of the synthesized data, the synthesized data specified by the identifying unit in the pre-synthesized audio data group is not used for the synthesis An information processing system characterized by skipping data, acquiring next audio data to be synthesized, and generating synthesized data for a client terminal that has received the notification. A client terminal that transmits input audio data to the information processing apparatus in a predetermined unit, and a predetermined unit of audio data received from the client terminal, the pre-synthesis audio data group for each client terminal that is the source of the audio data An information processing system control method including an information processing apparatus including an accumulation storage unit that accumulates and stores as
A synthesized data generating step for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
A transmitting step of transmitting the combined data generated by the combined data generating step to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Specific steps to
Including
In the synthesis data generation step, when a notification that there is a delay due to the transmission of the synthesis data is received from the client terminal, the voice that is not used for the synthesis specified in the specification step in the pre-synthesis voice data group A control method characterized by skipping data, acquiring next voice data to be synthesized, and generating synthesized data for the client terminal that has received the notification. A client terminal that transmits input audio data to the information processing apparatus in a predetermined unit, and a predetermined unit of audio data received from the client terminal, the pre-synthesis audio data group for each client terminal that is the source of the audio data A program for controlling an information processing system including an information processing apparatus including an accumulation storage unit that accumulates and stores as
The information processing system;
Synthesized data generating means for generating synthesized data using a predetermined unit of voice data acquired from the pre-synthesized voice data group for each client terminal stored in the storage unit;
Transmission means for transmitting the composite data generated by the composite data generation means to a client terminal that is a transmission destination among a plurality of client terminals;
When receiving notification from the client terminal that there is a delay due to the transmission of the synthesized data, the voice data not used for synthesis is identified from the pre-synthesis voice data group for each client terminal stored in the storage unit Function as a specific means to
If the synthesized data generating means receives a notification from the client terminal that there is a delay due to transmission of the synthesized data, the voice that is not used for the synthesis specified by the specifying means in the pre-synthesis voice data group A program for skipping data, acquiring the next voice data to be synthesized, and functioning as means for generating synthesized data for the client terminal that has received the notification. A storage unit that stores and stores a predetermined unit of audio data received from the client terminal as an audio data group that can communicate with a client terminal that transmits the input audio data to the information processing apparatus in a predetermined unit. An information processing apparatus,
Transmitting means for transmitting audio data of a predetermined unit acquired from the audio data group to a client terminal as a transmission destination;
A means for identifying, from the audio data group, audio data that is not transmitted when a notification that there is a delay due to the transmission of the audio data is received from the client terminal;
With
When the transmission unit receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the transmission unit skips the audio data specified by the specifying unit from the audio data group and receives the next audio An information processing apparatus that acquires data and transmits the data to a client terminal that has received the notification. A storage unit that stores and stores a predetermined unit of audio data received from the client terminal as an audio data group that can communicate with a client terminal that transmits the input audio data to the information processing apparatus in a predetermined unit. A method for controlling an information processing apparatus,
Transmitting a predetermined unit of audio data acquired from the audio data group to a client terminal as a transmission destination;
A specific step of identifying, from the audio data group, audio data that is not transmitted when a notification that there is a delay due to transmission of the audio data is received from the client terminal;
Including
When the transmission step receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the audio data specified in the specifying step is skipped from the audio data group and the next audio is skipped. A control method comprising: acquiring data and transmitting the data to a client terminal that has received the notification. A storage unit that stores and stores a predetermined unit of audio data received from the client terminal as an audio data group that can communicate with a client terminal that transmits the input audio data to the information processing apparatus in a predetermined unit. A program for controlling an information processing apparatus,
The information processing apparatus;
Transmitting means for transmitting audio data of a predetermined unit acquired from the audio data group to a client terminal as a transmission destination;
When notifying that there is a delay due to the transmission of the audio data from the client terminal, the audio data not to be transmitted functions as a specifying unit that specifies from the audio data group,
When the transmission means receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the audio data specified by the specifying means is skipped from the audio data group and the next audio is skipped. A program for obtaining data and functioning as means for transmitting to the client terminal that has received the notification. Information processing apparatus comprising: a client terminal that transmits input voice data to an information processing apparatus in a predetermined unit; and a storage unit that stores and stores the predetermined unit of voice data received from the client terminal as a voice data group An information processing system including
Transmitting means for transmitting audio data of a predetermined unit acquired from the audio data group to a client terminal as a transmission destination;
A means for identifying, from the audio data group, audio data that is not transmitted when a notification that there is a delay due to the transmission of the audio data is received from the client terminal;
With
When the transmission unit receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the transmission unit skips the audio data specified by the specifying unit from the audio data group and receives the next audio An information processing system that acquires data and transmits the data to a client terminal that has received the notification. Information processing apparatus comprising: a client terminal that transmits input voice data to an information processing apparatus in a predetermined unit; and a storage unit that stores and stores the predetermined unit of voice data received from the client terminal as a voice data group A method for controlling an information processing system including:
Transmitting a predetermined unit of audio data acquired from the audio data group to a client terminal as a transmission destination;
A specific step of identifying, from the audio data group, audio data that is not transmitted when a notification that there is a delay due to transmission of the audio data is received from the client terminal;
Including
When the transmission step receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the audio data specified in the specifying step is skipped from the audio data group and the next audio is skipped. A control method comprising: acquiring data and transmitting the data to a client terminal that has received the notification. Information processing apparatus comprising: a client terminal that transmits input voice data to an information processing apparatus in a predetermined unit; and a storage unit that stores and stores the predetermined unit of voice data received from the client terminal as a voice data group A program for controlling an information processing system including:
The information processing system;
Transmitting means for transmitting audio data of a predetermined unit acquired from the audio data group to a client terminal as a transmission destination;
When notifying that there is a delay due to the transmission of the audio data from the client terminal, the audio data not to be transmitted functions as a specifying unit that specifies from the audio data group,
When the transmission means receives a notification from the client terminal that there is a delay due to the transmission of the audio data, the audio data specified by the specifying means is skipped from the audio data group and the next audio is skipped. A program for obtaining data and functioning as means for transmitting to the client terminal that has received the notification.

Images