JP7632479B2 - Video and audio synthesis device, method and program - Google Patents

Description

This relates to a video and audio synthesis system that synthesizes and outputs a single image and audio from multiple video and audio input signals.

In recent years, many video devices have come into use. The images produced by these many video devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in terms of physical signals, control signals, etc. depending on the standard, one screen is transmitted in a time period equal to one half of the frame rate.

One way to use these images is to display multiple cameras on fewer monitors than the number of cameras, such as in video conferencing. In such cases, multiple video/audio signals are, for example, split and displayed on one screen, or a video screen is fitted with a reduced display of another video screen, and the corresponding audio signals are synthesized at the same time as the images are synthesized.

Normally, the timing of a video signal is not synchronized, and since the timing of other video signals to be mixed is different, the signals are temporarily buffered in memory before being mixed. As a result, delays occur in the output of the mixed screen and in the audio signal.

If we imagine an ensemble of people in remote locations performing together over a video conference with this type of image/audio synthesis, the delay involved in this synthesis would greatly impair its feasibility. For example, in a piece of music with 120 beats per minute (hereafter referred to as 120 BPM (beats per minute)), the duration of one beat is 60/120 seconds = 500 milliseconds. If we were to match this with an accuracy of 5%, then the delay between capturing the image with a camera and displaying it would need to be kept to 500 x 0.05 = 25 milliseconds or less.

In reality, the process from capturing an image with a camera to displaying it must include other delays, such as the time it takes to process the image on the camera, the time it takes to display on the monitor, and the time it takes to transmit, in addition to the processing involved in synthesis. The accompanying audio signal also includes a delay. As a result, with conventional technology, it is difficult to collaborate in applications where timing is important, such as playing an ensemble while watching each other's videos from remote locations. However, if there is low latency in either the video or audio signal, collaborative work in applications where timing is important may be possible.

Therefore, for collaborative work that requires strict low latency, we provide a system that synthesizes multiple screens and multiple audio signals from multiple locations, etc., and reduces the latency from the video input/audio signal input of asynchronous video to the audio output in particular.

VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013

The present disclosure aims to reduce the delay time until an acoustic signal is output.

The video and audio synthesizer of the present disclosure comprises:
Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos;
Separating a video signal and an audio signal from the plurality of audio-accompanying video signals;
The synthesis of the video signal is performed separately from the processing of the audio signal,
To output an audio signal without waiting for completion of a synthesis process of a video signal.

The video and audio synthesis method of the present disclosure includes:
The video and audio synthesis device
Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos;
Separating a video signal and an audio signal from the plurality of audio-accompanying video signals;
The synthesis of the video signal is performed separately from the processing of the audio signal,
To output an audio signal without waiting for completion of a synthesis process of a video signal.

The program of the present disclosure is a program for causing a computer to realize each functional unit of the video and audio synthesis device of the present disclosure, and is a program for causing a computer to execute each step of the method performed by the video and audio synthesis device of the present disclosure.

The present disclosure can reduce the delay time until an acoustic signal is output.

3 shows an example of screen information included in a video signal. An example of screen composition is shown below. 1 illustrates an example of a video and audio synthesis method related to the present disclosure. 1 illustrates an example of a video and audio synthesis method according to the present disclosure. 1 illustrates an example of a video and audio synthesis method according to the present disclosure. 1 shows an example of the configuration of a video and audio synthesis device according to an embodiment of the present invention.

Below, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are considered to be identical to each other.

Figure 1 shows an example of screen information contained in a video signal. The screen information is transmitted by scanning the screen horizontally, one scan line 21 at a time, and then scanning the scan lines 21 below in sequence. This scanning includes scanning of the display screen 24, as well as blanking portion 22 and overhead information/signals such as border portion 23. The blanking portion 22 may contain information other than video information, such as control information and audio information. (For example, see Non-Patent Document 1, Chapter 3.)

Figure 2 shows an example of video signal synthesis. In this disclosure, as an example, four video signals are input to a video and audio synthesizer, which synthesizes them into one video signal and outputs it. In a video signal, one screen is transmitted using a time of one frame rate. For example, if the video signal has 60 frames per second, one screen of the video signal is transmitted over 1/60 seconds, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frame per Second)). The information of one screen at each time included in the video signal is called a "video frame", the information of one screen of each video signal input to the video and audio synthesizer is called an "input frame", and the information of one synthesized screen output from the video and audio synthesizer is called an "output frame". In this disclosure, the "input frame" and "output frame" may include an audio signal.

For example, as shown in Figure 3, consider a case where a video and audio synthesis device inputs video signals and audio signals with four different timings, synthesizes them into a single output frame, and outputs the input frame by reading all of the input frames before synthesizing them. In this case, if the frame time is T_f and the synthesis processing time is T_p, the output frame into which the video and audio signals are synthesized will be delayed by a maximum of 2T_f+T_p from the time of input of the first input frame, input 1. For example, when considering a 60 fps video, this means that the synthesized video may contain a delay of more than two frame times, i.e., more than 33.3 milliseconds.

Here, in this disclosure, the input video x,k is the kth video frame of the input x, and the audio x,k is the audio signal contained in the kth video frame of the input x. Also, the output signal video k,k,k,k is the kth output frame synthesized from video 1,k, video 2,k, video 3,k, and video 4,k.

The present disclosure relates to a system that inputs video signals and audio signals constituting multiple asynchronous audio-accompanied videos, and synthesizes those video and audio signals, characterized in that it separates the audio signal from the input signals, synthesizes the video signal separately from the audio signal, and synthesizes and superimposes the audio signal onto the video signal without waiting for the original video signal to be synthesized.

Figure 4 shows an example of synthesis according to the present disclosure. As shown in the figure, audio 1,k, audio 2,k, audio 3,k, and audio 4,k should originally be synthesized as audio into images k,k,k,k, which are images synthesized from the original images. However, the present disclosure separates only the audio signals and synthesizes and superimposes them into the previous image frame as much as possible.

For example, as shown in FIG. 4, if each input frame of the audio signal can be output without delay, the audio video synthesizer outputs the output frame of the audio signal without delay. In this case, if the input time of the input frames of video 1, k and audio 1, k coincides with the output time of the output frames of video k-2, k-2, k-2, k-2, k-2, the audio video synthesizer superimposes audio 1, k on video k-2, k-2, k-2, k-2. If input frames of video 2, k and audio 2, k, video 3, k and audio 3, k, video 4, k and audio 4, k, are input while the output frames of video k-2, k-2, k-2, k-2, k-2 and video k-1, k-1, k-1, k-1 are being output, the audio video synthesizer superimposes audio 2, k, audio 3, k, and audio 4, k on video k-2, k-2, k-2, k-2 and video k-1, k-1, k-1, k-1, k-1. This allows the delay of the audio signal to be significantly reduced. Note that, although the example of Fig. 4 shows an example in which each input frame of the audio signal can be output without delay, the present disclosure is not limited to this, and the audio signal can be output at any timing prior to the output timing of the video signal input at the same time.

For example, the output audio signal at time t1 will be audio 1,k, audio 2,k, video 3,k and audio 3,k, video 4,k and audio 4,k, and each of these can be superimposed on video k-2,k-2,k-2,k-2 as different audio channels, or some or all of the audio signals can be synthesized at any intensity and then superimposed on video k-2,k-2,k-2,k-2 as the same audio channel. By separating the channels, it becomes possible for the audiovisual device that receives the output signal and presents the video and audio to control the audio intensity for each input signal. Furthermore, by synthesizing the channels, it is possible to avoid restrictions on the number of channels of audio signals that can be multiplexed onto a video signal.

In addition, while the example in FIG. 4 shows an example in which each input frame of the audio signal is output without synthesis, the present disclosure is not limited to this, and synthesis processing may also be performed on the audio signal. For example, as shown in FIG. 5, audio 1,k, audio 2,k, audio 3,k, and audio 4,k may be synthesized so as to match the timing of the output frame of audio 4,k that arrives last. This allows the video signal and audio signal to be treated as one frame each.

Figure 6 shows an example of the configuration of a video and audio synthesis device according to this embodiment. The video and audio synthesis device 10 according to this embodiment comprises a separation unit 101, an up/down converter 102, a buffer 103, and a superimposition unit 104. The figure shows four inputs and one output, but any number of inputs and outputs may be used.

Reference numeral 101 denotes a functional unit capable of separating an input signal into a video signal and an audio signal. The separated signals are input to 104 described below, or may be output separately to the outside.
An up-down converter 102 enlarges or reduces the number of pixels to an arbitrary size. For example, the number of pixels of the input 1 is enlarged or reduced so as to match the size of the screen shown in FIG.
A buffer 103 stores each input frame. The input of 102 can be buffered and output in any order.
Reference numeral 104 denotes a pixel synthesis, audio synthesis, and superimposition unit, which reads out the pixels of each input from 103, synthesizes the entire screen, and further synthesizes an audio signal, and then instantly superimposes the video signal as an audio signal regardless of the original video on which the audio signal was superimposed, and outputs the superimposed signal.
The superimposing unit 104 may add an arbitrary control signal to the blanking portion of the screen. In addition, when there is any audio signal that does not require synthesis processing among the audio signals separated by the separating unit 101, the video audio synthesizing device 10 may output the audio signal without going through the superimposing unit 104.

The video and audio synthesis device disclosed herein can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided over a network.

In the above embodiment, an example of four inputs and one screen divided into four is shown, but this disclosure is not limited to this and can be applied to any input.

(Effects of the present disclosure)
By separating the asynchronous video/audio input signals, synthesizing the audio signal without waiting for completion of the synthesis process of the video signal before separation, and superimposing the audio signal on the video signal that has already been synthesized, the delay time until the audio signal is output after synthesis can be shortened.
This will enable collaborative work that requires strict low latency in a system that synthesizes multiple screens at multiple locations.

This disclosure can be applied to the information and communications industry, which distributes video and game content, as well as the film, advertising, and game industries involved in video production.

10: Video and audio synthesizer 21: Scanning line 22: Blanking portion 23: Border portion 24: Display screen 101: Separation section 102: Up/down converter 103: Buffer 104: Superimposition section

Claims (5)

Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos;
Separating a video signal and an audio signal from the plurality of audio-accompanying video signals;
The synthesis of the video signal is performed separately from the processing of the audio signal,
Outputting an audio signal without waiting for completion of the synthesis process of the video signal;
synthesizing the audio signals included in the plurality of audio-accompanied videos in accordance with the last audio signal of the plurality of audio-accompanied videos in one frame;
Audio-visual synthesis device. starting synthesis of an audio signal corresponding to the acquired video signal with a video signal acquired prior to the acquired video signal without waiting for the start of synthesis of a video frame corresponding to the acquired video signal;
2. The video and audio synthesis apparatus according to claim 1. The video and audio synthesis device
Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos;
Separating a video signal and an audio signal from the plurality of audio-accompanying video signals;
The synthesis of the video signal is performed separately from the processing of the audio signal,
Outputting an audio signal without waiting for completion of the synthesis process of the video signal;
synthesizing the audio signals included in the plurality of audio-accompanied videos in accordance with the last audio signal of the plurality of audio-accompanied videos in one frame;
A method for synthesizing audio and video. starting synthesis of an audio signal corresponding to the acquired video signal with a video signal acquired prior to the acquired video signal without waiting for the start of synthesis of a video frame corresponding to the acquired video signal;
The video/audio synthesis method according to claim 3 . 3. A program for causing a computer to realize each of the functional units of the video/audio synthesizing apparatus according to claim 1 .

Images