JP7614749B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to a technology for playing video and audio in sync.

When playing content that includes video data and audio data, it is necessary to synchronize the video and audio because any delay between the video and audio can cause viewers to feel uncomfortable. Patent Document 1 discloses that when using a speech speed conversion function that plays audio at a specified playback speed, the video data corresponding to time information indicating the playback position of the audio is searched for and played, thereby reducing the delay between the video and audio.

JP 2013-5423 A

When playing content that includes video data and audio data at variable speeds, the video and audio playback speeds can be changed to the same speed at the same time, allowing the video and audio to be played back in sync. However, if the playback speed can be changed at different times for the video data and the audio data, the playback speed of one will change later than the other, resulting in a delay between the video and audio.

In view of the above problems, the present invention aims to reduce the misalignment between video and audio when changing the playback speed of content that includes video data and audio data.

In order to solve the above problem, the information processing device according to the present invention has, for example, the following configuration. That is, the information processing device has an acquisition means for acquiring playback information indicating the playback speed of content including video data and audio data and the playback position included in the playback period of the content, a video output means for outputting the video data of a portion of the video data after the frame corresponding to the playback position indicated by the playback information acquired by the acquisition means from among the multiple frames constituting the video data at a playback speed corresponding to the playback speed indicated by the playback information, an audio output means for outputting the audio data of a portion of the audio blocks constituting the audio data after the audio block corresponding to the playback position indicated by the playback information acquired by the acquisition means from among the multiple audio blocks constituting the audio data at a playback speed corresponding to the playback speed indicated by the playback information, a detection means for detecting a time lag between the video data output by the video output means and the audio data output by the audio output means, and a change means for changing the playback speed of the audio data output by the audio output means so that the lag is reduced based on the lag detected by the detection means.

The present invention makes it possible to reduce the misalignment between video and audio when changing the playback speed of content that includes video and audio data.

1 is a block diagram showing a configuration of an information processing device according to an embodiment. 13 is a flowchart showing a process for calculating a time offset. 10 is a flowchart showing a process for controlling a sound reproduction speed. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device.

[System configuration]
1 is a diagram showing an example of the configuration of an information processing device 10 according to this embodiment. The information processing device 10 is a system that outputs stored video data and audio data according to a specified playback speed. The information processing device 10 has a playback information input unit 1, a data read control unit 2, a video data storage unit 3, an audio data storage unit 4, a time lag calculation unit 5, an audio playback control unit 6, a playback speed conversion unit 7, a video time extraction unit 8, and an audio time extraction unit 9.

The playback information input unit 1 outputs playback information related to the playback of content including video data and audio data to the data read control unit 2 and audio playback control unit 6. The playback information includes playback position information indicating the position (time point) at which playback starts within the playback period of the content, and playback speed information indicating the playback speed of the content. The playback position information is information used to determine the read position of the accumulated video data and audio data, and is expressed, for example, by a time code. The playback speed information is information used to determine the playback speed of the video data and audio data, and is expressed, for example, by a playback speed multiplier, such as 1.0 indicating normal speed playback, 2.0 indicating double speed playback, and 0.5 indicating 0.5 times speed playback. The playback information is output from the playback information input unit 1 at the same cycle as the frame rate of the output video. For example, if the frame rate of the output video is 60 fps, the playback information is output at a cycle of 1/60th of a second.

The playback information may include other information in addition to the playback position information and playback speed information. The format of the playback speed information and playback position information is not limited to the above example. The output timing of the playback information is also not limited to the above example. The content of the playback information to be output is determined based on a user operation performed on the information processing device 10. For example, if an operation is performed to change the playback speed of content while the content is being played back, the playback information indicates the playback speed specified by the operation and the playback position according to the timing of the operation. However, the present invention is not limited to this, and the content of the playback information may be determined based on information received by the information processing device 10 from an external device.

The data read control unit 2 outputs playback position information to the video data storage unit 3 and the audio data storage unit 4. The video data storage unit 3 stores video data consisting of multiple frames that make up a video, and each frame is assigned a timestamp indicating the time point corresponding to that frame. The video data storage unit 3 compares the playback position information input from the data read control unit 2 with the timestamp assigned to each frame of the video data, and outputs data of a frame having a timestamp that matches the playback position information. The video data storage unit 3 also outputs the timestamp assigned to the output frame to the video time extraction unit 8. The destination of the video output by the video data storage unit 3 is, for example, a display or an external storage device, but is not limited to these.

The playback position information output from the data read control unit 2 to the video data storage unit 3 is controlled according to the playback speed indicated by the playback speed information. For example, when the playback speed is 2.0x, the playback position information input to the video data storage unit 3 increases by two frames, and when the playback speed is 0.5x, playback position information indicating the same point in time is input twice to the video data storage unit 3. Video frames according to the playback position information input in this way are output from the video data storage unit 3 at a constant frame rate, thereby achieving high-speed and low-speed video playback.

The sound data storage unit 4 stores sound data consisting of a number of sound blocks corresponding to a number of consecutive time periods, and each sound block is assigned a time stamp indicating the time point corresponding to that sound block. The sound data storage unit 4 compares the playback position information input from the data read control unit 2 with the time stamp of each sound block, and outputs the data of the sound block having a time stamp matching the playback position information and that time stamp to the playback speed conversion unit 7. After outputting the sound block corresponding to the first playback position information input from the data read control unit 2, the sound data storage unit 4 outputs sound blocks that are consecutive in time in sequence, regardless of the playback position information input thereafter.

The sound playback control unit 6 determines the sound playback speed based on the playback speed information input from the playback information input unit 1 and the time lag amount input from the time lag calculation unit 5, and outputs the determined sound playback speed to the playback speed conversion unit 7. The playback speed conversion unit 7 converts the playback speed of the sound data input from the sound data storage unit 4 according to the sound playback speed input from the sound playback control unit 6, and outputs the converted sound data and a timestamp. The playback speed conversion unit 7 also outputs a timestamp corresponding to the output sound data to the sound time extraction unit 9. The destination of the sound output by the playback speed conversion unit 7 is, for example, a speaker or an external storage device, but is not limited to these.

The video time extraction unit 8 outputs information on the time indicated by the time stamp of the video data input from the video data storage unit 3 to the time lag calculation unit 5 as video time information. The audio time extraction unit 9 outputs information on the time indicated by the time stamp of the audio data input from the playback speed conversion unit 7 to the time lag calculation unit 5 as audio time information. The time lag calculation unit 5 calculates the amount of time lag between video and audio based on the video time information input from the video time extraction unit 8 and the audio time information input from the audio time extraction unit 9, and outputs the calculated amount of time lag to the audio playback control unit 6.

In this embodiment, the period length corresponding to an audio block does not match the period length corresponding to one frame of video data. Therefore, when the playback speed information output from the playback information input unit 1 changes, the timing at which the playback speed is reflected in the video data and the timing at which it is reflected in the audio data may differ, resulting in a misalignment between the video and audio.

For example, consider a case where an instruction to change the playback speed is given while content is being played back. Here, in the content playback information output from the playback information input unit 1, the playback speed information from playback position 01:00:00:00 to playback position 01:00:10:10 indicates a playback speed of 1.0. And the playback speed information from playback position 01:00:10:11 to playback position 01:00:20:10 indicates a playback speed of 2.0. The timestamp of the video data is assigned in the same format as the playback position information. In this case, the video data storage unit 3 outputs the video data up to playback position 01:00:10:10 one frame at a time, and outputs the video data from the frame corresponding to playback position 01:00:10:11 onwards two frames at a time.

However, the period length corresponding to the audio block does not match the time unit of the playback position information, and there is no audio block that starts at the same time as playback position 01:00:10:11. Therefore, of the audio blocks whose starting time is after the time corresponding to playback position 01:00:10:11, the audio data from the audio block with the earliest starting time onwards is played at double speed. And if the difference between the time corresponding to playback position 01:00:10:11, where the video becomes double speed, and the start time of the first audio block played at double speed is t seconds, then the sound will lag behind the video by t seconds.

The information processing device 10 therefore detects such a time lag between the video data and the audio data using the time lag calculation unit 5, and changes the playback speed of the audio data using the audio playback control unit 6 so as to reduce the lag, thereby suppressing the lag between the video and audio. Note that even if the period length corresponding to an audio block matches the period length corresponding to one frame of video data, if there is a lag between the start point of the audio block and the time corresponding to the video frame, a time lag between the video data and the audio data may occur as in the above example. Even in such cases, the lag between the video and audio can be suppressed by applying this embodiment.

Each of the components of the information processing device 10 shown in FIG. 1 can be realized by dedicated hardware. However, this is not limited to the above, and some or all of the components of the information processing device 10 may be configured by software. An example of the hardware configuration of the information processing device in this case is shown in FIG. 4.

The CPU 501 controls the entire information processing device 10 using computer programs and data stored in the RAM 502 and ROM 503, and executes processing corresponding to each component shown in FIG. 1. The RAM 502 has an area for temporarily storing computer programs and data loaded from the external storage device 504, and data acquired from the outside via the I/F 507. The RAM 502 also has a work area used by the CPU 501 when executing various processes. That is, the RAM 502 can be allocated as a frame memory, or can provide various other areas as appropriate. The ROM 503 stores computer setting data, boot programs, and the like. The operation unit 506 is composed of a keyboard, a mouse, and the like, and can input various instructions to the CPU 501 according to user operations. The output unit 505 outputs the processing results by the CPU 501. The output unit 505 is composed of, for example, a liquid crystal display and a speaker.

The external storage device 504 is a large-capacity information storage device composed of a hard disk drive device or the like. The external storage device 504 stores an OS (operating system) and computer programs for causing the CPU 501 to realize the functions of each unit shown in FIG. 1. The external storage device 504 may also store video data and audio data to be processed. The computer programs and data stored in the external storage device 504 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and become the subject of processing by the CPU 501. The I/F 507 can be connected to a network such as a LAN or the Internet, and other devices such as a projector or display device, and the information processing device 10 can obtain and send various information via this I/F 507. The bus 508 connects the above-mentioned processing units constituting the information processing device 10.

As described above, the functions of the information processing device 10 can be realized mainly by the CPU 501. Note that the information processing device 10 may have one or more dedicated hardware components different from the CPU 501, and at least a portion of the processing by the CPU 501 may be executed by the dedicated hardware components. Examples of such dedicated hardware components include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[Operation flow]
In the following, the flow of the process of calculating the time lag between the sound and the video after the playback speed conversion and controlling the sound playback speed based on the calculated time lag will be described with reference to Figs. 2 and 3. Fig. 2 is a flowchart showing the process of the time lag calculation unit 5 calculating the time lag between the sound and the video. The process shown in Fig. 2 is realized by the CPU 501 expanding a program stored in the ROM 503 into the RAM 502 and executing it. Note that at least a part of the process shown in Fig. 2 may be realized by one or more dedicated hardware different from the CPU 501. The same applies to the process of the flowchart shown in Fig. 3. The process shown in Fig. 2 is started at the timing when an instruction to start outputting the content including the video data and the audio data is input to the information processing device 10. However, the start timing of the process shown in Fig. 2 is not limited to this.

In S201, the video time information and audio time information stored in the time lag calculation unit 5 are initialized to invalid values. In S202, the time lag calculation unit 5 determines whether valid video time information has been received from the video time extraction unit 8. If valid video time information has been received, the process proceeds to S203; if not, the process proceeds to S207. In S203, the received video time information is stored.

In S207, the time lag calculation unit 5 determines whether valid acoustic time information has been received from the acoustic time extraction unit 9. If valid acoustic time information has been received, the process proceeds to S208, and if not, the process proceeds to S209. In S208, the received acoustic time information is stored. In S209, it is determined whether an end instruction has been input. If an end instruction has been input, the process ends, and if not, the process returns to S202.

In S204, it is determined whether the stored video time information and audio time information are valid or invalid. If both pieces of information are valid, the process proceeds to S205, and if either is invalid, the process returns to S202. In S205, the amount of time lag between the point indicated by the stored video time information and the point indicated by the audio time information is calculated. Specifically, the amount of time lag is calculated by subtracting the value indicated by the video time information from the value of the audio time information. Note that the larger the value indicated by these pieces of time information, the later the point in time it corresponds to. In S206, the calculated amount of time lag is output to the audio playback control unit 6.

Figure 3 is a flowchart showing the process in which the sound playback control unit 6 controls the playback speed of sound data. The process shown in Figure 3 is started when an instruction to start outputting content including video data and sound data is input to the information processing device 10. However, the start timing of the process shown in Figure 3 is not limited to this.

In S301, the playback speed information and time lag amount stored in the sound playback control unit 6 are initialized to invalid values, and the speed offset is initialized to 0. In S302, the sound playback control unit 6 determines whether it has received playback speed information from the playback information input unit 1. If valid playback speed information has been received, the process proceeds to S303, and if not, the process proceeds to S305. In S303, the sound playback speed is calculated based on the playback speed information and the speed offset. The speed offset will be described later. In S304, the calculated sound playback speed is output to the playback speed conversion unit 7.

In S305, the sound playback control unit 6 determines whether it has received a time lag from the time lag calculation unit 5. If a valid time lag has been received, the process proceeds to S306; if not, the process proceeds to S308. In S306, it is determined whether the playback position of the sound data is ahead of the playback position of the video data. Specifically, the time lag is compared with a threshold value, and if the time lag is greater than the threshold value (if the sound is leading the video significantly), the process proceeds to S307; if the time lag is less than the threshold value, the process proceeds to S309.

The threshold value used in S306 is 0 or a positive value, and is set in advance in the information processing device 10. For example, this threshold value may be set to 27 ms, which is the amount of delay at which a human can recognize a delay between the image and the sound. However, the threshold value is not limited to this, and may be set automatically depending on the content of the image data and audio data.

In S307, the speed offset is set to a negative value in order to reduce the delay between the video and the sound by slowing down the playback speed of the audio data. The speed offset is a value indicating the amount of change related to the adjustment of the audio playback speed, and in S303, the value indicated by the playback speed information plus the speed offset is calculated as the audio playback speed. In other words, if the speed offset is positive, the audio playback speed is faster than the playback speed indicated by the playback speed information, and if the speed offset is negative, the audio playback speed is slower than the playback speed indicated by the playback speed information. In S307, the speed offset may be set to a predetermined value, or may be set so that the absolute value of the speed offset increases as the absolute value of the time lag increases.

In S309, it is determined whether the playback position of the audio data lags behind the playback position of the video data. Specifically, the amount of time lag is compared with a threshold, and if the amount of time lag is smaller than the threshold (if the delay of the audio relative to the video is large), the process proceeds to S310, and if the amount of time lag is larger than the threshold, the process proceeds to S311. The threshold used in S309 is 0 or a negative value, and is set in advance in the information processing device 10. For example, this threshold may be set to -60 ms, which is the amount of lag that allows humans to recognize the delay of the audio relative to the video. However, the threshold is not limited to this, and the threshold may be set automatically depending on the content of the video data and audio data.

In S310, the speed offset is set to a positive value in order to reduce the delay of the sound relative to the video by increasing the playback speed of the audio data. In S310, the speed offset may be set to a predetermined value, or may be set so that the absolute value of the speed offset increases as the absolute value of the time lag increases. On the other hand, in S311, since the sound and video are sufficiently synchronized, the speed offset is set to 0. In S308, it is determined whether an end command has been input. If an end command has been input, the process ends, and if not, the process returns to S302.

As described above, by adjusting the playback speed of the audio data using a speed offset that corresponds to the magnitude of the delay between the video and audio, the delay between the video and audio can be reduced. This reduces the sense of discomfort felt by users watching the content.

In the above-described embodiment, the time lag is calculated by comparing video time information corresponding to the timestamp of the video data with audio time information corresponding to the timestamp of the audio data, and the audio playback speed is controlled based on the calculated time lag. However, the method of calculating the time lag for adjusting the audio playback speed is not limited to this. For example, if the playback position information output from the playback information input unit 1 has the same format as the timestamp of the video data, the time lag calculation unit 5 can calculate the amount of lag between the playback position indicated by the playback information and the time point indicated by the audio time information as the amount of time lag between the video and the audio. In this case, it is sufficient for the playback information input unit 1 to output the playback information to the time lag calculation unit 5, and the information processing device 10 does not need to have a video time extraction unit 8. With such a configuration, the cost of the video time information extraction unit can be reduced.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The program may also be provided by recording it on a computer-readable storage medium.

10 Information processing device

Claims (15)

an acquisition means for acquiring playback information indicating a playback speed of a content including video data and audio data and a playback position included in a playback period of the content;
a video output means for outputting the video data of a portion subsequent to the frame corresponding to the playback position indicated by the playback information acquired by the acquisition means, among the plurality of frames constituting the video data, at a playback speed corresponding to the playback speed indicated by the playback information;
a sound output means for outputting the sound data of a portion subsequent to the sound block corresponding to the playback position indicated by the playback information acquired by the acquisition means, among the plurality of sound blocks constituting the sound data, at a playback speed corresponding to the playback speed indicated by the playback information;
a detection means for detecting a time lag between the video data output by the video output means and the audio data output by the audio output means;
a change means for changing a reproduction speed of the sound data output by the sound output means based on the deviation detected by the detection means so as to reduce the deviation. The information processing device according to claim 1, characterized in that the period length corresponding to the frame of the video data and the period length corresponding to the audio block of the audio data are different. the acquiring means acquires the playback information based on an instruction to change a playback speed of the content during playback of the content;
the video output means changes a playback speed of the video data to be output in accordance with a playback speed indicated by the playback information, from a frame corresponding to a playback position indicated by the playback information;
3. The information processing apparatus according to claim 1, wherein the sound output means changes a playback speed of the sound data to be output in accordance with a playback speed indicated by the playback information from a sound block corresponding to a playback position indicated by the playback information. the acquiring means acquires the playback information based on an operation for instructing a change in a playback speed of the content, the operation being performed during playback of the content;
4. The information processing apparatus according to claim 1, wherein the reproduction information acquired by the acquisition means indicates a reproduction speed designated in response to the operation and a reproduction position in response to a timing of the operation. The information processing device according to any one of claims 1 to 4, characterized in that the playback information indicates a playback position in a format similar to the timestamps of frames constituting the video data. The information processing device according to any one of claims 1 to 5, characterized in that the audio block corresponding to the playback position indicated by the playback information is the audio block with the earliest start time among the audio blocks whose start times are at or after the time corresponding to the playback position. 7. The information processing device according to claim 1 , wherein the detection means detects the deviation based on time information contained in the video data output from the video output means and time information contained in the audio data output from the audio output means. The information processing device according to any one of claims 1 to 6, characterized in that the detection means detects a time difference between a time point corresponding to the playback position indicated by the playback information acquired by the acquisition means and a time point indicated by time information of the audio data output from the audio output means as a time difference between the video data and the audio data. The information processing device according to any one of claims 1 to 8, characterized in that the change means determines the amount of change in the playback speed of the sound data based on the magnitude of the deviation detected by the detection means. The information processing device according to any one of claims 1 to 9, characterized in that the change means changes the playback speed of the sound data when the magnitude of the deviation detected by the detection means is greater than a threshold value. The information processing device according to claim 10, characterized in that the value used as the threshold when the video data output by the video output means is delayed from the audio data output by the audio output means is different from the value used as the threshold when the audio data output by the audio output means is delayed from the video data output by the video output means. an acquisition step of acquiring playback information indicating a playback speed of a content including video data and audio data and a playback position included in a playback period of the content;
a video output step of outputting, from among a plurality of frames constituting the video data, a portion of the video data subsequent to the frame corresponding to the playback position indicated by the playback information acquired in the acquisition step, at a playback speed corresponding to the playback speed indicated by the playback information;
a sound output step of outputting the sound data of a portion subsequent to the sound block corresponding to the playback position indicated by the playback information acquired in the acquisition step among the plurality of sound blocks constituting the sound data, at a playback speed corresponding to the playback speed indicated by the playback information;
a detection step of detecting a time lag between the video data output in the video output step and the audio data output in the audio output step;
a change step of changing a playback speed of the acoustic data outputted in the acoustic output step so as to reduce the deviation based on the deviation detected in the detection step. The information processing method according to claim 12, characterized in that the period length corresponding to the frame of the video data and the period length corresponding to the audio block of the audio data are different. In the acquiring step, the playback information is acquired based on an instruction to change a playback speed of the content during playback of the content,
a playback speed of the video data output in the video output step is changed according to a playback speed indicated by the playback information from a frame corresponding to a playback position indicated by the playback information;
14. The information processing method according to claim 12, wherein the playback speed of the sound data output in the sound output step is changed according to the playback speed indicated by the playback information from a sound block corresponding to the playback position indicated by the playback information. A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 11.

Images