JP7651835B2 - Information processing method, information processing system, and program - Google Patents

Description

This disclosure relates to technology for generating content that includes video and audio.

Various techniques have been proposed for creating content that includes video and sound. For example, Patent Document 1 discloses a technique for generating a slideshow video in which the video switches when the melody of the music changes.

JP 2007-188561 A

In the technology of Patent Document 1, the video is forcibly switched when the melody of the music changes, making it difficult to actually create content that includes the video intended by the creator. In consideration of the above circumstances, one aspect of the present disclosure aims to generate content that has a sense of unity between the video and the sound while suppressing the impact on the video represented by the video data.

In order to solve the above problems, an information processing method according to one aspect of the present disclosure is an information processing method for processing video data including a plurality of video segments and sound data including a plurality of sound segments, and processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment. An information processing method according to another aspect of the present disclosure is an information processing method for processing video data including a plurality of video segments and sound data representing sound, and processes the sound data so as to reduce the volume within a transition period including a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment.

An information processing system according to one aspect of the present disclosure is an information processing system that processes video data including a plurality of video segments and sound data including a plurality of sound segments, and includes a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment. An information processing system according to another aspect of the present disclosure is an information processing system that processes video data including a plurality of video segments and sound data representing sound, and includes a sound data processing unit that processes the sound data so as to reduce the volume within a transition period that includes a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment.

A program according to one aspect of the present disclosure is a program for processing video data including multiple video segments and sound data including multiple sound segments, and causes a computer to function as a sound data processing unit that processes the sound data so as to switch from a first sound segment among the multiple sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the multiple video segments and a second video segment subsequent to the first video segment. A program according to another aspect of the present disclosure is a program for processing video data including multiple video segments and sound data representing sound, and causes a computer to function as a sound data processing unit that processes the sound data so as to reduce the volume within a transition period that includes a boundary point between a first video segment among the multiple video segments and a second video segment subsequent to the first video segment.

1 is a block diagram illustrating a configuration of an information system according to a first embodiment. FIG. 1 is a block diagram illustrating a configuration of an editing system. FIG. 2 is a block diagram illustrating an example of a functional configuration of the editing system. 4 is a flowchart of an operation executed by a control device of the editing system. FIG. 4 is an explanatory diagram of an editing process in the first embodiment. 4 is a flowchart of an editing process in the first embodiment. FIG. 11 is an explanatory diagram of an editing process in the second embodiment. 13 is a flowchart of an editing process according to a second embodiment. FIG. 13 is an explanatory diagram of an editing process in the third embodiment. 13 is a flowchart of an editing process according to a third embodiment. FIG. 13 is an explanatory diagram of an editing process in the fourth embodiment. FIG. 13 is a block diagram illustrating a configuration of a terminal device in a fifth embodiment. FIG. 13 is a block diagram illustrating a functional configuration of a terminal device according to a fifth embodiment.

A: First embodiment Fig. 1 is a block diagram illustrating a configuration of an information system 100 according to a first embodiment. The information system 100 according to the first embodiment includes a terminal device 10 and an editing system 20. The terminal device 10 and the editing system 20 communicate with each other via a communication network 30 such as the Internet.

The terminal device 10 is an information terminal such as a mobile phone, a smartphone, a tablet terminal, or a personal computer. The terminal device 10 transmits material data D to the editing system 20. The material data D includes video data X1 and sound data Y1. The video data X1 is data representing a video. For example, the video data X1 is generated by an imaging device mounted on the terminal device 10. The sound data Y1 is data representing a sound to be played in parallel to the video of the video data X1. Specifically, the sound data Y1 in the first embodiment is data representing the performance sound (instrument sound or singing sound) of a piece of music played as background music for the video of the video data X1.

The editing system 20 is a computer system that generates content C using material data D received from the terminal device 10. Content C is video content including video data X2 and sound data Y2. Video data X2 is generated by editing video data X1. Sound data Y2 is generated by editing sound data Y1. In other words, material data D is data that becomes the material of content C. The editing system 20 transmits content C to the terminal device 10. The terminal device 10 plays content C received from the editing system 20. In other words, the video represented by video data X2 and the sound represented by sound data Y2 (specifically, the sound of a musical piece being played) are played in parallel.

Figure 2 is a block diagram illustrating the configuration of an editing system 20. The editing system 20 includes a control device 21, a storage device 22, and a communication device 23. Note that the editing system 20 may be realized by a single device, or may be realized by multiple devices configured separately from each other.

The control device 21 is a single or multiple processors that control each element of the editing system 20. Specifically, the control device 21 is configured with one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

The storage device 22 is a single or multiple memories that store the programs executed by the control device 21 and various data used by the control device 21. The storage device 22 is composed of a known recording medium, such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the editing system 20, or a recording medium (e.g., cloud storage) to which the control device 21 can write or read via the communication network 30 may be used as the storage device 22.

The communication device 23 communicates with the terminal device 10 via the communication network 30. Specifically, the communication device 23 receives material data D transmitted from the terminal device 10. The communication device 23 also transmits content C generated from the material data D to the terminal device 10.

Figure 3 is a block diagram illustrating an example of the functional configuration of the editing system 20. The control device 21 of the editing system 20 executes a program stored in the storage device 22 to realize multiple functions (a material data acquisition unit 51, a video data processing unit 52, an audio data processing unit 53, and a content providing unit 54) for generating and providing content C from material data D. The material data acquisition unit 51 acquires material data D transmitted from the terminal device 10 via the communication device 23.

The video data processing unit 52 generates video data X2 from video data X1 of the material data D. Specifically, the video data processing unit 52 generates video data X2 by executing image processing on the video data X1. The image processing includes various processes such as extracting a specific section of a video or adjusting image quality. Note that the video data X1 and video data X2 may be in any format.

The sound data processing unit 53 generates sound data Y2 from sound data Y1 of the material data D. In the first embodiment, the sound data Y1 and sound data Y2 are composed of a time series of samples representing sound waveforms, for example. Content C is composed of the video data X2 generated by the video data processing unit 52 and the sound data Y2 generated by the sound data processing unit 53. In other words, the video data processing unit 52 and the sound data processing unit 53 function as elements that generate content C from material data D. The content providing unit 54 transmits content C including the video data X2 and sound data Y2 from the communication device 23 to the terminal device 10.

Figure 4 is a flow chart illustrating the specific steps of the operation executed by the control device 21. The process in Figure 4 is started in response to an instruction from the user of the terminal device 10. When the process is started, the control device 21 (material data acquisition unit 51) receives material data D transmitted from the terminal device 10 via the communication device 23 (Sa). The video data processing unit 52 generates video data X2 from video data X1 of the material data D (Sb). The sound data processing unit 53 generates sound data Y2 from sound data Y1 of the material data D (Sc: editing process). The content providing unit 54 transmits content C including the video data X2 and sound data Y2 from the communication device 23 to the terminal device 10 (Sd).

Figure 5 is an explanatory diagram of the editing process Sc. The video data X2 includes multiple (M) video sections V1 to VM arranged on a time axis (M is a natural number equal to or greater than 2). Each video section Vm (m = 1 to M) is a period in which the video represented by the video data X2 is divided on the time axis by scene. The M video sections V1 to VM are continuous on the time axis without any gaps between them, and form a series of stories related to one theme. For example, a series of stories with a sense of conceptual unity, such as a video introducing a company or a video introducing products handled by the company, are composed of the M video sections V1 to VM. Any one of the M video sections V1 to VM is, for example, a section corresponding to a scene that has a semantic unity in the video, or a section captured in one imaging operation by an imaging device. However, it is also possible that one video section Vm includes multiple scenes, or that one video section Vm includes a period captured in multiple imaging operations. Note that the M video sections V1 to VM do not necessarily need to be temporally or conceptually continuous. For example, it is possible for each video section Vm to occur before and after another video section Vm with an interval between them, or for two or more video sections Vm to have different themes. Also, it is not essential that each video section Vm has a semantic unity.

Each video section Vm has a variable length. The duration of each video section Vm is basically different. However, the duration may be the same across any two or more video sections Vm. In video data X2, a time point Pm (hereinafter referred to as the "boundary point") at the boundary between video sections Vm and Vm+1, which are adjacent to each other on the time axis, is specified. The boundary point Pm can also be said to be the end point (specifically, the endpoint) of the video section Vm.

The sound data Y1 includes multiple (M) sound intervals A1 to AM arranged on the time axis. Each video interval Vm of the video data X2 corresponds to each sound interval Am of the sound data Y1. Specifically, the material data acquisition unit 51 acquires material data D from the terminal device 10, in which each video interval Vm of the video data X1 corresponds to each sound interval Am of the sound data Y1. For example, the sound interval Am corresponding to each video interval Vm is selected in response to an instruction from the user of the terminal device 10. Note that the sound data Y1 is a single piece of data that is continuous across M sound intervals A1 to AM, or a collection of multiple data that correspond to different sound intervals Am.

Each sound interval Am is a structural interval that divides the song represented by the sound data Y1 on the time axis according to its musical meaning. For example, each sound interval Am corresponds to a period such as the intro, verse, bridge, chorus, and outro. Data that specifies the endpoints (start or end points) of each sound interval Am is set in the sound data Y1. The data that specifies the endpoints of each sound interval Am corresponds to a rehearsal mark that indicates the start point of each structural interval. The duration of each sound interval Am exceeds the duration of the video interval Vm that corresponds to that sound interval Am.

The sound data processing unit 53 processes the sound data Y1 to generate the sound data Y2 so that the sound section Am corresponding to the video section Vm starts in each of the M video sections V1 to VM of the video data X2. That is, the sound data processing unit 53 generates the sound data Y2 from the sound data Y1 so that at the boundary point Pm between the video section Vm and the immediately succeeding video section Vm+1 among the M video sections V1 to VM, the sound data Y2 is generated from the sound data Y1 so that the sound section Am of the M sound sections A1 to AM is switched to the immediately succeeding sound section Am+1. For example, at the boundary point P1 between the video sections V1 and V2, the sound section A1 is switched to the sound section A2, and at the boundary point P2 between the video sections V2 and V3, the sound section A2 is switched to the sound section A3. Specifically, the sound data processing unit 53 of the first embodiment generates the sound data Y2 by processing the sound data Y1 so that the sound section Am is switched from the midpoint that coincides with the boundary point Pm to the immediately succeeding sound section Am+1.

Specifically, the sound data processing unit 53 of the first embodiment extracts a portion of a period (hereinafter referred to as a "specific section") Bm including the start point of a sound section Am of sound data Y1, and generates sound data Y2 by linking M specific sections B1 to BM corresponding to different sound sections Am in chronological order. The specific section Bm is a section of the sound section Am that extends from the start point of the sound section Am to the duration of the video section Vm. A portion of the sound section Am including the end point (sections other than the specific section Bm) is removed. As can be understood from the above explanation, the end section of the sound section Am is removed so that the duration of each sound section Am matches the duration of the video section Vm.

Figure 6 is a flowchart illustrating the specific steps of the editing process Sc. The editing process Sc is started when the video data processing unit 52 generates video data X2.

When the editing process Sc is started, the sound data processing unit 53 selects one video section Vm from the M video sections V1 to VM in the video data X2 (Sc11). Each video section Vm is selected sequentially in chronological order. The sound data processing unit 53 selects one sound section Am corresponding to the video section Vm from the M sound sections A1 to AM in the sound data Y1 (Sc12). The sound data processing unit 53 extracts a specific section Bm from the start point of the selected sound section Am and spanning the duration of the video section Vm (Sc13: extraction process). The sound data processing unit 53 concatenates the specific section Bm extracted in the extraction process Sc13 to the end of the specific section Bm-1 extracted in the previous extraction process Sc13 (Sc14: concatenation process). Note that the specific section B1 extracted from the first sound section A1 is placed at the beginning of the sound data Y2.

The sound data processing unit 53 determines whether the above processes (Sc11-Sc14) have been performed for all of the M sound sections A1-AM (Sc15). If an unprocessed sound section Am remains (Sc15: NO), the sound data processing unit 53 selects the moving image section Vm+1 immediately following the currently selected moving image section Vm among the M moving image sections V1-VM of the moving image data X2 as the moving image section Vm to be processed (Sc11), and performs selection of the sound section Am (Sc12), extraction process Sc13, and linking process Sc14 for the updated moving image section Vm. On the other hand, if all of the M sound sections A1-AM have been processed (Sc15: YES), the sound data processing unit 53 ends the editing process Sc. As can be understood from the above explanation, the moving image data X2 is not edited in the editing process Sc.

As shown in the above example, the sound data processing unit 53 processes the sound data Y1 so that the sound interval Am switches to the sound interval Am+1 at the boundary point Pm between the video interval Vm and the subsequent video interval Vm+1. Therefore, content C is generated in which each video interval Vm and each sound interval Am are parallel to each other and the sound interval Am starts at the start point of the video interval Vm. That is, at the boundary point Pm where the video played by the terminal device 10 transitions from the video interval Vm to the video interval Vm+1, the sound played by the terminal device 10 switches to the sound interval Am+1 from the middle of the sound interval Am (the end point of the specific interval Bm). On the other hand, the duration of each video interval Vm specified by the video data X2 is not changed. As can be understood from the above explanation, according to the first embodiment, it is possible to generate content C that has a sense of unity between the change in the video and the change in the sound while suppressing the influence on the video represented by the video data X2.

In addition, in the first embodiment, by a simple process of starting a sound section Am+1 at the boundary point Pm of each video section Vm, it is possible to cause switching from a sound section Am to an audio section Am+1 at the boundary point Pm between the video sections Vm and Vm+1. Note that the video section Vm is an example of a "first video section", and the video section Vm+1 is an example of a "second video section". Also, the sound section Am is an example of a "first sound section", and the immediately following sound section Am+1 is an example of a "second sound section".

B: Second embodiment A second embodiment will be described. Note that, in each of the following exemplary embodiments, for elements whose functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used and detailed descriptions of each will be omitted as appropriate.

Figure 7 is an explanatory diagram of the editing process Sc in the second embodiment. In the first embodiment, a portion of each sound interval Am, including the end point, was deleted to make the endpoints of each sound interval Am coincide with the endpoints of each video interval Vm on the time axis. The sound data processing unit 53 in the second embodiment shortens or extends each sound interval Am on the time axis to make the endpoints of each sound interval Am coincide with the endpoints of each video interval Vm on the time axis. Specifically, the sound data processing unit 53 shortens or extends each sound interval Am to the time length of the video interval Vm by adjusting the progression speed (e.g., tempo) of the sound interval Am.

Figure 8 is a flowchart illustrating the specific steps of the editing process Sc in the second embodiment. When the editing process Sc is started, the sound data processing unit 53 selects the video section Vm (Sc21) and the sound section Am (Sc22) in the same manner as in the first embodiment.

The sound data processing unit 53 of the second embodiment executes a stretching process Sc23 instead of the extraction process Sc13 of the first embodiment. The stretching process Sc23 is a process for shortening or extending the sound section Am so that the time length of the sound section Am matches the time length of the video section Vm. Any known stretching technique can be used for the stretching process Sc23. The sound data processing unit 53 connects the sound section Am stretched or shortened by the stretching process Sc23 to the end of the sound section Am-1 stretched or shortened by the previous stretching process Sc23 (Sc24). The first sound section Am is placed at the beginning of the sound data Y2. The above process is repeated for all M sound sections A1 to AM (Sc25). As in the first embodiment, the video data X2 is not edited in the editing process Sc of the second embodiment.

The editing process Sc illustrated above generates content C in which each video section Vm and each sound section Am run in parallel, and in which the sound section Am starts at the start point of the video section Vm. Therefore, as in the first embodiment, content C can be generated that has a sense of unity between changes in the video and changes in the sound, while suppressing the impact on the video represented by the video data X2. Furthermore, in the second embodiment, each sound section Am is expanded or contracted, reducing the possibility that each sound section Am will be interrupted discontinuously at some point along the way. Furthermore, when the duration of the sound section Am is shorter than the duration of the video section Vm, extending the sound section Am reduces the possibility that the sound section Am will be insufficient for the video section Vm.

The sound data processing unit 53 in the first and second embodiments is collectively represented as an element that processes sound data Y1 so as to switch from sound interval Am to sound interval Am+1 at boundary point Pm between video interval Vm and video interval Vm+1.

C: Third embodiment Fig. 9 is an explanatory diagram of the editing process Sc executed by the sound data processing unit 53 of the third embodiment. In the third embodiment, it is assumed that the duration of each sound section Am of the sound represented by the sound data Y1 is set to the same duration as each moving image section Vm of the moving image represented by the moving image data X2. Note that in the third and fourth embodiments, it is not essential to distinguish between each sound section Am.

The transition period Q in FIG. 9 is a period corresponding to the boundary point Pm between adjacent video sections Vm and Vm+1 among the M video sections V1 to VM of the video represented by the video data X2. Specifically, the transition period Q is a period that includes the boundary point Pm. In the first embodiment, the transition period Q is exemplified as a period whose end point is the boundary point Pm. The transition period Q is set to a predetermined time length. However, the time length of the transition period Q may be variable, for example, according to an instruction from the user of the terminal device 10.

The sound data processing unit 53 of the third embodiment processes the sound data Y1 so that the volume is lower within the transition period Q than outside the transition period Q, thereby generating sound data Y2. Specifically, the sound data processing unit 53 processes the sound data Y1 so that the volume decreases from the start point q1 to the end point q2 of the transition period Q, and increases at the end point q2 of the transition period Q. For example, the sound data processing unit 53 generates sound data Y2 by multiplying each sample constituting the sound data Y1 by an adjustment value G. The sound data processing unit 53 changes the adjustment value G over time within the range between the reference value gH and the minimum value gL. The reference value gH is a numerical value that exceeds the minimum value gL. For example, the reference value gH is set to 1, and the minimum value gL is set to 0.

First, the sound data processing unit 53 of the third embodiment maintains the adjustment value G at the reference value gH outside the transition period Q. Second, the sound data processing unit 53 decreases the adjustment value G over time from the reference value gH to the minimum value gL from the start point q1 to the end point q2 of the transition period Q. Within the transition period Q, the adjustment value G decreases linearly, for example, at a predetermined rate of change. However, the adjustment value G may change curvilinearly, for example, at a variable rate of change. Third, the sound data processing unit 53 increases the adjustment value G from the minimum value gL to the reference value gH at the end point q2 of the transition period Q.

Figure 10 is a flow chart illustrating the specific steps of the editing process Sc in the third embodiment. When the editing process Sc is started, the sound data processing unit 53 sets multiple transition periods Q corresponding to different boundary points Pm on the time axis by referring to the video data X2 (Sc31). The sound data processing unit 53 selects one of the multiple transition periods Q (Sc32).

The sound data processing unit 53 adjusts the volume of the sound data Y1 so that the volume decreases during the selected transition period Q (Sc33: adjustment process). Specifically, the sound data processing unit 53 decreases the adjustment value G over time from the reference value gH to the minimum value gL from the start point q1 to the end point q2 of the transition period Q, and increases the adjustment value G from the minimum value gL to the reference value gH at the end point q2.

The sound data processing unit 53 determines whether or not the adjustment process Sc33 has been performed for all of the multiple transition periods Q (Sc34). If an unprocessed transition period Q remains (Sc34: NO), the sound data processing unit 53 selects the transition period immediately following the currently selected transition period from among the multiple transition periods Q (Sc32) and performs the adjustment process Sc33 for the updated transition period Q. On the other hand, if the adjustment process Sc33 has been performed for all of the multiple transition periods Q (Sc34: YES), the sound data processing unit 53 ends the editing process Sc. As can be understood from the above explanation, the video data X2 is not edited in the editing process Sc.

As described above, in the third embodiment, the sound data Y1 is processed so that the volume decreases within the transition period Q corresponding to the boundary point Pm between the video section Vm and the video section Vm+1. Therefore, as in the first embodiment, it is possible to generate content C that has a sense of unity between changes in the video and changes in the sound while suppressing the effect on the video represented by the video data X2.

In particular, in the third embodiment, the volume decreases from the start point q1 to the end point q2 (boundary point Pm) of the transition period Q, and increases at the end point q2 (the start point of the video section Vm) of the transition period Q. Therefore, it is possible to generate content C in which the volume decreases over time toward the end point of the video section Vm, and sound is played at a sufficient volume at the start of the video section Vm+1.

D: Fourth embodiment Fig. 11 is an explanatory diagram of an editing process Sc in the fourth embodiment. In the fourth embodiment, as in the third embodiment, it is assumed that the time length of each sound section Am of the sound represented by the sound data Y1 is set to be equal to the time length of each video section Vm of the video represented by the video data X2. In the fourth embodiment, as in the third embodiment, a plurality of transition periods Q including different boundary points Pm of the video represented by the video data X2 are set. Each transition period Q is a period having a boundary point Pm as an end point.

The sound data processing unit 53 of the fourth embodiment generates sound data Y2 by processing sound data Y1 so that the volume is lower within the transition period Q than outside the transition period Q. Specifically, the sound data processing unit 53 processes the sound data Y1 so that the volume is lower at the start point q1 of the transition period Q and is increased from the start point q1 to the end point q2 of the transition period Q. For example, the sound data processing unit 53 changes the adjustment value G, which is multiplied by each sample of the sound data Y1, over time within the range between the reference value gH and the minimum value gL, as in the third embodiment.

First, the sound data processing unit 53 of the fourth embodiment maintains the adjustment value G at the reference value gH outside the transition period Q. Second, the sound data processing unit 53 decreases the adjustment value G from the reference value gH to the minimum value gL at the start point q1 of the transition period Q. Third, the sound data processing unit 53 increases the adjustment value G over time from the minimum value gL to the reference value gH from the start point q1 to the end point q2 of the transition period Q. Within the transition period Q, the adjustment value G increases linearly, for example, at a predetermined rate of change. However, the adjustment value G may change curvilinearly, for example, at a variable rate of change.

The operations of the editing process Sc in the fourth embodiment, other than the adjustment process Sc33, are the same as those in the third embodiment. In the adjustment process Sc33 in the fourth embodiment, the sound data processing unit 53 adjusts the volume of the sound data Y1 so that the volume decreases within the transition period Q. Specifically, the sound data processing unit 53 in the fourth embodiment decreases the adjustment value G from the reference value gH to the minimum value gL at the start point q1 of the transition period Q, and increases the adjustment value G over time from the minimum value gL to the reference value gH from the start point q1 to the end point q2 of the transition period Q.

As described above, in the fourth embodiment, the sound data Y1 is processed so that the volume decreases within the transition period Q that includes the boundary point Pm between the video section Vm and the video section Vm+1. Therefore, as in the third embodiment, it is possible to generate content C that has a sense of unity between changes in the video and changes in the sound while suppressing the effect on the video represented by the video data X2.

In particular, in the fourth embodiment, the volume decreases at the start point q1 of the transition period Q, and increases from the start point q1 to the end point q2 (boundary point Pm) of the transition period Q. Therefore, it is possible to generate content C in which the volume increases over time as the video section Vm switches to the video section Vm+1.

E: Fifth embodiment Fig. 12 is a block diagram illustrating a configuration of a terminal device 10 in a fifth embodiment. In the first to fourth embodiments, the editing system 20 generates the content C from the material data D. In the fifth embodiment, the terminal device 10 generates the content C from the material data D. In the fifth embodiment, the editing system 20 is omitted.

The terminal device 10 includes a control device 11, a storage device 12, and a playback device 13. The terminal device 10 may be realized as a single device, or as multiple devices configured separately from each other. For example, the playback device 13 may be configured separately from the terminal device 10 and connected to the terminal device 10 by wire or wirelessly.

The control device 11 is a single or multiple processors that control each element of the terminal device 10. Specifically, the control device 11 is configured by one or more types of processors, such as a CPU, SPU, DSP, FPGA, or ASIC.

The storage device 12 is a single or multiple memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a known recording medium, such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the terminal device 10, or a recording medium (e.g., cloud storage) that the control device 11 can write to or read from via the communication network 30 may be used as the storage device 12.

The playback device 13 plays back the content C. Specifically, the playback device 13 includes a display device 131 that displays the video represented by the video data X2 of the content C, and a sound emitting device 132 (e.g., a speaker or headphones) that emits the sound represented by the sound data Y2 of the content C.

FIG. 13 is a block diagram illustrating an example of the functional configuration of the terminal device 10. The control device 11 of the terminal device 10 executes a program stored in the storage device 12 to realize a number of functions (a material data acquisition unit 51, a video data processing unit 52, an audio data processing unit 53, and a playback control unit 55) for generating and providing content C from material data D. In other words, the functions of the terminal device 10 are similar to those of the editing system 20 in the first to fourth embodiments, except that the content providing unit 54 has been replaced with the playback control unit 55.

The material data acquisition unit 51 acquires material data D including video data X1 and audio data Y1. Specifically, the material data acquisition unit 51 generates or edits the material data D in response to an instruction from a user of the terminal device 10. Note that the material data acquisition unit 51 may receive the material data D from an external device with which the terminal device 10 can communicate via the communication network 30.

As in the first embodiment, the video data processing unit 52 generates video data X2 from video data X1 of material data D. The sound data processing unit 53 generates sound data Y2 from sound data Y1 of material data D. Specifically, the sound data processing unit 53 generates sound data Y2 by executing the editing process Sc exemplified in any of the first to fourth embodiments on the sound data Y1. As in the first embodiment, content C is composed of the video data X2 generated by the video data processing unit 52 and the sound data Y2 generated by the sound data processing unit 53.

The playback control unit 55 causes the playback device 13 to play content C. Specifically, the playback control unit 55 causes the display device 131 to display a video by supplying video data X2, and causes the sound emitting device 132 to emit sound by supplying sound data Y2. Thus, the user of the terminal device 10 can view content C. The fifth embodiment also achieves the same effects as the first embodiment.

The editing system 20 illustrated in the first to fourth embodiments and the terminal device 10 illustrated in the fifth embodiment are collectively expressed as an information processing system that processes video data X1 and audio data Y1.

F: Modifications Specific modifications to the above-mentioned embodiments are illustrated below. Two or more of the following embodiments may be combined as long as they are not mutually contradictory.

(1) The configurations according to the above-mentioned embodiments can be combined as appropriate. For example, each sound section Am may be adjusted to have a duration equivalent to each video section Vm by the extraction process (Sc13, Sc23) illustrated in the first or second embodiment, and the volume within each transition period Q may be adjusted by the adjustment process Sc33 illustrated in the third or fourth embodiment.

(2) In the third and fourth embodiments, the transition period Q is illustrated as having the boundary point Pm as its end point, but the relationship between the transition period Q and the boundary point Pm is not limited to the above example. For example, it is also possible to set the transition period Q with the boundary point Pm as its start point, or to set the transition period Q with the boundary point Pm as its midpoint.

(3) In each of the above-mentioned embodiments, the sound data Y (Y1, Y2) represents the performance sound of a piece of music (musical instrument sound or singing sound), but the sound represented by the sound data Y is not limited to musical sound. For example, a form in which the sound data Y represents speech (language sound) that does not include musical elements is also envisioned. For example, the sound data Y may represent speech (e.g., speech by a character in the video or commentary on the video) to be played in parallel with the video represented by the video data X2.

(4) In each of the above embodiments, the material data D includes sound data Y1, but a configuration in which the material data D includes character string data instead of the sound data Y1 is also envisioned. The character string data is data representing a character string corresponding to the sound to be played in parallel to the video of the video data X2. The sound data processing unit 53 generates sound data Y1 by voice synthesis using the character string data of the material data D, and generates sound data Y2 by editing processing Sc on the sound data Y1. Any known method can be used for the voice synthesis.

(5) In the first and second embodiments, the duration of each sound section Am in the sound data Y1 is adjusted (deleted or expanded) to the duration of the video section Vm. However, the sound data processing unit 53 may generate sound data Y2 in which the duration of each sound section Am is set according to the duration of the video section Vm by a synthesis process. The synthesis process is musical sound synthesis, which synthesizes a performance sound from control data representing a time series of musical notes, or voice synthesis, which synthesizes a voice such as a speech sound or a singing sound from control data representing a character string. For example, the sound data processing unit 53 generates sound data Y2 in which each sound section Am is set to the same duration as the video section Vm by a synthesis process that applies control data. As can be understood from the above description, in the first or second embodiment, the adjustment to the sound data Y1 may be omitted. Also, the sound data Y1 used in the third or fourth embodiment may be generated by the synthesis process exemplified above.

(6) In each of the above embodiments, the sound data Y (Y1, Y2) is configured as a time series of samples, but the format of the sound data Y is arbitrary. For example, sound data Y in a format conforming to the MIDI (Musical Instrument Digital Interface) standard may be used.

(7) In each of the above embodiments, the video data X (X1, X2) represents a video, but a configuration in which the video data X represents both a video and sound that are played in parallel with each other is also envisioned. When content C is being played, the sound represented by the video data X2 and the sound represented by the sound data Y2 are played in parallel.

(8) As described above, the functions of the editing system 20 in the first to fourth embodiments are realized by the cooperation of one or more processors constituting the control device 21 and the program stored in the storage device 22. Similarly, the functions of the terminal device 10 in the fifth embodiment are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12.

The program for realizing the above functions can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the storage device that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

G: Supplementary Note From the above-described exemplary embodiments, the following configurations, for example, can be understood.

An information processing method according to one aspect (aspect 1) of the present disclosure is an information processing method for processing video data including a plurality of video segments and sound data including a plurality of sound segments, and processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment. In the above aspect, the sound data is processed so as to transition from the first sound segment to the second sound segment at the boundary point between the first video segment and the second video segment. Therefore, it is possible to generate content that has a sense of unity between changes in the video and changes in sound while suppressing the effect on the video represented by the video data.

Note that "switching from the first sound section to the second sound section at the boundary point between the first video section and the second video section" means that the switching point from the first sound section to the second sound section substantially coincides with the boundary point between the first video section and the second video section. "Substantially coincident" includes cases where the switching point completely coincides with the boundary point, as well as cases where the switching point and the boundary point do not strictly coincide but can be regarded as coinciding. For example, even if the switching point and the boundary point are actually different, a state in which the switching point and the boundary point are close enough that a viewer of the content can perceive them as coinciding can be interpreted as "substantially coinciding."

In a specific example (Aspect 2) of Aspect 1, the sound data is processed so as to switch to the second sound section at an intermediate point in the first sound section that coincides with the boundary point. According to the above aspect, it is possible to switch from the first sound section to the second sound section at the boundary point between the first video section and the second video section by a simple process of starting the second sound section at an intermediate point in the first sound section that coincides with the boundary point.

In a specific example (aspect 3) of aspect 1, in processing the sound data, at least one of the first sound interval and the second sound interval is shortened or extended on the time axis, so that the time point at which the first sound interval switches to the second sound interval coincides with the boundary point. According to the above aspect, it is possible to reduce the possibility that the first sound interval will be cut off midway, or that the first sound interval will be insufficient for the first video interval. Note that the sound interval is expanded or contracted by, for example, adjusting the playback speed. That is, the sound interval is shortened by increasing the playback speed, and the sound interval is extended by decreasing the playback speed.

An information processing method according to another aspect (aspect 4) of the present disclosure is an information processing method for processing video data including a plurality of video segments and audio data representing sound, the audio data being processed so that the volume is reduced within a transition period including a boundary point between a first video segment and a second video segment following the first video segment among the plurality of video segments. In the above aspect, the audio data is processed so that the volume is reduced within a transition period including a boundary point between the first video segment and the second video segment. Therefore, it is possible to generate content that has a sense of unity between changes in the video and changes in the audio, while suppressing the effect on the video represented by the video data.

In a specific example (aspect 5) of aspect 4, the transition period is a period ending at the boundary point, and in processing the sound data, the sound data is processed so that the volume decreases from the start point to the end point of the transition period and increases at the end point of the transition period. In the above aspect, the volume decreases from the start point to the end point (boundary point) of the transition period and increases at the end point of the transition period (the start point of the second video section). Therefore, it is possible to generate content in which the volume decreases over time towards the end point of the first video section and sound is played at a sufficient volume as the second video section begins.

In a specific example (aspect 6) of aspect 4, the transition period is a period ending at the boundary point, and in processing the sound data, the sound data is processed so that the volume decreases at the start of the transition period and increases from the start to the end of the transition period. In the above aspect, the volume decreases at the start of the transition period and increases from the start to the end of the transition period (boundary point). Therefore, it is possible to generate content in which the volume increases over time as the first video segment switches to the second video segment.

An information processing system according to one aspect of the present disclosure is an information processing system that processes video data including a plurality of video segments and sound data including a plurality of sound segments, and includes a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment. An information processing system according to another aspect of the present disclosure is an information processing system that processes video data including a plurality of video segments and sound data representing sound, and includes a sound data processing unit that processes the sound data so as to reduce the volume within a transition period that includes a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment.

A program according to one aspect of the present disclosure is a program for processing video data including multiple video segments and sound data including multiple sound segments, and causes a computer to function as a sound data processing unit that processes the sound data so as to switch from a first sound segment among the multiple sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the multiple video segments and a second video segment subsequent to the first video segment. A program according to another aspect of the present disclosure is a program for processing video data including multiple video segments and sound data representing sound, and causes a computer to function as a sound data processing unit that processes the sound data so as to reduce the volume within a transition period that includes a boundary point between a first video segment among the multiple video segments and a second video segment subsequent to the first video segment.

100...information system, 10...terminal device, 11, 21...control device, 12, 22...storage device, 13...playback device, 131...display device, 132...sound emission device, 20...editing system, 23...communication device, 51...material data acquisition unit, 52...video data processing unit, 53...sound data processing unit, 54...content provision unit, 55...playback control unit.

Claims (6)

1. An information processing method for processing moving image data including a plurality of moving image segments and sound data representing sound, comprising the steps of:
processing the sound data so that a volume is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections, the method comprising:
the transition period is a period ending at the boundary point,
The sound data is processed such that the volume decreases from the start point to the end point of the transition period and the volume increases at the end point of the transition period.
Information processing methods. 1. An information processing method for processing moving image data including a plurality of moving image segments and sound data representing sound, comprising the steps of:
processing the sound data so that a volume is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections, the method comprising:
the transition period is a period ending at the boundary point,
In processing the sound data, the sound data is processed so that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period.
Information processing methods. An information processing system for processing video data including a plurality of video segments and sound data representing sound, comprising:
a sound data processing unit that processes the sound data so that a volume of the sound is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections ;
the transition period is a period ending at the boundary point,
The sound data processing unit processes the sound data such that the volume decreases from the start point to the end point of the transition period and the volume increases at the end point of the transition period.
Information processing system. An information processing system for processing video data including a plurality of video segments and sound data representing sound, comprising:
a sound data processing unit that processes the sound data so that a volume of the sound is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections ;
the transition period is a period ending at the boundary point,
The sound data processing unit processes the sound data such that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period.
Information processing system. A program for processing video data including a plurality of video segments and sound data representing sounds, the program comprising:
functioning as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections ;
the transition period is a period ending at the boundary point,
The sound data processing unit processes the sound data such that the volume decreases from the start point to the end point of the transition period and the volume increases at the end point of the transition period.
program. A program for processing video data including a plurality of video segments and sound data representing sounds, the program comprising:
functioning as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections ;
the transition period is a period ending at the boundary point,
The sound data processing unit processes the sound data such that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period.
program.

Images