JP7720428B2 - Image generation method - Google Patents

Description

The present invention relates to an image generation method, an image generation device, and an image generation program.

When recording video and audio, sounds other than the main audio (wind noise, environmental sounds, operation sounds, speaking voices, etc.) may also be recorded.

Patent document 1 describes a video camera equipped with a means for displaying the presence or absence and strength of wind noise, a means for manually selecting the presence or strength of wind noise countermeasures, and a means for selecting the presence or absence of wind noise countermeasures even after recording.

Patent document 2 also describes a video camera that has a function to automatically reduce wind noise and allows the operation of this function to be set arbitrarily.

JP 2010-4339 A JP 2009-124414 A

One embodiment of the technology disclosed herein provides an image generation method, an image generation device, and an image generation program that can generate an image with audio that emphasizes or reduces specific audio.

(1) A video generation method comprising: a video recording step of recording a first video captured by an imaging unit; a first audio recording step of recording a first audio in synchronization with the first video; a second audio recording step of recording a second audio different from the first audio; an audio generation step of processing the first audio using the second audio to generate a third audio including an enhanced or reduced second audio; and a video generation step of generating a second video by associating the first video with the third audio.

(2) A video generation method according to (1), in which the audio generation step synthesizes the emphasized or reduced second audio with the first audio to generate a third audio.

(3) The video generation method of (2) further includes an intensity setting step of setting the intensity of the second sound before the sound generation step, and the sound generation step combines the second sound with the first sound at the intensity set in the intensity setting step.

(4) A video generation method according to (1), in which the first audio includes a common component that is a common audio component with the second audio, and the audio generation step uses the second audio to perform processing on the first audio to emphasize or reduce the common component, thereby generating a third audio.

(5) A video generation method according to (4), further comprising a processing condition setting step for setting processing conditions for the common component before the audio generation step, wherein the audio generation step performs processing on the first audio to emphasize or reduce the common component in accordance with the processing conditions set in the processing condition setting step.

(6) Any one of the video generation methods (1) to (5) further includes a detection step of detecting movement of the imaging device body including the imaging unit, and the audio generation step performs predetermined processing on the first audio or the second audio to generate a third audio when movement is detected in the detection step.

(7) The image generation method according to any one of (1) to (6), further comprising a first information acquisition step of acquiring imaging information of a first image captured by an imaging unit, and a first display step of displaying the imaging information.

(8) The image generation method of (7), wherein the imaging information includes at least one of information about the movement of the imaging device body including the imaging unit and information about the focal length.

(9) The video generation method of any one of (1) to (8), further comprising a second information acquisition step of acquiring information about a sound collection unit that collects the first sound and the second sound, and a second display step of displaying the information about the sound collection unit.

(10) A video generation method according to any one of (1) to (9), wherein the second audio recording step records the second audio in synchronization with the first video.

(11) The video generation method of (10) further includes a second audio detection step of detecting the timing at which the second audio was recorded, and an associating step of associating the information detected in the second audio detection step with the first video.

(12) A video generation method according to any one of (1) to (11), wherein the second audio recording step records the second audio before the video recording step.

(13) A video generation method according to any one of (1) to (12), wherein the first audio recording step records the first audio via a first audio collection unit, and the second audio recording step records the second audio via a second audio collection unit different from the first audio collection unit.

(14) The image generation method of (13), wherein the second sound collection unit has directional sound collection characteristics, and the first sound collection unit has less directional sound collection characteristics than the second sound collection unit.

(15) The image generation method of (13) or (14), wherein the second sound collection unit has directional sound collection characteristics, and the second sound generation process detects the position of a sound source of the second sound and directs the second sound collection unit in the direction of the detected sound source.

FIG. 1 is a block diagram showing the schematic configuration of an imaging device having a function for generating an image using the image generation method according to the present invention. Block diagram of the main functions realized by the CPU when recording video and audio Block diagram of the main functions realized by the CPU when playing back recorded video Block diagram of the main functions realized by the CPU when generating video with audio Block diagram of functions of a third sound generation unit Block diagram of the main functions realized by the CPU when generating video with audio Block diagram of functions of a third sound generation unit FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus according to a third embodiment. Block diagram of the main functions realized by the CPU when recording video and audio 10 is a block diagram of functions of a third voice generation unit according to a third embodiment; FIG. 13 is a diagram illustrating a modification of the third voice generation unit of the third embodiment. 1 is a block diagram of functions realized by the CPU when acquiring and recording image information and when displaying image information. A block diagram of the functions realized by the CPU when acquiring and recording microphone information and when displaying microphone information. FIG. 10 is a block diagram of functions implemented by the CPU when detecting and recording the timing at which the second audio is recorded and when displaying the recorded information.

A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[First embodiment]
FIG. 1 is a block diagram showing a schematic configuration of an imaging device having a function for generating an image using an image generation method according to the present invention.

The imaging device 1 of this embodiment records first and second audio in synchronization with imaging. Then, after imaging, the first audio is processed using the second audio to generate third audio containing the second audio at a predetermined intensity (audio level). The generated third audio is then associated with the video obtained by imaging (first video) to generate video with audio (second video).

As shown in FIG. 1, the imaging device 1 includes an imaging unit 10, a first audio input unit 12, a second audio input unit 14, a display unit 16, a memory unit 18, an audio output unit 20, an operation unit 22, a CPU (Central Processing Unit) 24, a ROM (Read Only Memory) 26, and a RAM (Random Access Memory) 28. The imaging unit 10 captures video. The imaging unit 10 includes an imaging optical system 10A, an imaging element 10B, and an image signal processing unit 10C. The imaging optical system 10A forms an image of a subject on the light receiving surface of the imaging element 10B. The imaging element 10B converts the image of the subject formed on its light receiving surface by the imaging optical system 10A into an electrical signal. The image signal processing unit 10C performs predetermined signal processing on the signal output from the image sensor 10B to generate a video signal.

The first audio input unit 12 is an input unit for the main audio (first audio). The first audio input unit 12 includes a first microphone 12A and a first audio signal processing unit 12B. The first microphone 12A collects the first audio as the main audio. This first audio is audio that does not include the second audio (including cases where a small amount of the second audio is included). The first microphone 12A is an example of a first audio collection unit. The first audio signal processing unit 12B performs predetermined signal processing on the signal from the first microphone 12A to generate an audio signal for the first audio.

The second audio input unit 14 is an input unit for a specific audio (second audio) to be synthesized with the main audio. The second audio input unit 14 includes a second microphone 14A and a second audio signal processing unit 14B. The second microphone 14A collects the second audio, which is a specific audio. This second audio is audio that does not include the first audio (including cases where it is deemed not to substantially include the first audio). The second microphone 14A is an example of a second audio collection unit. The second audio signal processing unit 14B performs predetermined signal processing on the signal from the second microphone 14A to generate an audio signal for the second audio.

The display unit 16 displays the video being captured by the imaging unit 10 in real time. It also displays the played-back video. It also displays operation screens, menu screens, messages, and the like as needed. The display unit 16 is configured to include, for example, a display device such as an LCD (Liquid Crystal Display), its drive circuitry, and the like.

The memory unit 18 mainly stores captured images and collected audio. The memory unit 18 is configured to include, for example, a storage medium such as a non-volatile memory, its control circuit, etc.

The audio output unit 20 outputs the reproduced audio. The audio output unit 20 also outputs warning sounds and the like as necessary. The audio output unit 20 includes a speaker and a signal processing circuit that processes the audio signal to be output from the speaker.

The operation unit 22 accepts operation inputs from the user. The operation unit 22 is composed of various operation buttons such as a recording button, as well as circuits for detecting these operations.

The CPU 24 executes a predetermined control program to function as a control unit for the entire device. Based on user operations, the CPU 24 controls the operation of each component and provides overall control over the operation of the entire device. By executing a predetermined program, the CPU 24 also functions as an image generation device that generates images with audio using recorded images and audio. Functioning as an image generation device, the CPU 24 processes recorded images and audio based on user operations to generate images with audio. The ROM 26 stores various programs executed by the CPU 24, as well as data necessary for control. The RAM 28 provides working memory space for the CPU 24.

Figure 2 is a block diagram of the main functions realized by the CPU when recording video and audio. As shown in the figure, the CPU 24 functions as an imaging control unit 101, a video output unit 102, a first video recording unit 103, a first audio recording unit 104, and a second audio recording unit 105, among others.

The imaging control unit 101 controls imaging by the imaging unit 10. Based on the video signal obtained from the imaging unit 10, the imaging control unit 101 controls the imaging unit 10 so that video is captured with proper exposure. Based on the video signal obtained from the imaging unit 10, the imaging control unit 101 also controls the imaging unit 10 so that the main subject is in focus.

The video output unit 102 outputs the video captured by the imaging unit 10 to the display unit 16 in real time. This allows a live view to be displayed on the display unit 16.

The first video recording unit 103 records the video (first video) captured by the imaging unit 10 in the storage unit 18. The first video recording unit 103 starts recording the video in response to a command from the user. It also stops recording the video in response to a command from the user. The user issues commands to start and stop recording via the operation unit 22. The video (first video) is recorded in the storage unit 18 in association with the first audio and second audio collected in synchronization with the capture.

The first audio recording unit 104 records the first audio (main audio) input from the first audio input unit 12 in the storage unit 18 in synchronization with the capture of the first video. The first audio is recorded in the storage unit 18 in association with the first video.

The second audio recording unit 105 records the second audio (specific audio) input from the second audio input unit 14 in the storage unit 18 in synchronization with the capture of the first video. The second audio is recorded in the storage unit 18 in association with the first video.

Figure 3 is a block diagram of the main functions implemented by the CPU when playing back recorded video. As shown in the figure, the CPU 24 functions as a video playback unit 111, an audio playback unit 112, etc.

The video playback unit 111 plays video recorded in the storage unit 18 on the display unit 16 in response to a playback instruction from the user. The user uses the display unit 16 and operation unit 22 to select the video to play and issue a playback instruction. The video playback unit 111 reads the selected video from the storage unit 18 and plays it.

When audio is associated with video, the audio playback unit 112 plays the audio in synchronization with the video. When first and second audio are associated with video, the audio playback unit 112 synthesizes and plays the first and second audio. The played audio is output from the audio output unit 20.

Figure 4 is a block diagram of the main functions realized by the CPU when generating video with audio. As shown in the figure, the CPU 24 functions as a first video acquisition unit 121, a first audio acquisition unit 122, a second audio acquisition unit 123, a third audio generation unit 124, an intensity setting unit 125, a video generation unit 126, and a second video recording unit 127, among others.

The first image acquisition unit 121 reads and acquires the image (first image) selected by the user as the image to be processed from the storage unit 18. The user selects the image to be processed using the display unit 16 and operation unit 22. The acquired image data is added to the image generation unit 126.

The first audio acquisition unit 122 reads and acquires data of the first audio (main audio) associated with the video selected for processing from the storage unit 18. The acquired first audio data is added to the third audio generation unit 124.

The second audio acquisition unit 123 reads and acquires the second audio (specific audio) associated with the video selected for processing from the storage unit 18. The acquired second audio data is added to the third audio generation unit 124. Note that the first audio acquisition unit 122 and the second audio acquisition unit 123 may acquire the corresponding audio data directly from the first audio input unit 12 and the second audio input unit 14, without going through the storage unit 18. The imaging device 1 may also record audio data in an external storage unit rather than the internal storage unit 18. In this case, the first audio acquisition unit 122 and the second audio acquisition unit 123 may acquire the audio data from the external storage unit.

The third audio generation unit 124 processes the first audio using the second audio to generate the third audio. The third audio is generated as audio in which the second audio is included in the first audio at a predetermined intensity (audio level). The predetermined intensity is set by the user. Figure 5 is a block diagram of the functions of the third audio generation unit. As shown in the figure, the third audio generation unit 124 has the functions of an intensity adjustment unit 124A and a synthesis unit 124B. The intensity adjustment unit 124A adjusts the intensity of the second audio according to the setting of the intensity setting unit 125. The synthesis unit 124B synthesizes the intensity-adjusted second audio with the first audio to generate the third audio. This generates audio (third audio) in which the second audio is included in the first audio at a predetermined intensity. Note that, as described above, the first audio is recorded in synchronization with the video, so the generated third audio is also synchronized with the video. The generated third audio data is input to the video generation unit 126.

The intensity setting unit 125 sets the intensity (audio level) of the second audio when it is synthesized with the first audio. The intensity setting unit 125 sets the intensity based on operation input from the operation unit 22. By setting the intensity of the second audio via the operation unit 22, the user can emphasize or attenuate the second audio relative to the first audio.

The video generation unit 126 associates the video (first video) acquired by the first video acquisition unit 121 with the third audio generated by the third audio generation unit 124 to generate video with audio (second video). For example, it containerizes the video file and audio file to generate a video file in a specific video format. For example, it generates files such as AVI (Audio Video Interleave) and MP4 (MPEG-4 Part 14 (ISO/IEC 14496-14:2003, ISO/IEC JTC 1)).

The second video recording unit 127 stores the video with audio (second video) generated by the video generation unit 126 in the memory unit 18.

Next, we will explain the procedure (image generation method) for generating video with audio using the imaging device 1 configured as described above.

First, imaging is performed, and the image, first audio, and second audio are recorded. Specifically, the image (first image) captured by the imaging unit 10 is recorded in the storage unit 18 (image recording process). Furthermore, in synchronization with the imaging, the first audio and second audio are collected and recorded in the storage unit 18 (first audio recording process and second audio recording process). Here, the main audio is recorded as the first audio. On the other hand, a specific audio is recorded as the second audio. Here, the "specific audio" refers to audio that is different from the main audio and is to be included in the main audio. For example, when capturing video of a person talking in a windy environment, the person's speaking voice can be recorded as the main audio, and wind noise (sound caused by wind hitting a microphone) can be recorded as the specific audio. Or, when capturing video of a person talking on the beach, the person's speaking voice can be recorded as the main audio, and the sound of waves can be recorded as the specific audio.

Next, the intensity (audio level) of the second audio when synthesizing the second audio with the first audio is set (intensity setting process). The user sets the intensity via the operation unit 22. This setting allows the user to arbitrarily emphasize or attenuate the second audio relative to the first audio.

Next, the first sound is processed using the second sound to generate a third sound that includes the second sound at a predetermined intensity within the first sound (sound generation process). The predetermined intensity is the intensity set in the intensity setting process. In this process, the second sound is first adjusted to the intensity set by the user. This generates a second sound that is emphasized or attenuated relative to the first sound. The intensity-adjusted second sound is then synthesized with the first sound to generate a third sound. This generates a sound (third sound) that includes the second sound at a predetermined intensity within the first sound. For example, if wind noise is recorded as the second sound, a sound (third sound) is generated in which the wind noise is included at a predetermined intensity within the main sound. Furthermore, for example, if the sound of waves is recorded as the second sound, a sound (third sound) is generated in which the sound of waves is included at a predetermined intensity within the main sound.

Next, the audio (third audio) generated in the audio generation process is associated with the video obtained by imaging (first video), and an audio-accompanied video (second video) is generated (video generation process).

Video with audio is generated through the above series of steps. The generated video with audio is recorded in the storage unit 18.

The video generation method of this embodiment allows specific audio (second audio) to be recorded separately from the main audio (first audio), allowing the specific audio to be isolated and edited. This allows video with audio to be generated in line with the user's intentions.

[Modification of the first embodiment]
(1) Modification of synthesis of second audio The second audio can be synthesized only in a specific section of the video (a section on the time axis). In this case, the section to be synthesized is specified, and the second audio is synthesized with the first audio. The section to be synthesized is specified, for example, while the first video and the first audio are being played back.

The intensity of the second audio when synthesized can also be changed partially along the time axis. This makes it possible to generate video with audio in which the intensity of specific audio is changed depending on the scene, for example.

(2) Variations on Intensity Setting In the above embodiment, the user can arbitrarily set the intensity of the second voice when it is synthesized with the first voice, but it is also possible to configure the synthesis to be performed at an intensity selected from a plurality of predetermined intensity settings (for example, strong reduction, weak reduction, strong emphasis, weak emphasis).

(3) Modifications for Recording the Second Audio The second audio does not necessarily need to be recorded in synchronization with the capture of the first video. For example, when the second audio is to be synthesized only in a specific section as described above, the second audio may be recorded in advance or afterward.

(4) Modifications of the First Audio Input Unit and the Second Audio Input Unit The first audio input unit 12 and the second audio input unit 14 preferably perform filtering on the audio signal of the collected audio as needed. For example, the first audio input unit 12 preferably performs filtering so that the main audio is recorded clearly. Similarly, the second audio input unit 14 preferably performs filtering so that a specific audio is recorded clearly.

It is also preferable to use microphones appropriate for the purpose for the first audio input unit 12 and the second audio input unit 14. For example, if a wide-range sound is to be collected as the main sound, a microphone with less omnidirectional (preferably omnidirectional) sound collection characteristics than the second microphone 14A is used for the first microphone 12A. Furthermore, a microphone with directional sound collection characteristics (such as a gun microphone) is used for the second microphone 14A, which collects specific sounds. This allows the first sound and second sound to be recorded with high accuracy.

The first microphone 12A and the second microphone 14A may be built into the main body of the imaging device 1, or may be externally attached.

Second Embodiment
As in the first embodiment, an example in which an image is generated using an imaging device will be described.

In this embodiment, audio including a second audio is recorded as a first audio. Then, using the second audio recorded separately from the first audio, processing is performed on the first audio to emphasize or reduce the second audio, thereby generating a third audio. The basic configuration of the imaging device is the same as in the first embodiment described above, but the functions realized by the CPU 24 are different.

Figure 6 is a block diagram of the main functions realized by the CPU when generating video with audio. As shown in the figure, the CPU 24 functions as a first video acquisition unit 121, a first audio acquisition unit 122, a second audio acquisition unit 123, a third audio generation unit 124, a processing condition setting unit 128, a video generation unit 126, and a second video recording unit 127, among others. The functions of each unit except for the third audio generation unit 124 and the processing condition setting unit 128 are substantially the same as those in the first embodiment described above. Therefore, only the functions of the third audio generation unit 124 and the processing condition setting unit 128 will be described here.

Figure 7 is a block diagram of the functions of the third voice generation unit.

As described above, in this embodiment, audio including the second audio is recorded as the first audio. The third audio generation unit 124 processes the first audio to emphasize or reduce audio components that are common to the second audio (common components), thereby generating the third audio. Specifically, audio components with the same frequency as the second audio are considered common components, and these audio components with the same frequency as the second audio are processed under processing conditions set by the user to generate the third audio. For this reason, the third audio generation unit 124 has the functions of a frequency detection unit 124C and an audio processing unit 124D.

The frequency detection unit 124C analyzes the data of the second audio and detects the frequency of the second audio. The second audio is a specific audio component of the first audio component that the user wishes to emphasize or reduce. As in the first embodiment, the second audio component is collected by the second microphone 14A. The information detected by the frequency detection unit 124C is sent to the audio processing unit 124D.

The audio processing unit 124D acquires information about the frequency of the second audio detected by the frequency detection unit 124C, and processes the first audio using the processing conditions set by the processing condition setting unit 128 to generate the third audio. In other words, the audio components (common components) with the same frequency as the second audio are processed using the processing conditions set by the user to generate the third audio.

The processing condition setting unit 128 sets the processing conditions when processing the first audio. Specifically, it sets the processing conditions (sound emphasis or reduction processing) for the common components, which are audio components that are common to the second audio. The processing condition setting unit 128 sets the processing conditions based on operation input from the operation unit 22. By setting the processing conditions when processing the first audio via the operation unit 22, the user can emphasize, reduce, or cancel the second audio included in the first audio.

Next, we will explain the procedure (image generation method) for generating video with audio using the imaging device configured as described above.

First, imaging is performed, and the image, first audio, and second audio are recorded. Specifically, the image (first image) captured by the imaging unit 10 is recorded in the storage unit 18 (image recording process). Furthermore, in synchronization with the imaging, the first audio and second audio are collected and recorded in the storage unit 18 (first audio recording process and second audio recording process). As described above, audio including the second audio is recorded as the first audio. That is, audio including a common component, which is an audio component common to the second audio, is recorded. Meanwhile, a specific audio component within the first audio is recorded as the second audio. Here, "specific audio" refers to audio included in the first audio that the user wishes to emphasize or reduce. For example, when capturing video of a person talking in a windy environment, wind noise can be recorded as the second audio. Or, when capturing video of a person talking on the beach, the sound of waves can be recorded as the second audio.

Next, processing conditions for the common components of the first audio when processing the first audio are set (processing condition setting process). The user sets these processing conditions via the operation unit 22. This setting allows the user to arbitrarily emphasize, reduce, or cancel the second audio included in the first audio.

Next, the first sound is processed using the second sound to emphasize or reduce the common components, thereby generating a third sound (sound generation process). In this process, the frequency of the second sound is first detected. Then, the first sound is processed to emphasize or reduce the sound components of that frequency according to the processing conditions set in the processing condition setting process, thereby generating a third sound. This generates a sound (third sound) that includes the second sound contained in the first sound at the intensity intended by the user, or that has been canceled. For example, if the first sound is recorded in a windy environment and wind noise is recorded as the second sound, it is possible to generate a sound in which the wind noise has been reduced or canceled. It is also possible to generate a sound in which wind noise has been emphasized as necessary.

Next, the audio (third audio) generated in the audio generation process is associated with the video obtained by imaging (first video), and an audio-accompanied video (second video) is generated (video generation process).

Video with audio is generated through the above series of steps. The generated video with audio is recorded in the storage unit 18.

The video generation method of this embodiment allows you to record a specific audio (second audio) included in the main audio separately from the main audio (first audio), allowing you to isolate and edit the specific audio. This allows you to generate video with audio according to the user's intentions.

[Modification of the second embodiment]
(1) Modification of the Generation of the Third Audio The process of emphasizing, reducing, or canceling the common component can be performed only in a specific section of the video. In this case, the process is performed by specifying the section.

In addition, the processing conditions for the common components can be partially varied along the time axis. This makes it possible, for example, to generate video with audio in which the intensity of specific sounds is changed depending on the scene.

(2) Modifications for Recording the Second Audio The second audio does not necessarily need to be recorded in synchronization with the capture of the first video. The second audio may be recorded in advance or afterward. For example, the audio of the environment to be edited (such as wind noise, the sound of a waterfall, or the sound of construction work) may be recorded in advance as the second audio. Alternatively, the audio of the environment to be edited may be recorded in advance as a sample audio, and the second audio may be created from the common components of the recorded audio and the sample audio during the video recording process.

In addition, representative second audio can be stored in advance as preset data in the imaging device. This makes it possible to generate audio that emphasizes or reduces audio when audio containing audio stored as preset data is recorded together with video. For example, if wind noise is stored as preset data, video with audio that emphasizes or reduces the wind noise of the first audio can be generated using the wind noise frequency data contained in that preset data. The preset data is stored, for example, in ROM 26 or memory unit 18. When generating video, the user selects the audio data to be edited.

(3) Modifications of the First and Second Audio Input Units As in the first embodiment, the first and second audio input units 12, 14 preferably perform filtering on the audio signal of the collected audio as needed. Also, the first and second audio input units 12, 14 preferably use microphones suited to the purpose.

Incidentally, except when the second audio is recorded in synchronization with the image capture, the second audio can also be configured to be collected by the first microphone 12A. In other words, when recording the second audio beforehand or afterward, the first audio input unit 12 can be used to collect and record the second audio. Therefore, when recording the second audio beforehand or afterward, the second audio input unit 14 is not required on the device main body.

[Third embodiment]
In this embodiment, the movement of the imaging device body is detected while recording video and audio, and the third audio is generated taking into account information about that movement.

Figure 8 is a block diagram showing the general configuration of an imaging device according to this embodiment. As shown in the figure, the imaging device 1 according to this embodiment differs from the imaging devices according to the first and second embodiments in that it further includes a motion detection unit 30.

The motion detection unit 30 detects the motion of the imaging device body, including the imaging unit 10. The motion detection unit 30 detects the motion of the imaging device body in synchronization with the imaging by the imaging unit 10. That is, motion detection begins simultaneously with the start of imaging and ends simultaneously with the end of imaging (motion detection process). The motion detection unit 30 is composed of, for example, an acceleration sensor. Note that if the imaging device body is equipped with an image shake correction function or the like, the sensor used for shake detection, etc., can also be used as the motion detection sensor.

Figure 9 is a block diagram of the main functions realized by the CPU when recording video and audio. As shown in the figure, the CPU 24 functions as an imaging control unit 101, a video output unit 102, a first video recording unit 103, a first audio recording unit 104, a second audio recording unit 105, a motion recording unit 106, and the like. The functions of each unit, except for the motion recording unit 106, are substantially the same as those in the first embodiment described above.

The motion recording unit 106 records information about the motion of the imaging device body detected by the motion detection unit 30 in the storage unit 18 in synchronization with the capture of the first video. The motion information is associated with the first video and recorded in the storage unit 18. When generating video with audio (second video), the motion information stored in the storage unit 18 is used to generate audio (third audio) to be associated with the video.

The third sound generation unit 124 generates the third sound by taking into account movement information. Here, we will explain an example in which the third sound is generated by synthesizing the second sound with the first sound. As explained in the first embodiment above, the first sound is a sound that does not substantially include the second sound, and the second sound is a sound that does not substantially include the first sound.

Figure 10 is a block diagram of the functions of the third sound generation unit of this embodiment. As shown in the figure, the third sound generation unit 124 of this embodiment has the functions of a second sound processing unit 124E, an intensity adjustment unit 124A, and a synthesis unit 124B. The functions of each unit except for the second sound processing unit 124E are substantially the same as those in the first embodiment described above.

The second audio processing unit 124E acquires information about the movement of the imaging device body when recording video and audio, and processes the second audio based on that movement information. Specifically, it processes the second audio according to predetermined processing conditions in response to the movement of the imaging device body. As an example, we will explain the case where audio (third audio) is generated that includes wind noise (second audio) in addition to the main audio (first audio). Assume that the second microphone 14A is composed of a pair of left and right microphones and is integrated into the imaging device body. Therefore, in this case, the second microphone 14A moves integrally with the imaging device body. When panning of the imaging device body is detected, the second audio processing unit 124E performs processing to change the intensity of the left and right audio in response to the movement of the imaging device body. Specifically, it weakens the audio from the moving microphone. This allows for appropriate processing of wind noise, which changes between the left and right microphones in response to the movement of the imaging device body. In other words, since wind noise is stronger in the moving microphone, weakening the noise in response to the movement allows for balanced left and right audio to be synthesized.

The intensity adjustment unit 124A adjusts the intensity of the processed second audio according to the setting of the intensity setting unit 125. The synthesis unit 124B synthesizes the intensity-adjusted second audio with the first audio to generate a third audio.

In this way, in this embodiment, the second audio is automatically processed in accordance with the movement of the imaging device body to generate the third audio. This makes it possible to automatically remove the effects of the movement of the imaging device body.

[Modification of the third embodiment]
Below, we will explain the case where a second audio is used to perform processing on a first audio to emphasize or reduce common components, thereby generating a third audio, and where information about the movement of the imaging device itself is added to generate the third audio.

Figure 11 is a block diagram of the functions of the third sound generation unit in this example. As shown in the figure, the third sound generation unit 124 in this example differs from the third sound generation unit 124 in the second embodiment in that the sound processing unit 124D processes the first sound based on information about the movement of the imaging device body.

The audio processing unit 124D processes the first audio in accordance with predetermined processing conditions in response to the movement of the imaging device body. As an example, we will explain a case where audio (first audio) that includes wind noise (second audio) in the main audio is recorded in synchronization with imaging, and audio (third audio) is generated in which the wind noise is emphasized or reduced. In this case, the first audio is recorded as audio that includes wind noise (second audio) in addition to the main audio. The second audio is recorded as the wind noise.

The frequency detection unit 124C analyzes the data of the second audio and detects the frequency of the second audio.

The user then selects one intensity setting from multiple predefined audio intensity settings. The audio processing unit 124D acquires information on the frequency of the second audio detected by the frequency detection unit 124C and information on the movement of the imaging device body, and processes the first audio by combining the processing conditions set by the processing condition setting unit 128 with the movement information to generate the third audio. For example, consider a case where video and audio are recorded while moving in an environment where wind noise (second audio) is recorded along with the main audio (first audio). The volume of the wind noise (second audio) changes depending on the speed of movement. Therefore, the audio processing unit 124D corrects the predefined intensity setting or changes the frequency to be processed (the frequency of the common component) depending on the speed of movement (movement) of the imaging device body. This allows the target audio (second audio) to be processed appropriately, even when video and audio are recorded while moving. For example, if the user's setting is to slightly reduce wind noise (second audio), a scene in which the imaging device body is recognized as moving fast will have a correction applied to it that significantly reduces wind noise compared to other scenes, resulting in the third audio. This prevents certain sounds (wind, waves, etc.) from becoming too loud in only certain scenes in the third audio.

[Fourth embodiment]
In this embodiment, when a video (first video) is captured, imaging information of the video is acquired and recorded in storage unit 18. The recorded imaging information is used when generating a video with audio (second video). Specifically, when generating a video with audio, the imaging information is displayed on display unit 16. The user uses the information displayed on display unit 16 to identify the section (scene) for which audio editing is to be performed. Here, imaging information is information related to the capture of the video. For example, it includes information on the focal length when the video was captured, information on the subject distance, information on exposure, etc. Furthermore, if the imaging device has a function for detecting the movement of the imaging device itself, information on the movement of the imaging device itself during imaging is also included.

Processing such as acquiring, recording, and displaying imaging information is performed by the CPU 24. Figure 12 is a block diagram of the functions realized by the CPU when acquiring and recording imaging information and when displaying imaging information. As shown in the figure, the CPU 24 functions as an imaging information acquisition unit 131, an imaging information recording unit 132, and an imaging information display unit 133.

The imaging information acquisition unit 131 acquires imaging information of the first video in synchronization with the imaging unit 10 capturing the first video. The imaging information acquired includes information on subject distance, focal length, and information on the movement of the imaging device (for example, the output of an acceleration sensor).

The imaging information recording unit 132 records the imaging information acquired by the imaging information acquisition unit 131 in the storage unit 18. The imaging information is recorded in association with the video (first video).

The imaging information display unit 133 acquires imaging information from the memory unit 18 based on operation input from the operation unit 22 and displays it on the display unit 16. For example, it displays the imaging information in chronological order.

Video with syllables can be generated using the imaging device of this embodiment, for example, as follows:

First, a first video is captured by the imaging unit 10 and recorded in the storage unit 18 (video recording process). Imaging information about the first video is acquired in synchronization with the imaging (first information acquisition process), and is recorded in association with the first video in the storage unit 18. Also, a first sound and a second sound are collected in synchronization with the imaging, and are recorded in association with the first video in the storage unit 18 (first sound recording process and second sound recording process).

Next, video with audio (second video) is generated using the recorded first video, first audio, and second audio. First, the imaging information of the first video is read from the storage unit 18 and displayed on the display unit 16 (first display process). The imaging information is displayed in chronological order of the first video. Displaying this imaging information makes it easier to identify areas where audio editing is required. The user specifies, via the operation unit 22, the areas where the second audio should be synthesized or the areas where the second audio should be edited (areas where the second audio should be emphasized, reduced, or canceled), and instructs the generation of video with audio (second video).

When an instruction to generate a video with audio is given, the first audio is processed using the second audio based on operation input from the operation unit 22, and a third audio is generated (audio generation process). The generated third audio is then associated with the first video, and a video with audio (second video) is generated (video generation process).

In this way, according to this embodiment, the imaging information of the first video is acquired and recorded. This makes it easy to identify areas where audio editing is required when generating video with audio.

[Modification of the Fourth Embodiment]
The uses of the acquired imaging information are not limited to the above examples. For example, imaging information can be used to automatically identify areas requiring audio editing, automatically process the first audio, and generate the third audio. For example, imaging information can be used to identify areas where the imaging device itself is moving significantly (areas where the movement is greater than a threshold) or areas where the subject is moving significantly (areas where the change in subject distance is greater than a threshold), and the like, and automatically process the identified areas. Furthermore, imaging information can be used to automatically change the audio editing method. When reducing the second audio throughout the entire video, the reduction intensity can be automatically changed based on imaging information (e.g., by changing the intensity to areas where the imaging device itself is moving significantly or areas where the subject is moving significantly). Furthermore, imaging information can be used to detect areas requiring audio editing and automatically cue the audio.

Fifth Embodiment
In the present embodiment, information from the first microphone 12A that collects the first sound and the second microphone 14A that collects the second sound is acquired and recorded in the storage unit 18. The recorded information from the first microphone 12A and the second microphone 14A is used when generating a video with sound (second video). Specifically, when generating a video with sound, the information is displayed on the display unit 16. The user uses the information displayed on the display unit 16 to set the intensity of the second sound, etc. Here, the microphone information includes, for example, information on whether or not a windshield is provided, information on the type of windshield (sponge type, fur type, cage type, etc.) if a windshield is provided, information on whether or not the device is directional, etc. In addition, the microphone information can also include information on various performance elements of the microphone (for example, information on the microphone's directional characteristics (information on how sensitivity changes depending on the direction from which sound arrives), information on frequency characteristics (information on how sensitivity changes depending on the pitch of the sound), information on maximum sound pressure level (the loudest sound level that the microphone can pick up), information on equivalent noise level (input converted noise level), information on output impedance, information on open circuit sensitivity, etc.).

The user inputs information from the first microphone 12A and the second microphone 14A to the imaging device 1 via the operation unit 22. The CPU 24 stores the information from the first microphone 12A and the second microphone 14A input via the operation unit 22 in the memory unit 18. When generating video with sound, the information from the first microphone 12A and the second microphone 14A recorded in the memory unit 18 is displayed on the display unit 16.

Figure 13 is a block diagram of the functions implemented by the CPU when acquiring and recording microphone information and when displaying microphone information. As shown in the figure, the CPU 24 functions as a microphone information acquisition unit 141, a microphone information recording unit 142, and a microphone information display unit 143.

The microphone information acquisition unit 141 acquires information about the first microphone 12A and the second microphone 14A. As described above, the information about the first microphone 12A and the second microphone 14A is input by the user via the operation unit 22. The user inputs information about the first microphone 12A and the second microphone 14A, such as whether or not they have a windshield, and if so, the type of windshield (sponge type, fur type, cage type, etc.), and whether or not they are directional.

The microphone information recording unit 142 records the information from the first microphone 12A and the second microphone 14A acquired by the microphone information acquisition unit 141 in the memory unit 18. The information from the first microphone 12A and the second microphone 14A is recorded in association with the video (first video).

The microphone information display unit 143 acquires information about the first microphone 12A and the second microphone 14A from the memory unit 18 based on operation input from the operation unit 22 and displays it on the display unit 16.

Video with audio is generated using the imaging device of this embodiment, for example, as follows:

First, a first video is captured by the imaging unit 10 and recorded in the storage unit 18 (video recording process). Furthermore, a first audio signal and a second audio signal are collected in synchronization with the capture and recorded in the storage unit 18 in association with the first video signal (first audio recording process and second audio recording process). Furthermore, information from the first microphone 12A and second microphone 14A used during the capture is input (second information acquisition process) and recorded in the storage unit 18.

Next, a video with audio (second video) is generated using the recorded first video, first audio, and second audio. First, information from the first microphone 12A and second microphone 14A is read from the storage unit 18 and displayed on the display unit 16 (second display process). The user sets the intensity when synthesizing the second audio based on the information from the first microphone 12A and second microphone 14A. The user also sets the processing conditions when processing the first audio based on the information from the first microphone 12A and second microphone 14A. For example, the intensity is set depending on the presence or absence of a windshield and its type.

When an instruction to generate a video with audio is given, the first audio is processed using the second audio based on operation input from the operation unit 22, and a third audio is generated (audio generation process). The generated third audio is then associated with the first video, and a video with audio (second video) is generated (video generation process).

In this way, according to this embodiment, information from the first microphone 12A and the second microphone 14A is acquired and recorded. This allows the third audio to be generated more appropriately when generating video with audio. For example, loss of the main audio can be minimized.

[Modification of the Fifth Embodiment]
The use of the acquired information from the first microphone 12A and the second microphone 14A is not limited to the above example. For example, a configuration may be adopted in which the first audio is automatically processed to generate a third audio using the information from the first microphone 12A and the second microphone 14A. For example, when generating a third audio by synthesizing a second audio with the first audio, the intensity of the second audio when synthesizing the second audio may be automatically set depending on the type of the second microphone 14A. Furthermore, when generating a third audio by processing audio components of the first audio with the same frequency as the second audio, the frequency to be processed may be automatically changed depending on the type of the second microphone 14A.

In addition, in the above example, the user inputs information from the first microphone 12A and the second microphone 14A into the imaging device 1, but it is also possible to configure the information from the first microphone 12A and the second microphone 14A to be collected automatically.

Sixth Embodiment
In this embodiment, the timing at which the second audio is recorded is detected and recorded while the first video is being captured. The recorded information is used when generating the video with audio (the second video). Specifically, when generating the video with audio, the information is displayed on the display unit 16. The user uses the information displayed on the display unit 16 to specify the section (scene) for which audio editing is to be performed.

The CPU 24 detects the timing at which the second audio is recorded. Figure 14 is a block diagram of the functions implemented by the CPU when detecting and recording the timing at which the second audio is recorded and when displaying the recorded information. As shown in the figure, the CPU 24 functions as a second audio detection unit 151, a timing information recording unit 152, and a timing information display unit 153.

The second audio detection unit 151 detects the timing when the second audio was recorded based on the audio signal of the second audio input from the second audio input unit 14. In other words, it detects the input of the audio signal of the second audio and detects the timing when the second audio was recorded.

The timing information recording unit 152 records information (timing information) about the recording timing of the second audio detected by the second audio detection unit 151 in the storage unit 18. The timing information is recorded in association with the video (first video).

The timing information display unit 153 acquires timing information from the memory unit 18 based on operation input from the operation unit 22 and displays it on the display unit 16. For example, it displays the timing at which the second audio was recorded on a time axis.

Video with syllables can be generated using the imaging device of this embodiment, for example, as follows:

First, a first video is captured by the imaging unit 10 and recorded in the storage unit 18 (video recording process). Timing information for the first video is acquired in synchronization with the capturing (first information acquisition process), and is recorded in association with the first video in the storage unit 18. Also, a first sound and a second sound are collected in synchronization with the capturing, and are recorded in association with the first video in the storage unit 18 (first sound recording process and second sound recording process). Also, the timing at which the second sound was recorded is detected (second sound detection process). The detected information is recorded in association with the first video in the storage unit 18 (association process).

Next, video with audio (second video) is generated using the recorded first video, first audio, and second audio. First, timing information is read from the storage unit 18 and displayed on the display unit 16. For example, the timing at which the second audio was recorded is displayed on the time axis. The user specifies the point at which the second audio is to be synthesized or edited via the operation unit 22, and instructs the generation of video with audio (second video).

When an instruction to generate a video with audio is given, the first audio is processed using the second audio based on operation input from the operation unit 22, and a third audio is generated (audio generation process). The generated third audio is then associated with the first video, and a video with audio (second video) is generated (video generation process).

In this way, according to this embodiment, the timing at which the second audio is recorded is detected and recorded. This makes it easy to identify locations (scenes) where audio editing is required when generating video with audio.

[Modification of the Sixth Embodiment]
The use of the acquired timing information is not limited to the above example, and for example, it may be configured to automatically identify portions that require audio editing based on the timing information.

The timing at which the second audio was recorded may also be detected after the capture of the first video has ended. In other words, the audio data of the second audio may be analyzed after the capture has ended to detect the timing at which the second audio was recorded.

[Other embodiments]
As described above, it is preferable to use a microphone with omnidirectional sound collection characteristics for the first microphone 12A that collects the first sound, which is the main sound. It is also preferable to use a microphone with directional sound collection characteristics (such as a gun microphone) for the second microphone 14A that collects the second sound. This improves the accuracy of recording the first sound and the second sound. It is also possible to adjust the sound while maintaining the sound characteristics of a specific voice, for example.

Furthermore, when collecting the second sound using a microphone with directional sound collection characteristics, if the directivity can be adjusted, it is more preferable to change the microphone's directivity in accordance with changes in the position of the sound source of the second sound. For example, the orientation of the second microphone 14A is changed in response to changes in the position of the sound source of the second sound within the video. Changes in the position of the sound source can be detected, for example, by analyzing the video. For example, the subject that is the sound source of the second sound is identified within the video, and the position of the subject is detected using image recognition, etc., to identify the position of the sound source.

In addition, while the above embodiment has been described with reference to an example in which the present invention is implemented in an imaging device, the devices and systems implementing the present invention are not limited to this. For example, the present invention can also be implemented in a portable electronic device (e.g., a smartphone, tablet computer, laptop computer, etc.) equipped with imaging and recording functions. It is also possible to import the recorded first video, first audio, and second audio into a computer (e.g., a personal computer, etc.), generate the third audio on the computer, and generate video with audio (second video).

The control unit that performs functions such as generating the third audio and generating the second video can be implemented using various processors. These include, for example, a CPU, a general-purpose processor that executes software (programs) to implement various functions. These processors also include a GPU (Graphics Processing Unit), a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture. Furthermore, these processors also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to perform specific processing.

The control unit may be implemented by a single processor, or by multiple processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple functions may also be implemented by a single processor. Examples of multiple functions implemented by a single processor include: 1) a single processor implemented by a combination of one or more CPUs and software, as in computers such as servers, which implements multiple functions; 2) a processor that implements the functions of the entire system on a single integrated circuit (IC) chip, as in systems on chips (SoCs). In this way, various functions are implemented as a hardware structure using one or more of the above-mentioned processors. Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit (circuitry) that combines circuit elements such as semiconductor devices. These electrical circuits may be electrical circuits that achieve the above functions using logical operations such as logical sum, logical product, logical negation, exclusive logical sum, and combinations of these.

When the above processor or electrical circuit executes software (programs), the processor (computer) readable code of the software to be executed is stored in a non-transitory recording medium such as a ROM, and the processor references that software. Software stored in a non-transitory recording medium includes programs for image input, analysis, display control, etc. Code may also be recorded in a non-transitory recording medium such as various magneto-optical recording devices or semiconductor memory instead of a ROM. When processing using the software, RAM, for example, is used as a temporary storage area, and data stored in an EEPROM (Electronically Erasable and Programmable Read Only Memory) (not shown) may also be referenced.

1 Imaging device 10 Imaging unit 10A Imaging optical system 10B Imaging element 10C Image signal processing unit 12 First audio input unit 12A First microphone 12B First audio signal processing unit 14 Second audio input unit 14A Second microphone 14B Second audio signal processing unit 16 Display unit 18 Storage unit 20 Audio output unit 22 Operation unit 24 CPU
26 ROM
28 RAM
30 Motion detection unit 101 Imaging control unit 102 Video output unit 103 First video recording unit 104 First audio recording unit 105 Second audio recording unit 106 Motion recording unit 111 Video playback unit 112 Audio playback unit 121 First video acquisition unit 122 First audio acquisition unit 123 Second audio acquisition unit 124 Third audio generation unit 124A Intensity adjustment unit 124B Synthesis unit 124C Frequency detection unit 124D Audio processing unit 124E Second audio processing unit 125 Intensity setting unit 126 Video generation unit 127 Second video recording unit 128 Processing condition setting unit 131 Imaging information acquisition unit 132 Imaging information recording unit 133 Imaging information display unit 141 Microphone information acquisition unit 142 Microphone information recording unit 143 Microphone information display unit 151 Second audio detection unit 152 Timing information recording unit 153 Timing information display unit

Claims (10)

a video recording step of recording a first video captured by the imaging unit;
a first audio recording step of recording a first audio in association with the first video;
a second audio recording step of recording a second audio different from the first audio in association with the first video;
a second audio detection step of detecting a timing at which the second audio is recorded in the first video ;
a step of displaying the timing at which the second audio was recorded on a time axis and displaying information on the timing at which the second audio was recorded in the first video on a display unit ;
playing the first video and the first audio;
receiving , by using the information displayed on the display unit, a designation of a section on a time axis that requires editing by combining the second audio with the first video and the first audio;
an intensity setting step of receiving an instruction to set the intensity of the second sound;
a sound generating step of synthesizing the second sound, which has been emphasized or reduced based on the setting instruction, for a specified section of the first sound to generate a third sound;
an image generating step of generating a second image by associating the first image with the third audio;
An image generation method comprising: the first voice includes a common component that is a voice component common to the second voice,
the sound generating step performs a process of emphasizing or reducing the common component on the first sound using the second sound to generate the third sound.
The image generation method according to claim 1 . a processing condition setting step of setting processing conditions for the common component before the sound generating step,
the sound generating step performs processing on the first sound to emphasize or reduce the common component in accordance with the processing conditions set in the processing condition setting step;
The image generation method according to claim 2 . a detection step of detecting a movement of the imaging device body including the imaging unit,
the sound generating step, when the motion is detected in the detecting step, performs a predetermined process on the first sound or the second sound to generate the third sound.
The image generating method according to any one of claims 1 to 3. a first information acquisition step of acquiring imaging information of the first video by the imaging unit;
a first display step of displaying the imaging information;
The image generating method according to claim 1 , further comprising: The imaging information includes at least one of information on the movement of the imaging device body including the imaging unit and information on the focal length.
The image generation method according to claim 5 . a second information acquisition step of acquiring information about a sound collection unit that collects the first sound and the second sound,
a second display step of displaying information about the sound collection unit;
The image generating method according to claim 1 , further comprising: the first sound recording step includes recording the first sound via a first sound collection unit;
the second sound recording step includes recording the second sound via a second sound collection unit different from the first sound collection unit.
The image generation method according to any one of claims 1 to 7. the second sound collection unit has directional sound collection characteristics,
The first sound collection unit has a sound collection characteristic with lower directivity than the second sound collection unit.
The image generation method according to claim 8. the second sound collection unit has directional sound collection characteristics,
the second sound recording step includes detecting a position of a sound source of the second sound, directing the second sound collection unit in the direction of the detected sound source, and recording the second sound.
10. The image generating method according to claim 8 or 9.