JP7710909B2 - Imaging device, control method, and program - Google Patents

Description

The present invention relates to a voice processing device that performs voice processing on a person's voice.

When shooting video using an imaging device, it is important to capture the shooting conditions exactly as the photographer imagined them, and this applies not only to video but also to audio.

Patent document 1 discloses a method for extracting the sound of a subject and individually adjusting the extracted sound signal according to the position of the subject to create an acoustic space with a sense of realism and stereoscopic sensation.

JP 2012-138930 A

However, when humans listen to a conversation, the accurately reproduced acoustic space does not necessarily match what humans imagine. For example, even when many people are chatting away, humans can naturally hear the conversation of people they are interested in, as well as their own name. It is also said that humans use visual information as well as audio information, and by visually confirming who is speaking, they supplement how they hear by using information obtained from the person's mouth movements and gestures. In other words, it is also important that the audio recorded on video is recorded so that it is the same as the conversational audio that is remembered (imaged).

However, in Patent Document 1, the aim is to accurately reproduce the acoustic space of the voice based on the relative positions of people (sound sources), so there is a risk that the resulting video may differ from the image of the photographer.

Therefore, the present invention aims to record video and audio that matches the photographer's imagination.

The imaging device of the present invention is characterized by having a detection means for detecting a subject from a video, a selection means for selecting a main subject from the subjects detected from the video, a determination means for determining the sound of the subject from the video, an association means for associating the subject detected by the detection means with the sound extracted by the determination means, a determination means for determining a subject associated with the main subject selected by the selection means, and an audio processing means for differentiating audio processing for the sound associated with the main subject and the sound of a subject determined by the determination means to be associated with the main subject from audio processing for a subject not determined by the determination means to be associated with the main subject.

The present invention makes it possible to record video and audio in line with the photographer's imagination.

FIG. 1 is a block diagram of an imaging apparatus according to a first embodiment. FIG. 2 is a block diagram of an imaging processing unit and an audio processing unit according to the first embodiment (during recording); FIG. 2 is a block diagram of an imaging processing unit and an audio processing unit according to the first embodiment (during post-processing); FIG. 4 is a diagram illustrating a main target selection method according to the first embodiment. FIG. 4 is a diagram showing an operation flow of a moving image recording sequence according to the first embodiment. FIG. 2 is a diagram illustrating an assumed scene in the first embodiment. FIG. 4 is a diagram illustrating the content of audio processing according to the first embodiment. FIG. 11 is a block diagram of an imaging processing unit and an audio processing unit according to a second embodiment. FIG. 11 is a diagram showing an operation flow of a video recording sequence according to the second embodiment. FIG. 11 is a diagram for explaining a problem of the second embodiment; FIG. 11 is a diagram illustrating a problem scene according to the second embodiment.

A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[First embodiment]
In this embodiment, an audio processing device included in an imaging device will be described with reference to FIGS. 1 to 3. FIG.

Figure 1 is a block diagram showing the configuration of an imaging device 100 according to the first embodiment.

The imaging unit 101 converts the optical image of the subject captured by the optical lens into an image signal using an imaging element, and performs analog-to-digital conversion and image adjustment processing using the image processing unit 102 to generate image data. The imaging optical lens may be a built-in optical lens or a removable optical lens. The imaging element may be a photoelectric conversion element such as a CCD or CMOS. The audio input unit 103 collects audio from the surroundings of the imaging device 100 using a built-in microphone or a microphone connected via an audio terminal, and performs various audio processing on the analog-to-digital converted audio in the audio processing unit 104 to generate audio data. The microphone may be either directional or non-directional. The memory 105 temporarily stores image data obtained by the imaging unit 101 and the image processing unit 102, and audio data obtained by the audio input unit 103 and the audio processing unit 104. The display control unit 106 displays images related to the image data obtained by the image processing unit 102, an operation screen of the imaging device 100, a menu screen, etc. on the display unit 107 or on an external display via a video terminal (not shown). The display unit 107 has a touch panel function, and the photographer can operate it to select menus and subjects.

The encoding processing unit 108 reads out the image data and audio data temporarily stored in the memory 105 and performs a predetermined encoding to generate compressed image data, compressed audio data, etc. Also, the audio data may not be compressed. The compressed image data may be compressed by any compression method, such as MPEG2 or H.264/MPEG4-AVC. The compressed audio data may also be compressed by a compression method such as AC3(A)AC, ATRAC, ADPCM, etc. The recording and reproducing unit 109 records the compressed image data, compressed audio data or audio data, and various data generated by the encoding processing unit 108 to the recording medium 110, and reads them from the recording medium 110. Here, the recording medium 110 includes any type of recording medium, such as a magnetic disk, optical disk, or semiconductor memory, as long as it is capable of recording image data, audio data, etc.

The control unit 111 can control each block of the imaging device 100 by transmitting control signals to each block of the imaging device 100 and the imaging unit 101, and is composed of a CPU and memory for executing various controls. The memory 105 used by the control unit 111 is a ROM for storing various control programs, a RAM for arithmetic processing, etc., and also includes a memory external to the control unit 111. The operation unit 112 is composed of buttons, dials, etc., and transmits instruction signals to the control unit 111 in response to user operations. In the imaging device of this embodiment, it is composed of a shooting button for instructing the start and end of video recording, a zoom lever for instructing optical or electronic zoom operation on an image, a cross key for various adjustments, a decision key, etc. The audio output unit 113 outputs audio data or compressed audio data reproduced by the recording and playback unit 109, or audio data output by the control unit 111 to a speaker 114 or an audio terminal. The external output unit 115 outputs compressed video data, compressed audio data, audio data, etc. reproduced by the recording and playback unit 109 to an external device. The data bus 116 supplies various data such as audio data and image data, as well as various control signals, to each block of the imaging device 100.

Here, we will explain the normal operation of the imaging device 100 of this embodiment.

In this embodiment, the imaging device 100 supplies power to each block of the imaging device from a power supply unit (not shown) in response to a user operating the operation unit 112 to issue an instruction to turn on the power.

When power is supplied, the control unit 111 checks whether the mode switch of the operation unit 112 is in a shooting mode, a playback mode, or the like, based on an instruction signal from the operation unit 112. In the video recording mode, the image data (video data) obtained by the imaging unit 101 and the image processing unit 102 and the audio data obtained by the audio input unit 103 and the audio processing unit 104 are saved as a video file. In the playback mode, the compressed image data recorded on the recording medium 110 is played back by the recording and playback unit 109 and displayed on the display unit 107.

In video recording mode, first, the control unit 111 sends a control signal to each block of the imaging device 100 to transition to a shooting standby state, and causes the following operations to be performed. The imaging unit 101 converts the optical image of the subject captured by the shooting optical lens into an image signal using an imaging element, and the image processing unit 102 performs image adjustment processing and the like to generate image data. The obtained image data is then sent to the display control unit 106, which displays it on the display unit 107. The user prepares for shooting while looking at the screen displayed in this way.

The audio input unit 103 converts analog audio signals obtained by multiple microphones into digital form, and processes the resulting digital audio signals to generate multi-channel audio data. The resulting audio data is then sent to the audio output unit 113, which outputs the audio from a connected speaker 114 or an earphone (not shown). The user can adjust the manual volume to determine the recording volume while listening to the audio output in this manner.

Next, when the user operates the record button on the operation unit 112 to send an instruction signal to start shooting to the control unit 111, the control unit 111 sends an instruction signal to start shooting to each block of the imaging device 100, causing them to perform the following operations.

The imaging unit 101 converts an optical image of a subject captured by a photographing optical lens into an image signal using an imaging element, and performs image adjustment processing and the like in the image processing unit 102 to generate image data. The obtained image data is then sent to the display control unit 106, which displays it on the display unit 107. The obtained image data is also sent to the memory 105.

The audio input unit 103 digitally converts analog audio signals obtained by multiple microphones, and processes the multiple digital audio signals obtained by the audio processing unit 104 to generate multi-channel audio data. The obtained audio data is then sent to the memory 105. Also, when there is only one microphone, the obtained analog audio signal is digitally converted to generate audio data, and the audio data is sent to the memory 105.

The encoding processing unit 108 reads the image data and audio data temporarily stored in the memory 105 and performs a specified encoding process to generate compressed image data, compressed audio data, etc.

Then, the control unit 111 combines the compressed image data and compressed audio data to form a data stream, which it outputs to the recording and playback unit 109. If the audio data is not compressed, the control unit 111 combines the audio data stored in memory 105 with the compressed image data to form a data stream, which it outputs to the recording and playback unit 109. The recording and playback unit 109 writes the data stream to the recording medium 110 as a single video file under file system management such as UDF or FAT. The above operations continue while shooting is in progress.

When the user operates the record button on the operation unit 112 to send an instruction signal to end shooting to the control unit 111, the control unit 111 sends an instruction signal to end shooting to each block of the imaging device 100, causing them to perform the following operations.

The imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 each stop generating image data and audio data. The encoding processing unit 108 reads out the remaining image data and audio data stored in memory and performs the specified encoding, and stops operating when it has finished generating compressed image data, compressed audio data, etc. If the audio data is not compressed, it naturally stops operating when it has finished generating the compressed image data.

The control unit 111 then combines these final compressed image data with the compressed audio data or the audio data to form a data stream, which it outputs to the recording and playback unit 109. The recording and playback unit 109 writes the data stream to the recording medium 110 as a single video file under file system management such as UDF or FAT. When the supply of the data stream stops, it completes the video file and stops the recording operation. When the recording operation stops, the control unit 111 sends a control signal to each block of the imaging device 100 to transition to a shooting standby state, and returns to the shooting standby state.

Next, in the playback mode, the control unit 111 sends a control signal to each block of the imaging device 100 to transition to the playback state, and causes the following operations to be performed. The recording and playback unit 109 reads out a video file consisting of compressed image data and compressed audio data recorded on the recording medium 110, and sends the read compressed image data and compressed audio data to the encoding processing unit 108.

The encoding processing unit 108 decodes the compressed image data and compressed audio data and transmits them to the display control unit 106 and audio output unit 113, respectively. The display control unit 106 displays the decoded image data on the display unit 107. The audio output unit 113 outputs the decoded audio data from a built-in or attached external speaker.

As described above, the imaging device 100 of this embodiment can record and play back images and audio.

In this embodiment, when obtaining an audio signal, the audio input unit 103 and audio processing unit 104 perform processing such as level adjustment of the audio signal obtained by the microphone. This processing may be performed at all times after the device is started, or may be performed after a shooting mode is selected, or may be performed after a mode related to audio recording is selected. Furthermore, in a mode related to audio recording, the above processing may be performed in response to the start of audio recording. In this embodiment, the above processing is performed at the timing when video recording begins.

Figure 2 is a block diagram showing an example of a detailed configuration of the imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 of the imaging device 100 of this embodiment.

The imaging unit 101 has an optical system such as an optical lens 201 that captures an optical image of a subject, and an imaging element 202 that converts the optical image of the subject captured by the optical lens 201 into an electrical signal (image signal). It also has an optical lens control unit 203 that has a known driving mechanism such as a position sensor and a motor for moving the optical lens 201. In this embodiment, the imaging unit 101 is described as having the optical lens 201 and the optical lens control unit 203 built in, but these may be detachable and replaceable optical lenses. For example, when a user inputs an instruction such as a zoom operation or focus adjustment by operating the operation unit 112, the control unit 111 transmits a control signal (drive signal) to the optical lens control unit 203 to move the optical lens. In response to this control signal, the optical lens control unit 203 checks the position of the optical lens 201 with a position sensor and moves the optical lens 201 with a motor or the like.

The image processing unit 102 performs various image quality adjustment processes on the image signal converted by the image sensor 202 in the image adjustment unit 221 to form image data, and transmits the image data to the memory 105 via the data bus 116. Based on the image data formed here, the control unit 111 performs various adjustments such as focus adjustment and light amount adjustment.

Furthermore, in this embodiment, the image processing unit 102 has various detection functions. The person detection unit 222 extracts features of the person's face, such as the eyes, nose, and mouth, from the image data formed by the image adjustment unit 221, and detects the position of the person in the image data and the size of the face. Then, by storing information on these features in the memory 105, it is possible to individually recognize the subject person based on the information. In addition, the person detection unit 222 has a person movement detection unit 223 that detects the movement of the lips and head, and a person speech detection unit 224 that determines whether the person is speaking or not. In addition, the image processing unit 102 has a main object selection unit 225 that selects which person, among the people detected by the person detection unit 222, is to be the main subject of the voice processing (hereinafter, also referred to as the main object or main object). The main object selection unit 225 selects the main object based on conditions determined by the control unit 111. The conditions for selecting the main object by the main object selection unit 225 will be described later. Furthermore, the image processing unit 102 has a conversation group detection unit 226. Conversation group detection unit 226 detects a person who is conversing with the person selected by main target selection unit 225 from among the people detected by person detection unit 222. The detection is determined based on the positional relationship between the people, the direction of the face, the movement, and the like. For example, conversation group detection unit 226 determines the subject closest to the main target subject as the person conversing with the main target (a related person). Also, for example, conversation group detection unit 226 determines the subject facing the direction of the main target's body, face, line of sight, and the like as the person conversing with the main target. Also, when the main target is moving, conversation group detection unit 226 determines the subject in the direction of the movement as the person conversing with the main target. This is because such a subject is considered to be conversing with the main target in the near future.

If the conversation group detection unit 226 determines that a person who is conversing with the main target has not conversed with the main target for longer than a predetermined time, the conversation group detection unit 226 determines that the person is not conversing with the main target (is not related to the main target). In other words, if a person who is conversing with the main target has not conversed with the main target for a predetermined time, the person is determined to be conversing with the main target even if it is determined that the person has not conversed with the main target.

Next, the audio input unit 103 and audio processing unit 104 will be described. The audio input unit 103 is a microphone 211 that converts audio vibrations into an electrical signal and outputs it as an audio signal. In this embodiment, the microphone 211 is a stereo type consisting of two channels, Lch/Rch, but it may be a mono type with one channel, or a configuration that holds multiple microphones with two or more channels. The A/D conversion unit 212 is a means for converting an analog audio signal obtained by the microphone 211 into a digital audio signal.

The voice processing unit 104 is a block that performs various voice processing on the voice signal converted by the voice input unit 103. In this embodiment, the voice processing unit 104 has a voice extraction unit 213, a voice adjustment unit 215, and a voice synthesis unit 217. The voice extraction unit 213 can extract (determine) a person's voice and other voices (hereinafter referred to as "non-human voices"). Furthermore, the person voice extraction unit 214 can extract the voices of people into individual voices of each person based on information from the person detection unit 222. For example, the person voice extraction unit 214 extracts individual voices based on the frequency, volume, and intonation of the voice. Furthermore, in the first embodiment, the control unit 111 can associate the subject with the voice based on the voice extracted by the person voice extraction unit 214 and the movement of the subject detected by the image processing unit 102. For example, the movement of the subject is the frequency of speech, the timing of speech, and the movement of the mouth.

The audio adjustment unit 215 can individually perform audio processing for each frequency band using level adjustment, equalizer, etc., on the audio extracted by the audio extraction unit 213. In particular, the conversation audio adjustment unit 216 performs adjustment based on information from the conversation group detection unit 226, emphasizing the extracted audio so that it is easier to hear, or reducing the volume so that it is harder to hear. The adjustment content will be described later. Furthermore, the audio synthesis unit 217 synthesizes the audio individually adjusted by the audio adjustment unit 215 and returns it to a single audio signal. The amplitude of the synthesized audio signal is then adjusted to a predetermined level by an auto level controller (hereinafter, ALC 219). With the above configuration, the audio processing unit 104 performs predetermined processing on the audio signal, forms audio data, and transmits it to the memory 105.

Figure 3 is a block diagram showing an example of another configuration of the image processing unit 102 and the audio processing unit 104 of the imaging device 100 of this embodiment. The difference between Figure 3 and Figure 2 is that the input sources of image data and audio data are different. In Figure 2, the image signal is from the imaging unit 101, and the audio signal is from the audio input unit 103. On the other hand, in Figure 3, the image and audio input sources input data stored in the memory 105. By using data temporarily stored (held) in the memory 105 in this way, it is possible to use the proposed method not only for processing during shooting, but also for post-processing after recording. Also, the main subject selection unit 225 can select a person to be the target of audio processing from a series of video data.

Here, an example of a method for selecting a main object by the main object selection unit 225 will be described with reference to FIG. 4. In this embodiment, the main object will be described as a person on whom the photographer is likely to focus. For example, in the case of FIG. 4(a), the focus mark 402 is a mark indicating the object on which the imaging device 100 is focused. In FIG. 4(a), the main object 401 and the focus mark 402 match, so the imaging device 100 recognizes the main object 401 as the main subject, and focuses on the main object 401. The main object selection unit 225 determines that this main object 401 is the main object. In this way, the person recognized as the main subject can be selected as the main object.

Figure 4(b) also shows a method of using a registered face image. The registered face image 403 is an image of a subject that has been registered in advance in the memory 105. The main subject selection unit 225 selects a person whose face is determined to match that of the image as the main subject.

Figure 4(c) shows a method for determining the main subject at the photographer's discretion. The photographer selects the main subject from among the people displayed on the display unit 107 by touching the touch panel of the display unit 107. The main subject selection unit 225 determines that the subject selected by the photographer is the main subject.

Also, FIG. 4(d) shows a method of using recorded video data. For example, when recorded video data 404 is stored in memory 105, main subject selection unit 225 determines person 405 who appears most frequently in video data 404 as the main subject. Alternatively, for example, main subject selection unit 225 may select a person who is frequently in focus.

Note that, for example, when the main object selection unit 225 selects a focused subject as the main object, even if the main object is out of focus, the main object selection unit 225 maintains the subject as the main object if the focus returns to the main object within a predetermined time. In other words, if the focus is out of focus from the main object for longer than a predetermined time, the main object selection unit 225 selects a new main object.

Next, the operation of the imaging device 100 of this embodiment will be described with reference to Figures 5 to 7.

Figure 5 is a flowchart showing an example of a series of video recording sequences of the imaging device 100. The processing of the imaging device 100 is realized by software recorded in a ROM (not shown) being loaded into the memory 105 and executed by the CPU. The processing of this flowchart is triggered by the imaging device 100 being powered on.

In step S501, the control unit 111 accepts an instruction to start video recording by a user operating the operation unit 112.

In step S502, the control unit 111 connects an audio path for recording audio.

In step S503, after the audio path is established, the control unit 111 performs initial setting of signal processing including the control described in this embodiment, and starts signal processing for video recording. The following description focuses on the recording sequence. Until the signal processing for video recording is completed, the control unit 111 records the image to be recorded in the video.

In step S504, the person detection unit 222 of the image processing unit 102 detects the subject.

In step S505, the main object selection unit 225 of the image processing unit 102 selects (determines) a main object from the subjects detected in step S504.

In step S506, the conversation group detection unit 226 of the image processing unit 102 determines the person (subject) who is conversing with the main subject selected in step S505.

In step S507, the voice extraction unit 213 of the voice processing unit 104 extracts the person's voice.

The audio adjustment unit 215 of the audio processing unit 104 performs an adjustment process on the audio extracted in step S507. The content of the audio adjustment process differs depending on whether the subject (person) of the audio extracted in step S507 is a subject (person) that belongs to the main conversation group. Details of the audio adjustment process will be described later using Figures 6 and 7, but will be explained simply in this flowchart.

In step S508, the audio adjustment unit 215 of the audio processing unit 104 determines whether the person whose voice was extracted in step S507 is a subject belonging to the main conversation group. If the person whose voice was extracted is a subject belonging to the main conversation group, the process of step S509 is executed. If the person whose voice was extracted is not a subject belonging to the main conversation group, the process of step S510 is executed.

In step S509, the audio adjustment unit 215 of the audio processing unit 104 adjusts the level of the extracted audio so that the volume is increased.

In step S510, the audio adjustment unit 215 of the audio processing unit 104 adjusts the level of the extracted audio so that the volume of the audio is reduced. In step S511, the audio adjustment unit 215 of the audio processing unit 104 performs adjustment processing other than the volume on the extracted audio.

In step S512, the voice synthesis unit 217 of the voice processing unit 104 synthesizes the extracted voices that have been individually adjusted to generate a single voice data.

In step S513, the control unit 111 determines whether to end the video recording. For example, the control unit 111 determines to end the video recording when the user operates the operation unit 112 to instruct the end of the video recording, or when it is determined that the remaining capacity of the recording medium 110 is low. If it is determined not to end the video recording, the process returns to step S504, and the sound recording sequence process continues. If it is determined to end the video recording, the process of step S514 is executed.

If it is determined that the video recording should not end, the process returns to step S504. That is, during video recording, the main subject and the people who are conversing with the main subject are repeatedly determined. This allows the control unit 111 to determine a different subject as the main subject, for example, even if the main subject disappears outside the angle of view or goes out of focus. Also, even if the number of people conversing with the main subject increases or decreases, the control unit 111 can determine the people who are conversing with the main subject accordingly.

In step S514, the control unit 111 disconnects the audio path and ends signal processing.

Here, we will explain the audio adjustment process using Figures 6 and 7.

Figure 6 shows an assumed scene for audio adjustment processing. Four subjects (people), person 602 to person 605, are present within the angle of view 601, and person 602 is conversing (speaking) with person 603, and person 604 is conversing (speaking) with person 605. In this case, if person 602 is selected as the main target of audio processing by main target selection unit 225, people 602 and 603 are detected as a conversation group 610 from the image data by conversation group detection unit 226. In this case, the voices of people 602 and 603 are adjusted to be emphasized as noteworthy voices, and the voices of people 604 and 605 are adjusted to be unnecessary voices that are not to be emphasized.

Figures 7(a) to (c) are diagrams showing the audio adjustment process. In Figure 7, person 602, person 603, and person 604 in Figure 6 are represented as person A, person B, and person C, respectively (person 605 is not shown).

Figure 7(a) shows the voice signals of persons A to C extracted by the person voice extraction unit 214. That is, signal 701 shows the extracted voice signal of person A, signal 702 shows the extracted voice signal of person B, and signal 703 shows the extracted voice signal of person C. In each signal, the sections with large amplitude indicate the period when the person is speaking (uttering) (voiced timing), and the sections with small amplitude indicate the period when the person is not speaking (silent timing). For example, comparing signals 704 and 705, since persons A and B are conversing, voiced timing and silent timing appear almost alternately. On the other hand, since person C is not the conversation partner of persons A and B, signal 706 does not alternate between voiced timing and silent timing as much as signals 704 and 705.

Figure 7 (b) shows the audio correction coefficient for each person. In this embodiment, a correction coefficient of 1.0 indicates that no level adjustment (gain adjustment) is performed. Furthermore, when the correction coefficient is greater than 1.0, processing is an audio adjustment process for emphasizing the audio to make it easier to hear (at a higher volume), and when the coefficient is less than 1.0, processing is a process for making the audio harder to hear (at a lower volume).

For example, a case will be described in which the conversation group detection unit 226 has determined that person A and person B are conversing during the period 710. In this case, since person A is the main target, the conversation sound adjustment unit 216 recognizes the voices of person A and person B as targets to be emphasized, and sets the correction coefficients for each voice to large values (coefficients 714 and 715). In this embodiment, the correction coefficients for the voices of person A and person B are set to the same value. This is because it is assumed that the user who is the photographer hears both voices equally. On the other hand, the conversation sound adjustment unit 216 sets the correction coefficient for the voice of person C, who is determined not to be conversing with person A, to a small value, so that the voice of person C is relatively difficult to hear (coefficient 716). In this way, the conversation sound adjustment unit 216 emphasizes the voices of the main target person A and person B, who is his conversation partner, and reduces the other voices. For example, the conversation sound adjustment unit 216 increases the gain and level for the voices of the main target person A and person B, who is his conversation partner, more than those for the other voices. This results in video data whose images and audio match the imagination of the user who filmed it.

Figure 7(c) shows the audio signal that has been adjusted based on the correction coefficient of Figure 7(b) described above. For example, if the audio adjustment by the conversation audio adjustment unit 216 is realized by gain adjustment, during the period 710, the audio of persons A and B who have been determined to be having a conversation (signals 724 and 725) will have a correction coefficient greater than 1.0, making them louder and easier for the user to hear. Meanwhile, the audio of person C who has not been determined to be having a conversation (signal 726) will have a correction coefficient less than 1.0, making them quieter and harder to hear. The extracted audio that has been individually adjusted in this way is synthesized by the audio synthesis unit 217, and as a result, only the conversation determined to be the subject of attention is generated as audio data that is easy to hear.

Note that in this embodiment, the sound related to the main subject is emphasized (corrected to be louder) and the sound unrelated to the main subject is made hard to hear (corrected to be quieter), but adjustments can be applied to only one of them. In other words, it is sufficient that the correction coefficients for the main subject (person) and the subjects (people) with whom it is speaking are larger than the correction coefficients for the other subjects.

The emphasis method used by the conversation voice adjustment unit 216 is also not limited to adjusting the overall gain as described above, but may be adjusted by frequency in the frequency band of human voices using an equalizer or the like.

[Second embodiment]
In the first embodiment, after selecting a main target, a conversation group of people who are conversing with the main target is detected based on their positional relationship with the main target and their movements, and the audio of the conversation group is emphasized or other unnecessary audio is suppressed, thereby obtaining audio data that makes the noteworthy conversation easy to hear.

In the first embodiment, the method of detecting a conversation group is based on the positional relationship, facial orientation, and movement of people who are conversing with the person selected by the main subject selection unit 225 from among the people detected by the person detection unit 222. In this way, in the first embodiment, the detection by the conversation group detection unit 226 is performed based on people present within the angle of view 601 of the imaging device 100.

Now, assume that person A, who is the main subject 602, and person B and person D (603, 606) within the angle of view 601 are detected as a conversation group as shown in Figure 10(a). If person B moves out of the angle of view due to a zoom or panning operation by the photographer, even if the conversation between person A, person B, and person D continues, person B will be removed from the conversation group when the next conversation group is detected, as shown in Figure 10(b). As a result, even if person B is participating in the conversation, the conversation group detection unit 226 does not determine that person B is part of the conversation group, and there is a risk that only person B's voice will not be emphasized, resulting in a conversation that is difficult to hear.

The second embodiment aims to maintain the conversation group in the state it was in before it moved out of the field of view, and continue to capture easy-to-listen audio, when it is determined that the conversation of the person who moved out of the field of view is continuing, even if at least one person in the conversation group who was within the field of view moves out of the field of view.

The second embodiment will be described in detail below with reference to the accompanying drawings. Note that the configuration of the imaging device 100 in FIG. 1 is the same as that of the first embodiment, so a description thereof will be omitted.

Figure 8 is a block diagram showing the detailed configuration of the imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 of the imaging device 100 of this embodiment. Note that blocks having the same functions as those in Figure 2 are assigned the same numbers, and descriptions are omitted.

The feature extraction unit 801 associates the voice extracted by the person voice extraction unit 214 with the person corresponding to that voice. For example, the feature extraction unit 801 associates the extracted voice with the corresponding person based on the voice characteristics and the movement of the subject within the field of view. For example, the voice characteristics are frequency, volume, and intonation. For example, the movement of the subject is the frequency of speech, the timing of speech, and the movement of the mouth. This association can improve the accuracy of identifying the speaker. This allows the control unit 111 to identify the speaker from the voice even if the person in the conversation group is out of the field of view.

The conversation group correction unit 802 determines whether a person outside the field of view is continuing a conversation based on the voice characteristics associated with the person acquired by the characteristics extraction unit 801. The control unit 111 corrects the conversation group based on this result and the detection result of the conversation group detection unit 226 to take into account the person outside the field of view.

In the second embodiment, the feature extraction unit 801 and the conversation group correction unit 802 are added to the block diagram shown in FIG. 2, but the operation is the same when the conversation group correction unit 802 is added to the block diagram shown in FIG. 3.

Next, the operation of the imaging device 100 of the second embodiment will be described with reference to Figures 9 and 11.

Figure 9 is a flow chart explaining a series of recording operations of the imaging device 100. In Figure 9, blocks that perform the same operations as in Figure 5 are given the same step numbers as in Figure 5. Here, an example of an assumed scene for the operation in Figure 9 will be explained first with reference to Figure 11.

Figure 11(a) shows the scene at the time when the photographer presses the record button. In the scene shown in Figure 11(a) (hereinafter referred to as the initial shooting scene), Person A, Person B, and Person D (602, 603, 606) are present within the angle of view. Person A is the main subject, and a conversation group containing the main subject is detected as consisting of three people: Person A, Person B, and Person D. Next, we will explain the scene when at least one person in the conversation group is out of the angle of view.

Examples of scenes in which a person in a conversation group is out of the field of view are shown in Figs. 11(b) to (e). Figs. 11(b) to (d) show scenes in which person B (603) is out of the field of view 601. Fig. 11(e) shows a scene in which all members of the conversation group are out of the field of view due to the cameraman panning the imaging device 100. The horizontal "V" characters written near the mouths of the people in each figure represent the vocalization state of each person, and the thickness of the line represents the volume of the voice and the frequency of participation in the conversation. Each scene in Fig. 11 is described as Fig. 11(a) as the initial shooting scene, Fig. 11(b) as scene b, Fig. 11(c) as scene c, Fig. 11(d) as scene d, and Fig. 11(e) as scene e. The person 602 appearing in each figure is described as person A, the person 603 as person B, and the person 606 as person D. The main subject of each scene is person A. In addition, the angle of view 601 in each diagram is the shooting angle of the imaging device 100, and the conversation group 610 is the conversation group.

The assumptions for each scene in Figures 11 (a) to (e) are as follows:

In scene b, person B is no longer in the field of view as in the initial shot, but continues to talk as if he were in the field of view.

Scene c shows a scene in which person B is out of the field of view and is not engaged in a conversation, as compared to the initial shooting scene. Note that in scene c, neither person A nor person D is facing the direction of person B.

In scene d, compared to scene b, person B has moved into the distance, but the conversation continues. In scene d, the voice of person B is input to the imaging device 100. Also, person A, who is within the angle of view, is facing the direction of person B, and is speaking louder.

Scene e shows a scene in which Person A, Person B, and Person D are out of the field of view compared to the initial shot scene. Note that in scene e, Person A, Person B, and Person D are continuing a conversation.

An example of an assumed scene for the operation in FIG. 9 has been described above with reference to FIG. 11. Hereinafter, the operation of the imaging device 100 will be described with reference to the flowchart in FIG. 9. In the description of this embodiment, steps S901 to S904 will be mainly described.

First, steps S501 to S507 are processed to detect people within the field of view, identify the main subject, detect people conversing with the main subject, and extract audio.

In step S901, control unit 111 determines whether the number of people conversing with the main target detected in step S506 matches the number of people in the conversation group associated by feature extraction unit 801 and conversation group correction unit 802. For example, control unit 111 makes this determination by taking the difference between the number of people in the conversation group detected in step S506 and the number of people in the current conversation group. If it is determined that the number of people has decreased, it means that there is a person in the conversation group associated by feature extraction unit 801 and conversation group correction unit 802 who is outside the field of view. If it is determined that the numbers match, processing of step S904 is executed. If it is determined that the numbers do not match, processing of step S902 is executed.

In step S902, conversation group modification unit 802 determines whether or not a conversation is ongoing between a person outside the field of view and a person within the field of view. If it is determined that a conversation is not ongoing between a person outside the field of view and a person within the field of view, control unit 111 performs processing from step S904 onward, treating the current conversation group as the detection result in step S506. If it is determined that a conversation is ongoing, processing in step S903 is executed.

In step S903, the control unit 111 modifies the person (subject) who is conversing with the main subject detected in step S506 so that the person outside the angle of view is included in the conversation group within the angle of view.

In step S904, the feature extraction unit 801 associates the voice with the person corresponding to that voice based on the voice extracted for each subject (person) extracted by the person voice extraction unit 214.

Here, using the above-mentioned scene, an example of the determination in step S902 of whether person B is continuing a conversation with person A and person D will be described.

In scene b, in steps S505 and S506 of FIG. 9, person D is identified as a person having a conversation with person A, who is the main subject. However, in step S901 of FIG. 9, it is found that person B, who belonged to the conversation group in the initial shooting scene, has moved out of the angle of view. Then, in step S902 of FIG. 9, the conversation group correction unit 802 determines that the person section is continuing a conversation with person A and person D. Therefore, in step 903 of FIG. 9, the control unit 111 adds person B to the subjects (people) having a conversation with person A, who is the main subject. That is, in scene b, the same conversation group as in the initial shooting scene is maintained.

Here, an example of determining whether person B is continuing a conversation with persons A and D will be described. When information from the feature extraction unit 801 indicates that there is no change in the volume or intonation of person B's voice and that the timing of speech during the conversation with persons A and D is in sync, the conversation group correction unit 802 determines that the conversation is continuing. In this case, the control unit 111 adds person B to the people (subjects) who are talking to person A, who is the main target. In addition, when the image processing unit 102 can determine the direction in which the subject is out of the field of view and the face direction of the subject, the conversation group correction unit 802 further determines whether the conversation is continuing based on the face direction of person A or person D within the field of view and the direction in which person B is out of the field of view. In other words, even if it is determined that the conversation is continuing based on the volume of the voice, bathing, and speech timing described above, if the face direction of person A or person D within the field of view does not match the direction in which person B is out of the field of view, the conversation group correction unit 802 determines that the conversation is not continuing.

In scene c, the voice of person B is not detected in scene b. In such a case, it is determined that person B is not participating in the conversation between person A and person D, and the control unit 111 does not modify the subject who is conversing with person A, who is the main subject, and leaves it as person D.

In scene d, person B has moved from the situation in scene b, moving away from people A and D, but the conversation continues. In this scene, person B's voice has become quieter, but the timing of his/her speech when conversing with people A and D is correct. Furthermore, while person B's voice has become quieter, person A's voice, in contrast, has become louder. From this information, conversation group modification unit 802 determines that person A and person B are having a conversation. In response, control unit 111 adds person B as a subject who is conversing with person A, who is the main subject.

In scene e, the photographer changes the imaging direction from the direction where person A, person B, and person D are located toward the fireworks. That is, person A, person B, and person D continue to talk, but person A, who is the main subject, has disappeared from the angle of view. However, according to the information of the feature extraction unit 801, the acquired voice contains the voice of person A, so in this case, the control unit 111 determines that person A is the main subject. In addition, according to the information of the feature extraction unit 801, the voices of person B and person D are also continuously detected, so the control unit 111 adds person B and person D as subjects who are talking with person A, who is the main subject. In this way, in a scene like scene e, no person is within the angle of view, but the voices of the conversation group are emphasized and recorded. Note that the control unit 111 may control the content of the conversation to be converted into text and displayed in a form such as a speech bubble, as shown in FIG. 11(f).

The above describes the operation of the imaging device 100 in the second embodiment.

The number of people conversing with the main subject detected in step S506 is based on the person detection within the angle of view in step S504, so in the initial shooting scene (Figure 11(a)), there are two people, person B and person D, and in Figure 11(b), there is one person, person D.

In the second embodiment, audio extraction is performed based on the detection results of the person detection unit 222 until a predetermined time has elapsed since the start of video recording, and thereafter based on the results of person detection 223 and information from the feature extraction unit 801.

As described above, according to the second embodiment, even if a person belonging to a conversation group moves out of the field of view, if the conversation is continuing, the conversation group can be corrected to an appropriate one.

The second embodiment has been described with reference to Figures 11(b), (c), and (d) with regard to the judgment of conversation continuation assuming that the cause of a person belonging to a conversation group moving out of the angle of view is not taken into consideration, but this may also be taken into consideration. For example, if the photographer operates the zoom of the lens 201 to cause a person belonging to a conversation group to move out of the angle of view, the control unit 111 may determine that the conversation is continuing without using information from the feature extraction unit 801 since the person has moved out of the angle of view unintentionally.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

[Other embodiments]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The present invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

Claims (17)

A detection means for detecting a subject from a video;
a selection means for selecting a main subject from the subjects detected from the moving image;
A determination means for determining a subject's voice from the video;
an association means for associating the subject detected by the detection means with the sound extracted by the determination means;
a determining means for determining a subject related to the main subject selected by the selecting means;
and an audio processing means for differentiating audio processing for audio associated with the main subject and audio of a subject determined by the determination means to be related to the main subject from audio processing for audio of a subject not determined by the determination means to be related to the main subject. The audio processing device according to claim 1, characterized in that the audio processing means adjusts the level of the audio associated with the main subject and the audio of the subject determined by the determination means to be related to the main subject differently from the level of the audio of the subject not determined by the determination means to be related to the main subject. The audio processing device according to claim 1 or 2, characterized in that the audio processing means sets a correction coefficient for the audio associated with the main subject and the audio of a subject determined by the determination means to be related to the main subject to be larger than a correction coefficient for the audio of a subject not determined by the determination means to be related to the main subject. The audio processing device according to claim 1 or 2, characterized in that the audio processing means increases the gain for the audio associated with the main subject and the audio of a subject determined by the determination means to be related to the main subject, more than the gain for the audio of a subject not determined by the determination means to be related to the main subject. The audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects a subject that is in focus in the video as the main subject. The audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects a main subject from the video based on an image recorded as the main subject. The audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects, as the main subject, a subject that appears most frequently among the subjects captured in the video. The audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines a subject related to the main subject based on the distance from the main subject. The audio processing device according to any one of claims 1 to 8, characterized in that the determining means determines that the subject closest to the main subject is the subject related to the main subject. The audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines that a subject facing the main subject is a subject related to the main subject. The audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines a subject related to the main subject based on the movement of the main subject. Further comprising image processing means,
12. The audio processing device according to claim 1, wherein the associating means associates the subject detected by the detection means with the audio extracted by the determination means based on the audio extracted by the determination means and the movement of the subject detected by the image processing means. The audio processing device according to claim 12, characterized in that the image processing means detects the frequency of speech, the timing of speech, or the movement of the mouth of the subject. The audio processing device according to any one of claims 1 to 13, characterized in that the determination means extracts the subject's audio based on the frequency, volume, and intonation of the audio. The audio processing device according to any one of claims 1 to 14, further comprising an imaging means for capturing the video. A method for controlling a sound processing device, comprising:
A detection step of detecting a subject from a video;
a selection step of selecting a main subject from the subjects detected from the video;
An extraction step of extracting a subject's voice from the video;
an associating step of associating the subject detected in the detecting step with the sound extracted in the extracting step;
a determination step of determining a subject related to the main subject selected in the selection step;
a sound processing step of processing the sound associated with the main subject and the sound of a subject determined to be related to the main subject in the judgment step, with respect to the sound of a subject not determined to be related to the main subject in the judgment step. A computer-readable program for causing a computer to function as each of the means of the voice processing device according to any one of claims 1 to 15.