JP4488091B2 - Electronic device, video content editing method and program - Google Patents

Description

  The present invention relates to an electronic device capable of editing video content, a video content editing method in the electronic device, and a program thereof.

2. Description of the Related Art Conventionally, editing work for adding background music (BGM), sound effects, and the like to video content shot with a camcorder or the like has been performed. For example, in Patent Document 1 below, features (recording time and number of images) of an editing target video are extracted, and music optimal for the editing target video is automatically generated based on a predetermined instruction given by a user. A video signal editing apparatus for adding to a video to be edited is disclosed.
JP 2001-202082 A (paragraphs [0024], [0031], FIG. 2 etc.)

  However, in the technique as described in Patent Document 1, the original audio signal recorded in the original editing target video is erased by adding music to the editing target video. Therefore, depending on the scene, it may be possible to make the video to be edited more impressive by leaving the original audio signal rather than adding music. Therefore, user convenience is lost. In general, the user can manually select and edit which section of the video to be edited and which section to leave as the original audio signal. Very cumbersome and cumbersome.

  In view of the circumstances as described above, an object of the present invention is to provide an electronic device and video content editing capable of adding another audio signal while effectively leaving the audio signal in the original video content depending on the scene. It is to provide a method and a program.

In order to solve the above-described problem, an electronic apparatus according to an aspect of the present invention includes a first input unit, a second input unit, a first calculation unit, a second calculation unit, and a setting unit. And generating means.
The first input means inputs an image signal and a first audio signal constituting the first video content.
The second input means inputs a second audio signal different from the first audio signal.
The first calculation means detects a face image area in which a human face appears from the input image signal, and calculates a face evaluation value for evaluating the likelihood of the detected face image area.
The second calculation means detects a voice of the person from the input first voice signal, and calculates a voice evaluation value for evaluating the detected voice level.
The setting means, based on the calculated face evaluation value and voice evaluation value, for each image signal, a first weight coefficient indicating a weight of the first audio signal and a weight of the second audio signal A second weighting coefficient indicating is set.
The generation means generates a third audio signal obtained by mixing the first and second audio signals based on the set first and second weighting factors, and generates the third audio signal and the Second video content configured by the image signal is generated.
Here, the electronic device refers to, for example, a recording / playback device using a recording medium such as a PC (Personal Computer), an HDD (Hard Disk Drive) / DVD / BD (Blu-ray Disc), a digital video camera, a portable AV device, Electric appliances such as mobile phones and game machines. The first video content is, for example, video content recorded by a device such as a camcorder or video content received via a network. The second audio signal is, for example, an audio signal for BGM or sound effect.
With this configuration, the electronic device can change the weights of the first and second audio signals based on the face image and voice included in the first video content, and change the weights of the first and second audio signals from the first video content to the second video content. Video content can be generated. Therefore, compared to the case where another audio is simply inserted into the first video content, the editing effect is enhanced by leaving the person's voice as it is or inserting another audio depending on the scene. Thus, a more impressive second video content can be generated.

When the face evaluation value is equal to or greater than a first threshold and the voice evaluation value is equal to or greater than a second threshold, the setting means sets the first weight coefficient to be greater than the second weight coefficient. A large first value may be set.
When both the face evaluation value and the voice evaluation value are large, it is considered that there is a high possibility that a person appearing in the first video content is speaking. Therefore, in such a case, the person can be more impressed by making the first weighting coefficient as large as possible than the second weighting coefficient and emphasizing the voice of the person. Here, the first value may be set to 1.

The setting means determines the first weighting factor as the second weighting factor when the face evaluation value is less than the first threshold value and the voice evaluation value is less than the second threshold value. You may set to the 2nd value smaller than this.
When both the face evaluation value and the voice evaluation value are small, it is highly likely that no person appears in the first video content. Accordingly, in such a case, the ordinary weight scene of the first video content is made more attractive by enhancing the second audio signal by making the first weighting factor as small as possible than the second weighting factor. You can edit anything. Here, the second value may be set to zero.

When the face evaluation value is greater than or equal to the first threshold value and the voice evaluation value is less than the second threshold value, the setting means, depending on the face evaluation value and the voice evaluation value, The first weighting factor may be set larger than the second weighting factor.
When the face evaluation value is large and the voice evaluation value is small, a person's face appears in the first video content. Therefore, even if the voice is small, it is considered that the person is producing some kind of voice. Therefore, in such a case, the effect of the second audio signal is enhanced while enhancing the first audio signal by adding the weight of the first audio signal while adding the second audio signal. Can be added.

The setting means, when the face evaluation value is less than the first threshold and the voice evaluation value is greater than or equal to the second threshold, according to the face evaluation value and the voice evaluation value, The first weighting factor may be set smaller than the second weighting factor.
When the face evaluation value is small and the voice evaluation value is large, since the person is hardly reflected in the first video content, even if a person's voice is included, the voice is not related to the image. It is considered to be a voice. Therefore, in such a case, by increasing the weight of the second audio signal while leaving the first audio signal, the effect of the second audio signal is obtained while leaving the effect of the first audio signal. Can be increased.

The electronic apparatus may further include a storage unit that stores facial feature data indicating facial features of a specific person.
In this case, the first calculation means may be able to detect a face image area in which the face of the specific person appears based on the stored face feature data.
Thus, even when a plurality of human faces appear in the video content, the face of a specific person can be detected separately from the faces of other persons. Therefore, it is possible to more effectively execute the weighting coefficient setting processing for the first and second audio signals specialized for a specific person.

The electronic apparatus may further include a storage unit that stores voice feature data indicating a voice feature of a specific person.
In this case, the second calculation means may be capable of detecting the voice of the specific person based on the stored voice feature data.
Thereby, even when a plurality of human voices are included in the video content, the voice of a specific person can be detected separately from the voices of other persons. Therefore, it is possible to more effectively execute the weighting coefficient setting processing for the first and second audio signals specialized for a specific person.

The video content editing method according to another aspect of the present invention inputs an image signal and a first audio signal constituting the first video content, and outputs a second audio signal different from the first audio signal. Including typing.
From the input image signal, a face image area in which a person's face appears is detected, and a face evaluation value for evaluating the likelihood of the detected face image area is calculated.
The voice of the person is detected from the input first voice signal, and a voice evaluation value for evaluating the magnitude of the detected voice is calculated.
Based on the calculated face evaluation value and voice evaluation value, for each image signal, a first weight coefficient indicating the weight of the first sound signal and a second weight indicating the weight of the second sound signal. A weighting factor is set.
Based on the set first and second weighting factors, a third audio signal in which the first and second audio signals are mixed is generated, and is configured by the third audio signal and the image signal. Second video content to be generated is generated.
With this configuration, it is possible to leave the person's voice as it is or insert another audio depending on the scene, compared to a case where another audio is simply inserted into the first video content. And more impressive second video content can be generated.

A program according to still another aspect of the present invention includes a first input step, a second input step, a first calculation step, a second calculation step, a setting step, and a generation step. And to execute.
In the first input step, an image signal and a first audio signal constituting the first video content are input.
In the second input step, a second audio signal different from the first audio signal is input.
In the first calculation step, a face image area in which a person's face appears is detected from the input image signal, and a face evaluation value for evaluating the likelihood of the detected face image area is calculated.
In the second calculation step, the voice of the person is detected from the input first voice signal, and a voice evaluation value for evaluating the magnitude of the detected voice is calculated.
The setting step includes, based on the calculated face evaluation value and voice evaluation value, a first weighting coefficient indicating a weight of the first audio signal and a weight of the second audio signal for each image signal. A second weighting coefficient indicating is set.
The generation step generates a third audio signal obtained by mixing the first and second audio signals based on the set first and second weighting coefficients, and generates the third audio signal and the third audio signal. Second video content configured by the image signal is generated.

  As described above, according to the present invention, it is possible to add another audio signal while effectively leaving the audio signal in the original video content according to the scene.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of a recording / reproducing apparatus according to an embodiment of the present invention.
As shown in the figure, the recording / reproducing apparatus 100 according to the present embodiment includes an image signal input unit 1 and 3, an audio signal input unit 2 and 4, an input image processing unit 5, an input audio processing unit 6, and an image feature detection unit. 7, an audio feature detection unit 8, a recording unit 9, and a recording medium 10. The recording / reproducing apparatus 100 also includes a reproducing unit 11, an output image processing unit 12, an output audio processing unit 13, a user interface unit 14, a CPU (Central Processing Unit) 15, and a RAM (Random Access Memory) 16.

  The image signal input units 1 and 3 are various wired communication terminals and wireless communication units. As a terminal for wired communication, for example, an S terminal, an RCA terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High-Definition Multimedia Interface) terminal, an Ethernet (registered trademark) terminal or the like, a USB (Universal Serial Bus) ) Terminal, IEEE 1394 terminal, and the like. Examples of the wireless communication unit include wireless units such as a wireless LAN, Bluetooth (registered trademark), wireless USB, and wireless HDMI. However, the wired communication terminal and the wireless communication terminal are not limited to these. The image signal input units 1 and 3 input video content image signals into the recording / reproducing apparatus 100 via various cables and a wireless network, and supply them to the input image processing unit 5. Here, the video content is, for example, content shot with a camcorder or the like or content on the Internet.

  The audio signal input units 2 and 4 are also various wired communication terminals and wireless communication units, and are substantially the same as the above terminals and units except for the S terminal and the DVI terminal. The audio signal input units 2 and 4 input the audio signal of the video content into the recording / reproducing apparatus 100 via the various cables and the wireless network, and supply them to the input audio processing unit 6.

  The image signal input units 1 and 3 and the audio signal input units 2 and 4 are antenna input terminals for inputting an image signal and an audio signal included in the digital broadcast signal into the recording / reproducing apparatus 100 via an antenna (not shown). And a tuner or the like.

The input image processing unit 5 performs various signal processing such as digital conversion processing and encoding processing on the input image signal, and outputs it to the image feature detection unit 7 and the recording unit 9 as a digital image signal.
The input audio processing unit 6 performs various signal processing such as digital conversion processing and encoding processing on the input audio signal, and outputs the digital audio signal to the audio feature detection unit 8 and the recording unit 9.

The image feature detection unit 7 detects a face image (face image region) in which a human face appears from the image signal supplied from the input image processing unit 5 and evaluates the likelihood of the face image region. A face evaluation value is calculated.
The voice feature detection unit 8 detects a human voice from the voice signal supplied from the input voice processing unit 6 and calculates a voice evaluation value for evaluating the detected voice level.

  The recording unit 9 multiplexes the image signal supplied from the input image processing unit 5 and the audio signal supplied from the input audio processing unit 6 and records them on the recording medium 10.

  Examples of the recording medium 10 include a built-in recording medium such as an HDD and a flash memory, and a general recording medium such as an optical disk and a memory card. Examples of the optical disc include BD, DVD, and CD. The recording medium 10 stores various video contents, various programs, data, and the like. When the recording medium 10 is a built-in recording medium, the recording medium 10 executes an OS, face image detection processing, voice detection processing, learning processing of these detection processing, audio editing processing of video content, and the like. Various programs and data are stored. When the recording medium 10 is a general-purpose recording medium, the recording / reproducing apparatus 100 is separately provided with a built-in recording medium (not shown) for recording the various programs and data.

  The reproduction unit 11 reads and separates the multiplexed image signal and audio signal recorded on the recording medium 10, decodes the separated image signal and audio signal, and outputs the image signal to the output image processing unit 12. The audio signal is supplied to the output audio processing unit 13. Examples of compression formats of video signals and audio signals include MPEG (Moving Picture Expert Group) -2 and MPEG-4.

The output image processing unit 12 performs various signal processing such as analog conversion processing and OSD (On Screen Display) processing, and outputs the image signal to an external device such as a liquid crystal display connected to the recording / reproducing apparatus 100 or recording / reproducing. The data is output to a liquid crystal display built in the device 100.
The output audio processing unit 13 performs various signal processing such as analog conversion processing, and outputs the audio signal to the external device or the built-in liquid crystal display.

  The user interface unit 14 is, for example, an infrared signal light receiving unit of a remote controller, an operation button, a switch, a mouse, a keyboard, or the like, and inputs various commands by a user operation and outputs them to the CPU 15.

  The CPU 15 appropriately accesses the RAM 16 or the like as necessary, and comprehensively controls each block of the recording / reproducing apparatus 100. The RAM 16 is used as a work area for the CPU 15 and temporarily holds an OS (Operating System), a program, processing data, and the like.

  The external audio source 17 is an external device such as a PC or various AV devices, for example, stores a BGM (or sound effect) audio signal (hereinafter referred to as BGM audio) for insertion into video content, and has various interfaces. The audio signal is input to the CPU 15 through the CPU 15. However, the external audio source 17 may be a recording medium built in or attached to the recording / reproducing apparatus 100 such as the recording medium 10.

  Next, the operation of the recording / reproducing apparatus 100 configured as described above will be described.

  In the present embodiment, the recording / reproducing apparatus 100 can edit the video content and insert the BGM audio stored in the external audio source 17 into the video content. When inserting the BGM sound, the recording / reproducing apparatus 100 detects the face image from the image signal of the video content, detects the voice from the sound signal, and determines whether or not the BGM sound is inserted accordingly, as described above. To do. Among these, the recording / reproducing apparatus 100 performs a learning process as a pre-process for the detection of a face image. Hereinafter, this learning process will be described.

FIG. 2 is a diagram conceptually showing a learning process for detecting a face image.
As shown in the figure, the recording medium 10 of the recording / reproducing apparatus 100 includes face image sample data representing various human face image samples and non-face image sample data representing non-face image samples. A database is stored as learning data.

  The image feature detection unit 7 of the recording / reproducing apparatus 100 applies each of the sample image data stored in the face image sample database and the non-face image sample database to a feature filter, extracts individual face features, and extracts feature vectors (feature data). ) Is detected.

As shown in the figure, the feature filter is a filter that detects, for example, a certain rectangular portion in an image and masks a certain portion. By this feature filter, the positional relationship between the face eyes, eyebrows, nose, cheeks, etc. is detected as face features from the face image sample data, and the shape of the object other than the face, the shape of the object is detected from the non-face image sample data. The positional relationship of each component is detected as a non-facial feature. As the feature filter, in addition to the rectangular filter, for example, a separability filter that detects a circular feature, a Gabor filter that detects the positional relationship of each part of the face using an edge in a specific direction, and the like may be used. In addition to the feature filter, for example, luminance distribution information, skin color information, or the like may be used for detecting the facial features.
Here, the image feature detection unit 7 cannot recognize the size and position of the face region from the sample image data. Therefore, the image feature detection unit 7 changes the size of the feature filter when the most probable detection value is obtained when the size of the frame of the feature filter is changed and applied to the feature filter. Recognize and extract facial features. In addition, the image feature detection unit 7 recognizes the position of the feature filter when the most probable detection value is obtained as the position of the face region when all regions of the sample image data are scanned with the feature filter. Extract facial features.

The image feature detection unit 7 generates a multidimensional feature vector from each feature extracted from the face image sample data and the non-face image sample data. Then, the image feature detection unit 7 expresses this feature vector in a multidimensional vector space, and generates a discriminant function by statistical machine learning. The generated discriminant function is stored, for example, in the recording medium 10 or the like, and is used when a face image is detected from video content to be edited.
Further, instead of the discriminant analysis process using the discriminant function, for example, a discriminant analysis process using a machine learning method such as a support vector machine (SMV), Ada-boost, or a neural network may be executed. In this case, instead of the discriminant function, a processing module that executes the discriminating process is incorporated in the recording / reproducing apparatus 100. The same applies to processing related to the discriminant function in the following description.

  Next, in the present embodiment, a process in which the recording / playback apparatus 100 edits video content and inserts BGM data into the video content will be described.

FIG. 3 is a flowchart showing a flow of BGM insertion processing for video content in the recording / reproducing apparatus 100.
As shown in the figure, first, video content to be edited is read from the recording medium 10 or input from the image signal input unit 1 or 3 and the audio signal input unit 2 or 4. Subsequently, the CPU 15 extracts an image signal and an audio signal in a predetermined section (a predetermined number of continuous frames) from the video content (step 31). The extracted image signal of the predetermined section is supplied to the image feature detection unit 7, and the sound signal of the predetermined section is supplied to the sound feature detection unit 8.

  Subsequently, the image feature detection unit 7 detects a face image region from the image signal in the predetermined section using the discriminant function (step 32). FIG. 4 is a diagram conceptually showing the face image area detection processing. As shown in the figure, the image feature detection unit 7 applies an image signal in a predetermined section to the feature filter, extracts face features, and generates a multidimensional feature vector. Then, the image feature detection unit 7 introduces the value of each dimension of the feature vector into the variable of each dimension of the discriminant function, and determines whether the output of the discriminant function is positive or negative according to the face image region. Whether or not is included is determined.

  Then, the image feature detection unit 7 calculates a face evaluation value Tf for evaluating the probability of detection of the face image based on the output value of the discriminant function (step 32). The face evaluation value is, for example, a value representing the output value of the discriminant function as a percentage when a feature vector is generated based on predetermined clear face image data and input to the discriminant function.

  Subsequently, the voice feature detection unit 8 detects a section including a human voice from the voice signal of a predetermined section (step 34). FIG. 5 is a diagram conceptually showing the voice detection process. In the figure, the power of the audio signal in the predetermined section is shown. A waveform A in the figure shows a human voice, and a waveform B in the figure shows a voice other than the human voice.

  As shown in the figure, the voice feature detection unit 8 first sets a threshold Ath relating to voice power in order to remove the influence of noise. Then, when the average power in the predetermined section is larger than Ath, the voice feature detection unit 8 determines that the section is a voice section, and when smaller than Ath, the section is a non-voice section. Is determined. That is, in the figure, the audio signals other than the waveforms A and B are non-audio intervals.

  Among voice sections, human voice includes consonants, vowels, breath breaths, etc., and therefore has a feature that a continuous section of a predetermined power or higher is shorter than voices other than voice such as music. Using this feature, the voice feature detection unit 8 sets a threshold Tth relating to time, and when the average duration time equal to or greater than a predetermined power is smaller than Tth, the segment is a voice segment, which is larger than Tth. In this case, it is determined that the section is a non-voice section.

  Subsequently, the voice feature detection unit 8 calculates a voice evaluation value Tv based on the detected voice level (power level, amplitude) (step 35). The maximum power level is 1, and the voice power level is expressed as a percentage.

  Subsequently, the CPU 15 determines whether or not the face evaluation value Tf is equal to or greater than a predetermined threshold value Tfs (step 36). When the face evaluation value Tf is equal to or greater than the threshold value Tfs (Yes), the CPU 15 determines whether or not the voice evaluation value Tv is equal to or greater than the predetermined threshold value Tvs (step 37).

  When the voice evaluation value Tv is equal to or greater than the threshold value Tvs (Yes), the CPU 15 sets the BGM audio weight coefficient k to a predetermined weight k1 smaller than 0.5, and the audio signal weight coefficient of the video content. Set m to 1-k1. For example, k1 is set to 0, but is set to be as close to 0 as possible even when it is not 0.

  If the voice evaluation value Tv is less than the threshold value Tvs in step 37 (No), the CPU 15 sets the weighting factors k and m according to the face evaluation value Tf and the voice evaluation value Tv (step 39). . That is, neither of the weighting factors k and m is 0 or 1, but the weighting factor k is set smaller than the weighting factor m.

  When the face evaluation value Tf is less than the threshold value Tfs (No) in step 36, the CPU 15 determines whether or not the voice evaluation value Tv is equal to or greater than a predetermined threshold value Tvs (step 40). When the voice evaluation value Tv is equal to or greater than the threshold value Tvs (Yes), the CPU 15 sets the weighting factors k and m according to the face evaluation value Tf and the voice evaluation value Tv (step 41). That is, neither of the weighting factors k and m is 0 or 1, but the weighting factor k is set larger than the weighting factor m.

  When the voice evaluation value Tv is less than the threshold value Tvs in Step 40 (No), the CPU 15 sets the weight coefficient k to a predetermined weight k2 greater than 0.5, and sets the weight count m to 1−. Set to k2. For example, k2 is set to 1, but even if it is not 1, it is set to be as close to 1 as possible.

  Based on the weighting factors k and m set in this way, the CPU 15 edits the video content for each predetermined section (for each frame) of the video content and inserts the BGM audio input from the external audio source 17. (Step 43).

  The CPU 15 executes the above processing for all the predetermined sections of the video content or until the user or the like instructs to stop the processing (steps 44 and 45). The CPU 15 finally multiplexes the edited video content with the original image signal and records the new video content on the recording medium 10.

  FIG. 6 is a table showing the setting processing of the weighting factors k and m described above. As shown in the figure, four pattern weight coefficients are set depending on whether the face evaluation value and the voice evaluation value are equal to or higher than the threshold values Tfs and Tvs.

  FIG. 7 is a graph showing the relationship between the face evaluation value and voice evaluation value, the weight coefficients k and m, and each frame image of the video content. As an example, frames f1 to f6 shown in the figure show some frames of video content in which a state of a school sports day is recorded by a camcorder or the like.

  As shown in the figure, in the frames f1 and f2 of the video content, the face evaluation value is low (less than the threshold Tfs) because the face is too small and the face image area is not detected by the image feature detection unit 7. Further, in the sections of the frames f1 and f2, the voice evaluation value is low (less than the threshold value Tvs) because the image is taken from a distance and almost no human voice is collected. Therefore, in this section, the weight coefficient k of the BGM audio is set high, and the weight coefficient m of the content audio signal is set low. This makes it possible to edit a mediocre scene into a more attractive one.

  In frames f3 and f4, a person is photographed slightly up and the voice collected is also slightly louder. Therefore, in this section, weight coefficients k and m are set according to the face evaluation value and the voice evaluation value. . Thereby, the effect by BGM insertion can also be acquired at the same time, leaving a human voice. That is, when the face evaluation value is equal to or higher than the threshold value Tfs and the voice evaluation value is lower than the threshold value Tvs, the image feature detection unit 7 can emphasize the voice of the person appearing in the image by reducing the weight of the BGM sound. it can. Further, when the face evaluation value is less than the threshold value Tfs and the voice evaluation value is greater than or equal to the threshold value Tvs, the image feature detection unit 7 increases the weight of the BGM sound so that the BGM is more effective than the voice of a person unrelated to the image. Can enhance the effect.

  In the frames f5 and f6, since the person is photographed up to such an extent that the face can be clearly detected, the face evaluation value is high (threshold value Tfs or more). Further, since the power level of the detected voice is high, the voice evaluation value is also high (less than the threshold value Tvs). Therefore, in this section, the weighting factor k is set low and the weighting factor m is set high. Thereby, the person can be more impressed by emphasizing the voice of the person.

  As described above, according to the present embodiment, since the BGM sound is inserted into the video content based on the face evaluation value and the voice evaluation value, the audio signal in the original video content is effectively used according to the scene. BGM sound can be inserted while leaving Thereby, it is possible to make the video content more impressive and memorable as compared with the case where the BGM sound is simply inserted uniformly.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the present invention.

  In the above-described embodiment, the image feature detection unit 7 may detect not only a human face image but also an animal face image. The voice feature detection unit 8 may detect not only a human voice but also an animal voice.

In the above-described embodiment, the image feature detection unit 7 may not only detect a face image but also recognize a face image of a specific person. This face recognition process is executed after the face detection process by the discriminant function. For this face recognition process, an edge intensity image, a frequency intensity image, a high-order autocorrelation, a color conversion image, or the like can be used.
FIG. 8 is a diagram conceptually showing the face recognition process using the edge intensity image.
As shown in the figure, a grayscale image and an edge strength image are stored in the recording medium 10 or the like as feature data (dictionary pattern) of a person whose face is to be recognized. The image feature detection unit 7 extracts a grayscale image and an edge strength image as feature data from the detected face image. Then, the image feature detection unit 7 compares the extracted grayscale image and edge strength image with the stored grayscale image and edge strength image of the person who wants to recognize the face by pattern matching. Can be recognized. In this case, the image feature detection unit 7 may represent the recognition rate (matching rate) of the face image as a percentage to obtain a face evaluation value. The image feature detection unit 7 can also use the information in addition to the edge intensity image or the like when information on facial feature points such as eyes and nose is obtained.
With this process, for example, in the example of FIG. 7 described above, BGM is inserted into the video content according to the recognition rate of a specific person's face, such as detecting and recognizing only the face of the user's child from among many children. can do. Thereby, the edited video content can be made more impressive.

In the above-described embodiment, the voice feature detection unit 8 may not only detect a voice but also recognize a specific person's voice. In this voice recognition process, for example, the voice feature detection unit 8 analyzes the frequency of a voice signal of a person to be recognized, detects a spectral characteristic, stores it in the recording medium 10 or the like, and detects the detected voice spectrum. It is executed by comparing with characteristics (pattern matching). As the spectral characteristics, spectral peak frequencies, spectral intervals, and the like of the consonant part and the vowel part are used. In addition, since the breath interval and the like vary depending on the individual, the voice feature detection unit 8 may use information regarding the breath interval in combination with the spectral characteristics. In this case, the speech feature detection unit 8 may express the voice recognition rate (matching rate) as a percentage to obtain a voice evaluation value.
By this process, BGM can be inserted into the video content in accordance with the recognition rate of a specific person's voice, so that the edited video content can be made more impressive.

  In the above-described embodiment, the image feature detection unit 7 sets the weight of the BGM sound high when the face evaluation value is less than the threshold value Tfs and the voice evaluation value is equal to or greater than the threshold value Tvs. However, in this case, the image feature detection unit 7 may set the weight of the BGM sound low. Thereby, it becomes possible to leave the voices of both the person to be photographed and the photographer. If the voice can be recognized, the photographer's voice is recognized. If the voice evaluation value of the photographer's voice is equal to or greater than the threshold Tvs even if the face evaluation value is less than the threshold Tfs, the weight of the BGM sound is set. It may be set low. Thereby, a photographer's voice can be left more effectively and reliably.

  In the above-described embodiment, the recording / reproducing apparatus 100 does not perform the learning process for the voice detection process, but may naturally execute the learning process.

  In the above-described embodiment, an example in which the present invention is applied to a recording / reproducing apparatus has been described. However, the present invention is applied to other electronic devices such as a PC, a digital video camera, a portable AV device, a mobile phone, and a game device. Of course it is also possible to do.

It is a block diagram which shows the structure of the recording / reproducing apparatus which concerns on one Embodiment of this invention. It is the figure which showed notionally the learning process for the face image detection in one Embodiment of this invention. It is the flowchart which showed the flow of the BGM insertion process to a video content of the recording / reproducing apparatus which concerns on one Embodiment of this invention. It is the figure which showed notionally the detection process of the face image area | region in one Embodiment of this invention. It is the figure which showed notionally the voice detection process in one Embodiment of this invention. It is the table | surface which showed the setting process of the weighting factors k and m in one Embodiment of this invention. It is the graph which showed the relationship with each said frame image of the said face evaluation value and voice evaluation value, weighting coefficient k and m, and video content in one Embodiment of this invention. It is the figure which showed notionally the face recognition process using the edge strength image in other embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 3 ... Image signal input part 2, 4 ... Sound signal input part 5 ... Input image processing part 6 ... Input sound processing part 7 ... Image feature detection part 8 ... Sound feature detection part 9 ... Recording part 10 ... Recording medium 11 ... Playback unit 12 ... Output image processing unit 13 ... Output audio processing unit 14 ... User interface unit 15 ... CPU
16 ... RAM
17 ... External audio source 100 ... Recording / reproducing apparatus

Claims (9)

First input means for inputting an image signal and a first audio signal constituting the first video content;
Second input means for inputting a second audio signal different from the first audio signal;
First calculation means for detecting a face image area in which a person's face appears from the input image signal and calculating a face evaluation value for evaluating the likelihood of the detected face image area;
Second calculating means for detecting a voice of the person from the input first audio signal and calculating a voice evaluation value for evaluating the magnitude of the detected voice;
Based on the calculated face evaluation value and voice evaluation value, for each image signal, a first weight coefficient indicating the weight of the first sound signal and a second weight indicating the weight of the second sound signal. A setting means for setting a weighting factor;
Based on the set first and second weighting factors, a third audio signal is generated by mixing the first and second audio signals, and is configured by the third audio signal and the image signal. An electronic device comprising: generating means for generating second video content. The electronic device according to claim 1,
The setting means sets the first weight coefficient to be greater than the second weight coefficient when the face evaluation value is equal to or greater than a first threshold and the voice evaluation value is equal to or greater than a second threshold. Electronic device set to a large first value. The electronic device according to claim 2,
The setting means sets the first weighting factor to the second weighting factor when the face evaluation value is less than the first threshold value and the voice evaluation value is less than the second threshold value. Electronic device set to a second value smaller than The electronic device according to claim 3,
The setting means, when the face evaluation value is not less than the first threshold and the voice evaluation value is less than the second threshold, according to the face evaluation value and the voice evaluation value, An electronic device that sets the first weighting factor to be larger than the second weighting factor. The electronic device according to claim 3,
The setting means, when the face evaluation value is less than the first threshold and the voice evaluation value is greater than or equal to the second threshold, according to the face evaluation value and the voice evaluation value, An electronic device that sets the first weighting factor to be smaller than the second weighting factor. The electronic device according to claim 3,
Storage means for storing facial feature data indicating facial features of a specific person,
The electronic device is capable of detecting a face image area in which a face of the specific person appears based on the stored face feature data. The electronic device according to claim 3,
Storage means for storing voice feature data indicating the voice characteristics of a specific person;
The electronic device is capable of detecting the voice of the specific person based on the stored voice feature data. Input an image signal and a first audio signal constituting the first video content,
Input a second audio signal different from the first audio signal;
Detecting a face image area in which a person's face appears from the input image signal, and calculating a face evaluation value for evaluating the likelihood of the detected face image area;
Detecting the voice of the person from the input first audio signal, and calculating a voice evaluation value for evaluating the magnitude of the detected voice;
Based on the calculated face evaluation value and voice evaluation value, for each image signal, a first weight coefficient indicating the weight of the first sound signal and a second weight indicating the weight of the second sound signal. Set the weighting factor,
Based on the set first and second weighting factors, a third audio signal is generated by mixing the first and second audio signals, and is configured by the third audio signal and the image signal. A video content editing method for generating second video content. Electronic equipment,
Inputting an image signal and a first audio signal constituting the first video content;
Inputting a second audio signal different from the first audio signal;
Detecting a face image area in which a person's face appears from the input image signal, and calculating a face evaluation value for evaluating the likelihood of the detected face image area;
Detecting a voice of the person from the input first audio signal and calculating a voice evaluation value for evaluating the magnitude of the detected voice;
Based on the calculated face evaluation value and voice evaluation value, for each image signal, a first weight coefficient indicating the weight of the first sound signal and a second weight indicating the weight of the second sound signal. Setting a weighting factor;
Based on the set first and second weighting factors, a third audio signal is generated by mixing the first and second audio signals, and is configured by the third audio signal and the image signal. Generating the second video content.

Images