JP4219129B2 - Television receiver - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a television receiver, and more particularly to a television receiver capable of creating character data based on a received television broadcast signal and displaying the character data together with a video.
[0002]
[Prior art]
Conventionally, various television receivers have been developed for creating and displaying character data based on received broadcast signals.
[0003]
For example, Japanese Patent Laid-Open No. 2000-206986 discloses a language information detection device for detecting a part of the outer skin of a face with a three-dimensional digitizer and inputting language information such as speech without necessarily assuming pronunciation. It is disclosed.
[0004]
Japanese Utility Model Laid-Open No. 2-53670 discloses a television receiver that can convert a received audio signal into character data and display the character data as a caption subtitle.
[0005]
Japanese Patent Application Laid-Open No. 9-134194 discloses an image recognition system for performing reading recognition based on image information captured by a camera. This image recognition system identifies a person based on the characteristics of a part of a face captured by a camera and enables a sales contract by telephone.
[0006]
[Problems to be solved by the invention]
However, the language information detection technique described in the above Japanese Patent Application Laid-Open No. 2000-206986 is not necessarily premised on speech, and a part of the outer skin of the face is detected using a three-dimensional digitizer and compared with a spectrum table. By doing so, it is only possible to recognize who is talking.
[0007]
The television receiver described in Japanese Utility Model Laid-Open No. 2-53670 receives various sounds included in the audio signal of the television broadcast signal at the same time. In addition, there may be cases where unnecessary speech, such as noise, is included. For this reason, if these voices are simply taken into the subtitles as they are, it becomes a list of character strings. Furthermore, since all voices are expressed as character strings, it is impossible to specify who is talking, whether it is the surrounding voice, narration, or BGM. There was a problem that the situation of the video would not be understood if the characters displayed in the subtitle field were chased after being displayed in the subtitle field at the bottom of the screen.
[0008]
In addition, since the image recognition system described in Japanese Patent Application Laid-Open No. 9-134194 performs reading speech recognition based on image information captured using a camera, the camera is an indispensable component, and a television broadcast signal There is a problem that the image information included in can not be used as it is.
[0009]
The present invention has been proposed in view of the above-described circumstances, and converts a sound emitted from an audio source into audio data based on a video signal included in a television broadcast signal, and displays it as a caption subtitle. An object of the present invention is to provide a television receiver capable of easily understanding broadcast contents.
[0010]
[Means for Solving the Problems]
The television receiver according to the present invention has the following features in order to achieve the above-described object.
[0011]
That is, the television receiver according to the present invention analyzes audio input discriminating means for discriminating an audio source based on a received video signal and audio emitted from the audio source discriminated by the audio input discriminating means. Sound analysis means, image data determined as a sound source by the sound input determination means, and storage means for storing the results analyzed by the sound analysis means in association with each other,
An audio source detection / determination unit that detects and discriminates an audio source that emits audio using the received video signal and audio signal with respect to the audio source determined by the audio input determination unit;
Spatial position detecting means for detecting the surrounding spatial position of the sound source determined by the sound input determining means , character converting means for converting the sound analyzed by the sound analyzing means into character data, and the character The character data converted by the conversion means is synthesized and inserted into the spatial position obtained by the spatial position detection means in the sound source determined by the sound source detection / discrimination means and displayed on the screen. Character / video composition means for
The sound source detection / discrimination means compares the sound source image stored in the storage means with the screen image, identifies the sound source displayed on the screen, and further confirms from the utterance position based on the received sound signal And finally identifying the voice source uttered by comparing the voice analysis data stored in the storage means with the utterance analysis result by the voice analysis means, and storing the utterance analysis result by the voice analysis means in the storage means. It is characterized by storing data .
[0012]
The television receiver according to the present invention is characterized in that the sound source detection / discrimination means further reconfirms a sound source uttered from a mouth or facial expression of vowel pronunciation .
[0013]
In the television receiver according to the present invention, the sound source detection / discrimination means further picks up an appropriate number of analysis points of the face and analyzes the movement of the face of the sound source based on the movement of the analysis points. The voice source is reconfirmed from the face motion and facial expression .
[0014]
The television receiver according to the present invention further comprises character frame adding means for adding a character frame to character data to be combined with the video signal.
[0015]
In addition, the television receiver according to the present invention further includes transmission-side character signal determination means for determining whether or not a transmission-side character signal is included in the television broadcast signal, and the character / video composition means includes When the transmission side character signal discriminating unit determines that the transmission side character signal is included in the television broadcast signal, the character data based on the transmission side character signal and the character converted by the character conversion unit It arranges so that it may not overlap with data.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of a television receiver according to the present invention will be described based on the drawings.
[0017]
1 to 4 show a television receiver according to an embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of the television receiver, and FIG. 2 shows a screen display example in the television receiver. FIG. 3A is a flowchart showing a procedure for human detection, FIG. 3B is a schematic diagram of a display screen when performing human detection, and FIG. 3C is a face image when performing human image analysis. FIG. 3D is a schematic diagram of a display screen when a spatial region search other than a human image is performed, FIG. 4A is a flowchart showing a procedure of speech recognition, and FIG. 4B is a character frame generation. It is a flowchart which shows the procedure of.
[0018]
As shown in FIG. 1, a television receiver according to an embodiment of the present invention includes an intermediate frequency amplifier circuit 1, a sound detection circuit 2, a sound input determination circuit 3, a sound spectrum analysis circuit 4, a character generation circuit 5, a sound generation circuit, Character conversion processing circuit 6, person / character distinction processing circuit 7, character frame / character composition circuit 8, character frame / speech position generation circuit 9, character / video composition circuit 10, transmission side character signal presence / absence determination circuit 11, audio amplification circuit 12, speaker 13, CPU (A) 14, CPU (B) 15, CPU (C) 16, CPU (D) 17, video detection circuit 20, video memory 21, person detection / discrimination processing circuit 22, outside person / space A position detection circuit 23, a character insertion position determination processing circuit 24, a video amplification output circuit 25, a video display device 26, and a person confirmation memory 40 are provided.
[0019]
In a television receiver, the scene at the beginning of a program is not known as when the switch is turned on. Therefore, at the same time as the sound is emitted, spectrum analysis is performed on the sound and the video is temporarily stored in the video memory 21. Then, while applying the learning effect of voice and video, it is classified into person A, person B, person C, narration, and other voices, and the spoken words are converted into character data, and person A utters. If it is a word, a character is inserted in the vicinity of the person A. In the case of a dialogue with the person B, characters are inserted in the vicinity of the person B like a comic balloon. In addition, characters are displayed on the screen for narration, and characters are displayed at the bottom of the screen for other voices.
[0020]
Note that the insertion position of the character is not limited to that described above, and can be any position on the screen, and it is preferable to select the insertion location that is easy for the viewer to distinguish and display the character.
[0021]
Specifically, as shown in FIG. 2, words uttered by each person are displayed as balloons in the vicinity of person A and person B appearing on the screen, and person C not appearing on the screen is emitted. The words are displayed at the bottom of the screen.
[0022]
Next, a method for converting a voice into character data, determining a character insertion position, and finding a person who is emitting the voice will be described.
As conventional techniques, although individual methods and examples are slightly different, Japanese Patent Laid-Open No. 9-134194 discloses an image recognition system, a reading system, and a reading system for reading. These technologies detect mouth movements, facial movements, tongue movements, tooth movements, etc. using external devices such as cameras, microphones, microphone digitizers, etc., and also recognize the generated voices. The person is surely authenticated. However, this publication does not disclose a method for identifying a person who is producing a voice.
[0023]
On the other hand, the television receiver according to the present embodiment can identify a person who is emitting sound using the received video signal and audio signal.
First, regarding the uttered sound, it is necessary to determine whether or not a person exists in the video in the scene where the sound is uttered. Moreover, the number of persons displayed in the video is not limited to one, and if a plurality of persons are displayed, the person who uttered the voice must be specified.
[0024]
Here, as shown in FIG. 3B, a case where a person A, a person B, and a person C are displayed on the screen and a person D exists outside the screen will be described as an example.
Since an actual video screen has various aspects, it is necessary to first check whether a person appears. Here, an image that can be determined as a human is stored in the person confirmation memory 40 in advance, and the person is identified by comparing this image with a video at the time of determination.
[0025]
In order to specify the person who is producing the sound, as shown in FIG. 3A, the image stored in the person confirmation memory 40 is compared with the person actually displayed on the screen, It is detected that A, person B, and person C appear (S41).
[0026]
Subsequently, using a reading system or the like, it is determined which of the persons displayed on the screen is uttering (S42). Here, with the reading system alone, detection cannot be performed when laughing or scolding, so the person position is re-verified based on the audio signal (S43).
That is, if the audio signal received by the television receiver is, for example, stereo broadcasting, it can be determined whether the sound is produced by the left or right person, and the audio spectrum is analyzed to determine whether it is a woman. Or it can also be discriminated whether it is a man.
[0027]
Subsequently, the comparison with the person confirmation memory 40 is performed again to reconfirm the person who is emitting the voice (S44). In other words, if you speak Japanese vowels “A”, “I”, “U”, “E”, “O”, the expression of the mouth and face changes. It can be re-verified by comparing with a person.
[0028]
As another method for reconfirming the person, there is a method of performing person image analysis. For example, as shown in FIG. 3C, a few points in the face, such as a person's eyes, nose, ears, cheeks, mouth, etc., are captured and analyzed to laugh. The person who is currently speaking is reconfirmed.
If the person is specified incorrectly, the process returns to the beginning (S41) and verification is performed again.
[0029]
If the person who is producing the voice can be determined to be, for example, the person A, then the screen position of the person A, the positions of the other person B, the person C, and the distance relationship are detected (S45). For example, as shown in FIG. 3D, the position of each person is specified as broken line frames Q, R, S, and T. Then, it is analyzed whether or not a prescribed character can be inserted into these broken line frames Q, R, S, and T, and it is determined at which position the character should be inserted (S45). This processing is performed by the person detection / discrimination processing circuit 22 and the person outside / space position detection circuit 23.
[0030]
Subsequently, when it is possible to determine which person the voice is uttered is matched with the spectrum obtained by the voice spectrum analysis circuit 4 (S46) and stored in the person confirmation memory 40 (S47).
By repeating the above-described procedure for each scene, a learning effect is produced and a person can be immediately identified.
[0031]
Next, a procedure for converting the actually generated voice into text will be described.
As shown in FIG. 4A, the audio signal detected by the audio detection circuit 2 is input to the audio amplification circuit 12 (S51). At this time, if it is determined that the sound is louder than the predetermined input signal, it is determined that a main person appears on the screen, and a signal is transmitted to the CPU (A) 14 to transmit the video signal to the video memory 21. Store (S52).
[0032]
In addition, a process using a voice filter is performed to convert the voice into text, and the number of voices is selected (S53), and the S / N ratio is improved and noise is determined (S54). Here, the actual sound is mixed with car sounds, wind sounds, and other noises depending on the video state, and it is difficult to transcribe them unless they are distinguished from human voices.
[0033]
Subsequently, a person is discriminated, and it is discriminated whether the voice is a female voice or a male voice (S55). Further, narration, B / G, and other discrimination are performed (S56).
[0034]
By the way, in order to cope with various video scenes, how to select and pick up the main character and other characters to be converted into characters is an important requirement for creating a good subtitle supermarket. For this reason, the selection filter process is performed on various sounds, the sound to be subtitled is analyzed by the sound spectrum analysis circuit 4 (S57), and the spectrum is registered (S58).
[0035]
Subsequently, the character generation circuit 5 converts the voice signal into characters (simple characters) (S59), and the voice / character conversion processing circuit 6 performs character conversion (S60). Actually, it is necessary to take various considerations such as simplifying redundant words and changing the display color of dialects, Japanese-made English, and new words or displaying them in katakana characters.
[0036]
Next, a procedure for inserting spoken words into the character frame and a procedure for inserting characters will be described.
It should be noted that the creation of graphics such as character frames can be easily programmed using a well-known technique when creating software for the microcomputer.
[0037]
As shown in FIG. 4B, when a character is inserted, or when the caption super switch is turned on, the character frame / blowing position generation circuit 9 is activated (S61).
Subsequently, in order to fit the character string obtained by the character generation circuit 5 into the character frame, the size of the character frame and the number of display stages are determined (S62), and the character frame size is determined based on the determination result (S63). ).
[0038]
Subsequently, for example, if the person A utters a voice, the size of the periphery of the person A (the frame Q and the frame R in FIG. 3D) is detected, and it is determined whether or not a character frame is inserted (S64). ).
Subsequently, character frame insertion / position determination processing is performed (S65). In this character frame insertion / position determination process, when it is determined that the character string is long with respect to the insertion position, the text is divided on the way, the first character string is displayed first, and then the subsequent character string is displayed at the same position. A determination is made to display.
[0039]
Subsequently, in fact, it should be judged at the same time, but the position where the person is uttering is detected, and the generated balloon is added in the vicinity of the position (S66, S67). Here, if the character frame is an obstacle in the actual video, the character frame may be hidden using the character frame ON / OFF switch (S68).
Subsequently, the character frame and the character string are synthesized by the character frame / character synthesis circuit 8 (S69), and the character frame / character string and the video are synthesized by the character / video synthesis circuit 10 (S70). I can do expressions like cartoon characters. FIG. 2 shows a screen in which character frames and character strings are combined with video.
[0040]
Note that even if the voice is uttered by the person A and the person B and the person C exist as other characters, if the character B is represented on the face of the person B or the person C, the received image is displayed. It will be disturbed by the characters and it will be difficult to appreciate. In addition, the characters A, B, and C are not always sitting still. For this reason, in the person / character distinction processing circuit 7 shown in FIG. 1, it is necessary to determine the character insertion position by reading and judging the operation of each person in a fuzzy manner.
[0041]
In general, if characters can be expressed on the head, the lateral movement of the person will change slightly, but there will be no influence on the person. Furthermore, when there are many people such as customers in the studio, it can be dealt with appropriately by changing the design of the software.
[0042]
In addition, when a necessary telop is displayed on the broadcasting station side, such as breaking news, earthquake information, typhoon information, etc., it is not possible to display characters superimposed on the telop. In this case, it is preferable to appropriately change the display position and character color of the character string or to stop displaying the character string based on the determination by the transmission side character signal presence / absence determination circuit 11 shown in FIG.
[0043]
The television receiver according to the present invention can be applied to a television receiver using either an analog signal or a digital signal. The speech / character conversion processing circuit 6 can include a multilingual translation function, and the television receiver according to the present invention can be applied to multilingual broadcasting.
[0044]
【The invention's effect】
Since the television receiver according to the present invention has the above-described configuration, the audio generated by the audio source is converted into audio data based on the video signal included in the television broadcast signal, and is displayed as a caption subtitle. Thus, the viewer can easily understand the broadcast content.
For example, it helps a person with hearing impairment and can recognize a missed voice as a letter even for a healthy person. Moreover, even if the surroundings are noisy and the voice from the television receiver cannot be heard, the conversation contents of the characters can be understood.
[0045]
In addition, install two television receivers, select different broadcast stations, one of them listens to normal audio, and the other one displays character strings. You can also.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of a television receiver according to an embodiment of the present invention.
FIG. 2 is a schematic diagram showing a screen display example in the television receiver according to the embodiment of the present invention.
3A is a flowchart showing a procedure for human detection, FIG. 3B is a schematic diagram of a display screen when performing human detection, FIG. 3C is a schematic diagram of a face image when performing human image analysis, d) is a schematic diagram of a display screen when a spatial region search other than a person image is performed.
FIG. 4A is a flowchart showing a voice recognition procedure, and FIG. 4B is a flowchart showing a character frame generation procedure.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Intermediate frequency amplifier circuit 2 Voice detection circuit 3 Voice input discrimination circuit 4 Voice spectrum analysis circuit 5 Character generation circuit 6 Voice / character conversion processing circuit 7 Person / character distinction processing circuit 8 Character frame / character synthesis circuit 9 Character frame / speech position Generation circuit 10 Character / video synthesis circuit 11 Transmission side character signal presence / absence determination circuit 12 Audio amplification circuit 13 Speaker 14 CPU (A)
15 CPU (B)
16 CPU (C)
17 CPU (D)
20 Video detection circuit 21 Video memory 22 Person detection / discrimination processing circuit 23 Outside person / space position detection circuit 24 Character insertion position determination processing circuit 25 Video amplification output circuit 26 Video display device 40 Person confirmation memory

Claims (5)

Audio input discrimination means for discriminating the audio source based on the received video signal;
A voice analysis means for analyzing a voice emitted from a voice source determined by the voice input determination means;
Storage means for storing the image data determined as the sound source by the sound input determination means and the result of analysis by the sound analysis means in association with each other;
An audio source detection / determination unit that detects and discriminates an audio source that emits audio using the received video signal and audio signal with respect to the audio source determined by the audio input determination unit;
Spatial position detection means for detecting the peripheral spatial position of the audio source determined by the audio input determination means;
Character conversion means for converting the sound analyzed by the sound analysis means into character data;
The character data converted by the character conversion means is inserted into the spatial position obtained by the spatial position detection means in the voice source determined by the voice source detection / discrimination means by synthesizing and inserting the video signal. Character / video composition means for displaying on
With
The sound source detection / discrimination means is
The sound source image stored on the storage means is compared with the screen image to identify the sound source displayed on the screen, and further confirmed from the utterance position based on the received sound signal,
Finally, specify the voice source uttered by comparing the voice analysis data stored in the storage means and the utterance analysis result by the voice analysis means,
A television receiver, wherein the storage means stores data of a speech analysis result by the voice analysis means . 2. The television receiver according to claim 1, wherein the sound source detection / discrimination means reconfirms a sound source uttered from a mouth or expression of vowel pronunciation . The sound source detection / discrimination means further picks up an appropriate number of analysis points of the face, analyzes the movement of the face of the sound source based on the movement of the analysis points, and re-synthesizes the sound source from the face motion and expression. The television receiver according to claim 1, wherein the television receiver is confirmed . Television receiver according to any one of claims 1 to 3, further comprising a character frame adding means for adding a character frame for the character data to be synthesized with the video signal. A transmission-side character signal determining means for determining whether or not the television broadcast signal includes a transmission-side character signal;
The character / video synthesizing unit, when the transmitting side character signal determining unit determines that the transmitting side character signal is included in the television broadcast signal, the character data based on the transmitting side character signal and the character 4. The television receiver according to claim 1 , wherein the television receiver is arranged so as not to overlap the character data converted by the conversion means.

Images