JP4064902B2 - Meta information generation method, meta information generation device, search method, and search device - Google Patents

Description

  The present invention relates to a search apparatus that generates meta information describing characteristics of content data including video data and audio data, and searches for a desired scene from the content data using the meta information.

  With the spread of digital broadcasting such as CS and BS, and the spread of large-capacity recorders such as DVDs and HDDs, when you want to receive and store a large amount of digital broadcasting programs and view your favorite scenes from them Video viewing modes such as searching and viewing are becoming possible.

  Conventionally, there has been a great demand for searching and playing back a desired video scene by specifying a search condition related to a scene in the video content. For example, MPEG7 (Moving Pictures) is a standard for meeting this demand. Experts Group part 7) has already been proposed. MPEG7 is a standard for describing features relating to video content as meta-information. For example, by registering video content and its meta-information in a database, video content satisfying a certain condition of meta-information is locally This is a technology for searching within a system or via a network, and was standardized in 2001 by W3C. The meta information used when searching for video content is information (information related to information) describing the characteristics of the video content.

  MPEG7 is also a standard for XML (Extensible Markup Language). XML is a standard meta information description language with flexible extensibility and cooperation. (A1) Since the structure and display style of content data can be defined separately, the data structure can be changed and the content data can be Easy to use. (A2) Since the structure of content data can be strictly defined using a tag, it is suitable for a high-accuracy condition search using a tag. There are features such as. In addition, there is a feature that a schema (data structure definition) is not necessarily defined for content data.

  Here, as a method of creating a keyword to be added to video content, a method of manually creating a keyword is generally used. However, in recent years, these recognition results for video content are obtained by improving speech recognition and image recognition techniques. In some cases, a method of automatically extracting keywords using this method has been used. Automatic keyword extraction has the great advantage of saving the effort of creating keywords and tags.

  A technique of automatically extracting keywords from video content using voice recognition or image recognition and searching for video content is also disclosed (see, for example, Patent Document 1). This is because a keyword obtained as a result of voice recognition of voice information is compared with a keyword registered in advance in a database or the like in association with each content data, and if the two match, the content data is used as a search result. I will return it. This method assigns a keyword to each content in advance and stores it in a database. The user designates the keyword in the form of sound or image, and returns a search result that matches the keyword designated from the content of the database. To do. Therefore, keywords are automatically extracted for the search key, but keywords associated with content stored in the database are not assumed to be automatically extracted.

  On the other hand, by expanding the keyword automatic extraction function described above, it is also possible to apply keywords to each scene of video data and add keywords. This can be achieved, for example, by the following steps.

  (Step 1) One representative video scene is selected and associated with each stream section constituting the video content. (Step 2) For each video scene selected in Step 1, a keyword is extracted by performing voice recognition and image recognition on an audio stream or an image stream constituting a stream section corresponding to the video scene. (Step 3) The keyword obtained in this way is assigned as meta information to the video scene.

  However, there are some problems in applying the above method. First, the precondition that the speech recognition result maintains a recognition accuracy of a certain level or higher, that is, the reliability of speech recognition is high. If there is little noise such as news narration, there will be no problem, but if there are many sound elements in the stream section that cause noise for speech recognition (speech recognition) like BGM, The reliability of the speech recognition result is lowered.

  Here, when the keyword information extracted under the condition that the reliability of voice recognition is low is registered as meta information accompanying each scene, and the user performs a scene search by specifying a keyword, the following problems may occur. (B1) Missing search: As a result of recognition error, different keywords are extracted and registered, and video scenes that should be originally searchable with the specified keywords are not hit. (B2) Incorrect search: As a result of recognition error, different keywords are extracted and registered, and it is possible to search for those that should not be originally searched with the specified keyword. Of these two problems, a search omission is likely to occur in a tag search that specifies a tag condition such as XML and performs a high-accuracy search, and an erroneous search includes a keyword that is specified from the full text. It is more likely to occur in keyword searches that are more lenient than tag searches, such as searching. For this reason, there is a demand for a method of reducing the above-mentioned search omission and erroneous search problems by properly using tag search or keyword search according to the situation.

  Furthermore, in a section where the reliability of voice recognition is low, a search omission is likely to occur even in a keyword search. Therefore, a method of switching to a search method with more relaxed conditions is also desired.

  As described above, meta information for video search is described using an XML format represented by MPEG7, stored in a database together with content data, and video data that meets conditions within the local system or via a network There is a great need to search and display a desired video scene by searching for an internal scene and to reproduce it by selecting it.

On the other hand, when performing tag search or keyword search by keyword specification based on the speech recognition result, if there are many sound elements that cause noise for speech recognition (utterance recognition) like BGM, In general, the reliability of the speech recognition result decreases in the section. That is, if a keyword obtained as a result of speech recognition is used as meta information for searching a video scene (tag search, keyword search), a search omission or an erroneous search is likely to occur. In particular, in a section where the reliability of voice recognition is low, a search omission problem is likely to occur even in keyword search.
JP 2001-229180 A

  As described above, conventionally, when a keyword recognized from audio corresponding to video content is used as meta information of the video content, the reliability of an audio recognition result including noise such as BGM is particularly low. When a keyword obtained as a result of speech recognition of speech in a section is used as meta information, there is a problem that a search omission or an erroneous search is likely to occur.

  Therefore, in view of the above problems, the present invention is a meta data of content data that can avoid omission of search or erroneous search as much as possible even when content data composed of video data and audio data includes a section with low reliability of voice recognition. It is an object of the present invention to provide a meta information generation method and apparatus for generating information, and a search method and apparatus for searching content data including a desired scene using the meta information.

  (1) The present invention generates meta information composed of a plurality of element data describing characteristics of content data including video data and audio data, and (a) the content data divided into a plurality of sections. For each of the plurality of sections, a speech text that is a speech recognition result of the speech data of the section and a telop text that is a character recognition result of the telop included in the video data of the section are obtained, and (b) from the speech text A keyword included in the speech text and a category to which the keyword belongs, and (c) for each of the plurality of sections, when the same sound word is included in both the speech text and the telop text obtained from the section. It is determined that the confidence level for the speech text in the section is high, and when the homophone is not included (D) for each of the plurality of sections, at least first element data describing an identifier of the section and a second describing the speech text Element data, third element data describing the reliability of the speech text, and, when the keyword is included in the speech text, a fourth description describing the keyword and the category to which the keyword belongs in association with each other. The meta information including the element data is generated.

  (2) In the present invention, (a) content data including video data and audio data is stored in the first storage means, and (b) each of the plurality of sections of the content data divided into a plurality of sections. And corresponding to each is meta information composed of a plurality of element data describing characteristics of content data in one of the plurality of sections, and the plurality of element data includes an identifier of the section. 1st element data describing the text, 2nd element data describing the speech text that is the speech recognition result of the speech data in the section, and describing either one of high or low reliability for the speech text The fourth element data describing the third element data, the keyword extracted from the speech text, and the category to which the keyword belongs are described in association with each other. A plurality of pieces of meta information data corresponding to each of the plurality of sections including data are stored in the second storage means, and (c) when the first character string and the second character string are designated as search conditions The meta information data having a high reliability among the plurality of meta information data stored in the second storage means is set as a search target, and from the search target, the category of the same sound as the first character string and the first The meta information data including the fourth element data in which the character string of 2 and the keyword of the same sound are described in association with each other is searched. (D) When only the second character string is designated as the search condition among the first character string and the second character string, the sounds that are the same as or similar to the second character string, respectively. A plurality of third character strings are obtained, and the meta information data stored in the second storage unit is set as a search target, and from the search target, the voice text includes the plurality of third character strings. Search meta information data including any of. (E) Content data in a section corresponding to the identifier included in the meta information data searched in (c) and (d) is searched from the content data stored in the first storage means.

  According to the present invention, when searching for a desired scene from content data including video data and audio data, it is possible to avoid a search omission and an erroneous search.

  Before describing the embodiment of the present invention, XML will be briefly described. Here, for the sake of simplicity, meta information (video meta information) used when searching for content data will be described as an example. Note that the structure of content data can be similarly expressed in XML. In this case, at least a tag that can specify the time from the start of the video is included.

  The content data includes an audio stream (audio data) that is audio data continuous with an image (video) stream (video data).

  FIG. 6 shows an example of video meta information described in XML. In XML, tags are used to represent structures. There are a start tag and an end tag. By enclosing the structural information components with the start tag and end tag, it is possible to clearly identify the character string (text) delimiter and which structural element the text belongs to. Can be described.

  Here, the start tag is an element name closed with symbols “<” and “>”, and the end tag is an element name closed with symbols “</” and “>”. The content of the component following the tag is a text (character string) or a repetition of a child component. Further, attribute information such as “<element name attribute =“ attribute value ”>” can be set in the start tag. A component that does not include text, such as “<meta information> </ meta information>”, can also be expressed as “<meta information />” as a simple notation.

  The video meta information shown in FIG. 6 has an element set starting from the “section” tag as a child element with the element starting from the “section information” tag as a root. As child elements of each “section” tag, there is an element set starting from tags such as “start time”, “end time”, “voice text”, “telop”, “player name”, and the like. As attributes of the “section” tag, there are an “ID” attribute and a “reliability” attribute. Further, for example, an element starting from the “player name” tag can have one text (character string) as a text value, such as “Sato”. Each element can also be expressed in a multi-layer structure.

  Hereinafter, embodiments of the invention will be described with reference to the drawings. FIG. 1 shows an example of a schematic configuration of the entire system including a search device 11 according to this embodiment. The search device 11 and a playback device 7 are connected via a predetermined network 10.

  The search device 11 includes a video meta information generation unit 1, a content data storage unit 2, a video meta information storage unit 3, a content data search unit 4, a video meta information search unit 5, and a word dictionary 6. ing.

  The content data storage unit 2 stores a plurality of content data. The content data is composed of one or a plurality of stream sections. Each stream section has a section ID for identifying each content section and information such as start time and end time of each stream section in a description format such as XML. It is described using and stored in association with content data.

  The video meta information generation unit 1 generates video meta information used when searching the content data from the content data stored in the content data storage unit 2. The video meta information generated by the video meta information generation unit 1 is stored in the video meta information storage unit 3.

  The content data search unit 4 generates a query for searching video meta information for obtaining content data (video) satisfying the search condition transmitted from the playback device 7 using words registered in the word dictionary 6. And output to the video meta information search unit 5. Also, using the section ID and start / end time obtained by the video meta information search unit 5, the content data corresponding to the section ID and start / end time is searched from the content data storage unit 2. The content data obtained here is transmitted to the playback device 7 via the network 10.

  The playback device 7 includes a search request specifying unit 8 and a playback unit 9. The search request specifying unit 8 is for inputting search conditions for desired content data. The search condition is transmitted to the content data search unit 4 of the search device 11 via the network 10. The playback unit 9 plays back the content data transmitted as a search result from the content data search unit 4.

  FIG. 2 shows a configuration example of the video meta information generation unit 1. The video meta information generation unit 1 includes a demultiplexer unit 101, a voice recognition unit 102, a telop recognition unit 103, a keyword matching degree determination unit 104, a scene information extraction unit 105, a data generation unit 106, and a recognition dictionary 107. Each component of the video meta information generating unit 1 as shown in FIG. 2 can be realized by software.

  The content data stored in the content data storage unit 2 is input to the video meta information generation unit 1. The input content data is separated into an audio stream and an image stream by the demultiplexer unit 101. The audio stream is input to the audio recognition unit 102, and the image stream is input to the telop recognition unit 103. Further, the content data input to the video meta information generation unit 1 is input to the scene information extraction unit 105. The scene information extraction unit 105 extracts the section ID, start time, and end time of each stream section included in the content data from the content data, and outputs these to the data generation unit 106.

  The voice recognition unit 102 performs voice recognition of the voice stream using the words registered in the recognition dictionary 107, and the voice text as the voice recognition result, the words (keywords) included in the voice text, Output keyword categories. The keyword extracted here is a keyword that matches a word registered in advance in a predetermined category. The voice recognition unit 102 stores in advance a word dictionary in which a plurality of words are registered in advance for each category and a rule for determining whether or not the keyword is a keyword. By referring to the word dictionary and rules, the keyword and the category to which the keyword belongs are obtained from the speech text.

  The telop recognition unit 103 performs character recognition on a telop in the image stream, and outputs a telop text as a telop recognition result, a word (keyword) included in the telop text, and its category. The keyword extracted here is a keyword that matches a word registered in advance in a predetermined category. The telop recognition unit 103 stores in advance a word dictionary in which a plurality of words are registered in advance for each category and rules for determining such keywords. By referring to this word dictionary and rules, a keyword and a category to which the keyword belongs are obtained from the telop text.

  The keyword matching degree determination unit 104 compares the speech text output from the speech recognition unit 102 with the telop text output from the telop recognition unit 103 for each stream section constituting the content data, and the speech in the speech recognition unit 102 The speech recognition reliability for the recognition result (speech text) is obtained. For example, if both text and telop text contain the same character string (same sound (match reading)) (hereinafter referred to as a common word), it is considered highly reliable. The reliability is determined as “high”, and when the common words that match each other are not included in both the voice text and the telop text, the voice recognition reliability is determined as “low”. At this time, the telop text is once converted into hiragana by the dictionary and then compared with the hiragana voice text.

  The keyword matching degree determination unit 104 outputs the speech recognition reliability and the common word for each stream section. Although the output common word is written in hiragana, it may be unified into kanji notation based on a dictionary.

  The data generation unit 106 determines the speech text output from the speech recognition unit 102, the keyword, the category of the keyword, the telop text output from the telop recognition unit 103, the keyword, the category of the keyword, and the keyword matching degree. Based on the voice recognition reliability and common language output from the unit 104, the section ID and start / end times output from the scene information extraction unit 105, video meta information data as shown in FIG. 6 is generated. .

  As shown in FIG. 6, the video meta information includes, for each stream section, the section ID extracted from the stream section, the voice recognition reliability (reliability), the start time, the end time, the voice text, the telop text, and the common language. , Keywords, etc. are described. Video meta information corresponding to one stream section is described as a component starting from a “section” tag.

  The section ID and the voice recognition reliability (reliability) are described as attributes of the “section” tag. The start time and end time are described as components starting from the corresponding tag names. The voice text is described as an element value of a tag name (element name) “voice text”. The telop text is described as an element value of a tag name (element name) “telop”. The common language is described as an element value of a tag name (element name) “common tag value”. A keyword in a speech text or a telop text (here, a keyword that matches a word registered in a predetermined category) is described as an element value having a category to which the keyword belongs as a tag name.

  The meta information corresponding to each stream section is expressed as an XML format represented by, for example, MPEG7. MPEG7 is an XML-compliant standard that is becoming popular as a meta information standard for video data. Here, the meta information is expressed in the XML format as shown in FIG. This is not MPEG7 compliant, but there is no problem in the following description.

  Next, the processing operation of the video meta information generation unit 1 will be described with reference to the flowchart shown in FIG. The content data input here is assumed to be a video stream in which an audio stream and an image stream are multiplexed (multiplexed) as defined by the MPEG standard (MPEG2 or MPEG4). However, it is not limited to this.

  First, the demultiplexer unit 101 demultiplexes the input content data into an audio stream and an image stream (step S1). For simplicity, the image stream obtained here is composed of three sections obtained by determining two video scene switching points by video recognition processing, for example, as will be described later. It is assumed that three sections (stream sections) shown in 4 (c) and voice (shown here as text for convenience) are included. The section IDs of the three stream sections are “1”, “2”, and “3”, respectively. In addition, about the division | segmentation method of this area, you may divide an area in the silence part of an audio | voice, and may divide an area in the arbitrary positions by a user.

  Next, the voice recognition unit 102 performs voice recognition on the obtained voice stream and obtains text (voice text) corresponding to the voice data in the stream (step S2). Normally, a measure for increasing the speech recognition rate is performed by obtaining a word that most closely matches a phoneme combination candidate in speech data using a speech recognition dictionary (recognition dictionary 107). As such a voice recognition technique, a conventional technique may be used, and since it is not the gist of the present invention, description thereof is omitted. The speech text corresponding to each stream section (FIG. 4 (a) to FIG. 4 (c)) thus obtained is shown in FIG. 5 (a) to FIG. 5 (c). In addition, the voice recognition unit 102 extracts the keyword from the voice text if there is one.

  In parallel with the voice recognition process, the telop recognition unit 103 displays a predetermined area (for example, the lower ¼ area of the screen) in which the telop of each image frame of the obtained image stream can be displayed. Character recognition is performed to obtain a telop text (step S3).

  The telop recognition unit 103 performs video recognition processing on the image stream, determines a video scene switching point, and sets the switching point of the video scene as a segment break in the image stream. The video scene switching point is determined, for example, as a video scene switching point when a pixel value, luminance, color, or the like between image frames changes to a predetermined threshold value or more. Alternatively, it may be determined by an appropriate method based on the audio stream delimiter or telop recognition delimiter. Here, it is assumed that it is determined that there are two video scene switching points and that there are three stream sections. FIGS. 4A to 4C show images obtained by converting the audio and telop in the stream section of the video recognition result corresponding to each stream section into text. Further, telop recognition is performed on each obtained stream section to extract text (telop text). An existing technique for recognizing the telop and extracting text data may be used (see, for example, JP-A-2001-285716). The telop text corresponding to each stream section (FIG. 4 (a) to FIG. 4 (c)) thus obtained is shown in FIGS. 5 (a) to 5 (c). The telop recognition unit 103 extracts the keyword from the telop text if there is one.

  Next, the keyword matching degree determination unit 104 compares the voice text and the telop text, and if both the voice text and the telop text include common words that are identical to each other (the same word), the keyword recognition trust The degree of speech recognition is determined to be “high”, and the speech recognition reliability is determined to be “low” when both the speech text and the telop text do not include a common word (which is a homophone). The keyword coincidence determination unit 104 outputs the speech recognition reliability and the common word for each stream section (step S4).

  For example, from the audio stream of the stream section with section ID “1”, there are “Nozaki / Senshu” and “Ni-anda” as shown in FIG. The telop text obtained from the stream includes “Nozaki player” and “2 hits” as shown in FIG. Therefore, it is considered that the accuracy of speech recognition in the stream section is high. That is, the speech recognition reliability in this section is “high”, and the common words are “Nozaki / Senshu” and “Nidan”.

  As described above, the speech text and keyword output from the speech recognition unit 102 and the category of the keyword, the telop text and keyword output from the telop recognition unit 103, the category of the keyword, and output from the keyword matching degree determination unit 104. The data generation unit 106 generates the video meta information data as shown in FIG. 6 based on the voice recognition reliability and the common language, the section ID output from the scene information extraction unit 105, the start / end times, and the like. (Step S5).

  In the video meta information corresponding to the stream section with the section ID “1” shown in FIG. 6, the common words “Nozaki / Senshu” and “Ni” are the components of the tag name “common tag value”.・ "Anda" is described.

  In the voice text extracted from the stream section with section ID “1”, there is “Nozaki / Senshu”, which is “AAA Senshu (player)” or “AAA Senshu (player). ) "Or" AAA Sensei (player) 's "is followed," AAA "is determined to be a player name". Further, assuming that “nozaki” is registered in the category “player name” in the word dictionary, the speech recognition unit 102 extracts “nozaki” as a keyword in the speech text. The category of this keyword is “player name”. The keyword “NOZAKI” obtained by the voice recognition unit 102 in this way is described as the value of the component of the tag name “player name” in the video meta information shown in FIG.

  On the other hand, the speech recognition reliability when the speech text and the telop text extracted from the stream section with the section ID “2” and the stream section with the section ID “3” do not include a common word is “low”. Is determined.

  In the voice text extracted from the stream section with the section ID “3”, there is “Sato Senshu”, which is “AAA Senshu (player)” or “AAA Senshu ( Athlete) satisfies the rule that “AAA” is determined to be a player name when it is followed by “AAA's name”. Furthermore, assuming that “Sato” is registered in the category “player name” in the word dictionary, the speech recognition unit 102 extracts “Sato” as a keyword in the speech text. The category of this keyword is “player name”. The keyword “Sato” obtained by the voice recognition unit 102 in this way is described as the value of the component element of the tag name “player name” in the video meta information shown in FIG.

  In FIG. 6, the keyword “nozaki” belonging to the category “player name” is associated as a tag name and its element value (that is, <player name> nozaki </ player name>). Is not limited to this. It may be associated as an attribute name (that is, <category player name = "nozaki" />). Alternatively, the categories may be expressed hierarchically (that is, <player name = “nozaki” />). In this case, the category “player” and its sub-category “name” are combined to represent the category “player name”.

  The rule for extracting the category from the keyword and the category output unit may be provided in the speech recognition unit 102 or the telop recognition unit 103.

  Video meta information as shown in FIG. 6 generated by the video meta information generation unit 1 is stored in the video meta information storage unit 3.

  Next, for example, processing operations for searching for desired content data using the video meta information as shown in FIG. 6 and reproducing the content data will be described with reference to the flowchart shown in FIG.

  Search conditions for searching for desired content data are input from the playback device 7. The search request designating unit 8 of the playback device 7 displays a screen as shown in FIG. In this screen, “item name” is an area for designating a tag name (included in the video meta information), and “item value” is an area for designating a desired character string. From the screen shown in FIG. 8, a tag name (in the video meta information) and a character string included as a value of a component of the tag name can be designated as a search condition, and as a value of any component Only the character string included can be specified as a search condition. Note that the screen shown in FIG. 8 is provided with a region R1 for reproducing and displaying the content data of the search result.

  First, a case where the user designates a tag name and a character string as a search condition will be described as an example. For example, when inputting in the “item name” field, if the user selects the search button B1 on the screen shown in FIG. 8, a list of tag names that can be selected as search conditions included in the video meta information is displayed in a pull-down menu. Etc. are displayed. If the user selects a desired tag name from the list, the user can input the desired tag name in the “item name” field. This list includes “not specified”, and selecting “not specified” from this list means that the tag name is not specified as a search condition.

  As shown in FIG. 8, it is assumed that the user inputs “player name” in the “item name” field and “nozaki” in the “item value” field (step S11). Thereafter, when the user selects the button B2, the search request including the search condition of the tag name “player name” and the character string “NOZAKI” is transmitted to the search device 11, and the content data search unit 4 responds to the search request. Receive included search criteria.

  Since the tag name is included in the search condition (step S12), the content data search unit 4 generates a query (for database search) for a stream section having a high reliability (voice recognition reliability) (step S13). ). Although the query shows an example described in XQuery, it may be described using another query language such as SQL.

  A query generated at this time is shown in FIG. This query is “video meta information whose element value of the“ player name ”tag is“ nozaki ”from video meta information whose reliability is“ high ”in each section data (video meta information of each section). "I want all of you." In the query shown in FIG. 9, the video meta information to be searched is limited to video meta information whose reliability is “high”.

  The query shown in FIG. 9 is output to the video meta information search unit 5. The video meta information search unit 5 uses the tag name specified as the search condition and the character string specified as the search condition among the stream sections having high voice recognition reliability stored in the video meta information storage unit 3. Video meta information (also referred to as section information) of a stream section having components included as values is searched by tag search. That is, in the case of the query shown in FIG. 9, the section with the section ID “1” having the reliability “high” and the component value “player name” including the character string “nozaki” Information is obtained. The video meta information search unit 5 takes out the section ID or start / end time of the section information and passes it to the content data search unit 4 (step S14).

  The content data search unit 4 searches the content data storage unit 2 for content data corresponding to the obtained section ID or start / end time (step S15). Here, the content data of the stream section as shown in FIG. 4A whose section ID is “1” is retrieved, and the retrieved content data is transmitted to the playback device 7 via the network 10. .

  When the playback unit 9 of the playback device 7 receives the content data of the stream section shown in FIG. 4A whose section ID is “1”, the content data is displayed as a search result as shown in FIG. Playback and display are performed in the area R1 (step S16).

  Next, a case where the user specifies only a character string as a search condition without specifying a tag name will be described as an example. As shown in FIG. 11, it is assumed that the user inputs “not specified” in the “item name” field and “nozaki” in the “item value” field (step S11). Thereafter, when the user selects the search button B2, a search request including the search condition “NOZAKI” is transmitted to the search device 11, and the content data search unit 4 receives the search condition included in the search request. To do.

  Since the tag name is not included in the search condition (step S12), the content data search unit 4 does not distinguish between high and low reliability (speech recognition reliability) (for the audio text in all stream sections ( Generate a query (for database search). At this time, first, in step S17, the content data search unit 4 refers to the word dictionary 6 and reads characters that have the same reading (sound) or similar reading (sound) as the character string specified as the search condition. A column (similar character string) is obtained (step S17).

  In the word dictionary 6, a plurality of words and a word (reading) having a reading (sound) that is the same (same sound) or similar to the word are registered for each of the plurality of words. For example, when the designated character string is “NOZAKI”, the word dictionary 6 has “NOZAKI” with the same reading as “NOZAKI” and “OZAKI” with a reading (sound) similar to “NOZAKI”. Is registered.

  The content data search unit 4 generates a query as shown in FIG. 12 using the obtained similar character strings “Nozaki” and “Ozaki” (step S18). The query shown in FIG. 12A is a character string “Nozaki” in the “speech text” element from the video meta information of the stream section with low reliability and the stream section with high reliability (all stream sections). This is a query meaning “find all video meta information including“ as a value ”. The query shown in FIG. 12 (b) is obtained by adding a character string “Ozaki” to the “speech text” element from the video meta information of the stream segment with low reliability and the stream segment with high reliability (all stream segments). This is a query meaning “find all video meta information included as values”.

  The query shown in FIG. 12 is output to the video meta information search unit 5. The video meta information search unit 5 searches the video meta information (section information) including a similar character string in the audio text from all the section information stored in the video meta information storage unit 3. That is, by combining the search results by the query shown in FIG. 12, two pieces of section information with section IDs “1” and “2” are obtained. The video meta information search unit 5 takes out the section ID or start / end time of the section information and passes it to the content data search unit 4 (step S19).

  The content data search unit 4 searches the content data storage unit 2 for content data corresponding to the obtained section ID or start / end time (step S15). Here, the content data of the stream section as shown in FIG. 4A and FIG. 4B with the section IDs “1” and “2” is searched, and the searched content data is the network 10. Is transmitted to the playback device 7 via. As shown in FIG. 13, the playback unit 9 of the playback device 7 plays back and displays the two sections in the search result display area R1 (step S16). In this example, the two searched sections are reproduced and displayed at the same time, but may be reproduced one by one, or thumbnails representative of the two sections may be displayed first, and from these, You may make it make a user select the area to reproduce | regenerate.

  As described above, when the user specifies a tag name as a search condition in step S11 of FIG. 7, in steps S13 to S14, a tag search is performed for a highly reliable stream section, and the user selects a tag name as the search condition. When only a character string (keyword as a search key) is specified without specifying “”, keyword search is performed on the speech text of all stream sections.

  The advantage of using tag search is that it enables highly accurate search by reducing the ambiguity of the meaning of the keyword used for the search key. For example, if the keyword specified as the search key is "Kawasaki" and you search for text that includes the keyword "Kawasaki", it is the name of the place, such as "Kawasaki" city, or Mr. "Kawasaki" Thus, it becomes unclear whether the name is a person's name, and there arises a problem that even a noise which is not supposed to be a search result is searched. In the tag search, for example, the value of the “location” tag is clearly indicated as “Kawasaki” in the data, such as <location> Kawasaki </ location>, and “location =“ Kawasaki ”” is specified. Thus, the search noise as described above can be eliminated.

  However, in order to be tagged like <location> Kawasaki </ location> in the data in this way, the accuracy of text processing in the original data must be high. When text processing is performed using speech text obtained as a speech recognition result as in the present embodiment, even when the tag can be originally extracted when the recognition accuracy is reduced due to mixing of noise or the like, There may be cases where tag generation fails. For example, in the section with the section ID “2” as shown in FIG. 4B, as shown in FIG. 5B, the “nozaki player” is “nozaki / sansha” by voice recognition. Misunderstood.

    In “Nozaki / Sansha”, when the above rule “AAA teacher (player)” or “AAA teacher (player)” or “AAA teacher (player)” continues, “AAA” It does not match “determine that player is a player name” and “player name =“ nozaki ”” is not extracted. Therefore, even if the search is performed under the condition “player name =“ Nozaki ””, the stream section corresponding to the section having the section ID “2” does not hit the search. Therefore, it is considered better to use a keyword search instead of a tag search in a section where it can be estimated that the voice recognition accuracy (voice recognition reliability) is low.

  In this way, by changing the search method (tag search and keyword search) according to the level of voice recognition accuracy, it is possible to selectively use high-precision refinement search and search with few omissions. Yes.

  In step S17 in FIG. 7, the content search unit 4 obtains a character string (similar character string) having a reading that is the same as or similar to the character string specified as the search key. This is because the ambiguity of the speech recognition result is taken into consideration.

  The speech text includes many character strings (keywords) including ambiguity in speech recognition. Since these are likely to be recognized as character strings different from the original voice data, there is a high risk that search omission will occur when such a character string is searched for. Considering this, the speech recognition unit 102 of this embodiment does not convert kana characters into kanji.

  In steps S17 to S19 in FIG. 7, a common query is generated as a query for sections having different speech recognition reliability (a high section and a low section), but the present invention is not limited to this. That is, a query for a section with high reliability and a query for a section with low reliability may be generated separately. In this case, a query shown in FIG. 12C and a query shown in FIG. 12D are generated instead of the query shown in FIG. Further, a query shown in FIG. 12E is generated instead of the query shown in FIG.

  As a result, it is possible to search for a section including a specified character string for a section with high reliability and a section including a character string similar to the specified character string for a section with low reliability.

  As described above, according to the above-described embodiment, the video meta information generation unit 1 extracts audio text and telop text from each stream section of content data composed of audio and video, and from these, the audio recognition result is obtained. The reliability (speech recognition reliability) is obtained.

  For example, when there is a word (common word) whose reading matches in both the voice text and the telop text, it is determined that the voice recognition reliability of the voice text is high, and when the common word cannot be obtained, It is determined that the voice recognition reliability is low. Furthermore, keywords and their categories are obtained from speech texts using rules and word dictionaries stored in advance.

  Using the data obtained in this way, video meta-information used for searching the stream section (in which the characteristics of the stream section are described) is generated for each stream section. The video meta information includes voice text, a keyword obtained from the voice text, a category of the keyword, telop text, voice recognition reliability and common language, ID, start / end time, and the like. The keywords extracted from the speech text are described as element values having the keyword category as a tag name.

  On the other hand, when searching for a desired scene from the content data, the user specifies a tag name and a keyword as a search key, or specifies only a keyword. In the former case, a tag search is performed for a stream section with a high reliability of the speech recognition result, and in the latter case, a text search is performed on the speech text of all stream sections regardless of whether the reliability of the speech recognition result is low. Do.

  Search according to the reliability of speech recognition results, such as high-accuracy tag search for stream sections with high speech recognition result reliability and keyword search and fuzzy search for stream sections with low speech recognition result reliability. By switching methods, search omissions and erroneous searches can be avoided as much as possible.

  The search method switching method is not limited to this. For example, a button for specifying the search accuracy is provided, and in the high-accuracy mode, the tag search with high reliability is specified for the stream section with high reliability, and in the normal mode, the keyword search is specified for all sections, and the ambiguous mode is specified. In this case, the search screen may be configured so as to perform an ambiguous search for all sections.

  The method of the present invention described in the embodiment of the present invention (see FIGS. 3 and 7) is a program that can be executed by a computer, such as a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, DVD). Etc.) can also be stored and distributed in a recording medium such as a semiconductor memory.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  The present invention relates to a home server, for example.

The figure which showed the structural example of the whole system which concerns on embodiment of this invention. The figure which showed the structural example of the image | video meta information production | generation part. The flowchart for demonstrating the processing operation | movement of an image | video meta information generation part. The figure which showed the example of the image which displayed as a text the audio | voice and telop in the stream area of the video recognition result corresponding to three stream areas included in content data. The figure which showed the audio | voice text and telop text obtained from each of the three stream area | regions of FIG. The figure which showed an example of meta information. The flowchart for demonstrating the processing operation at the time of searching the content data in which the desired scene is contained. The figure which showed an example of the search screen which displays a search condition and a search result, and an example of a search condition. The figure which showed an example of the query for tag searches produced | generated in the content data search part 4. FIG. The figure which showed the example of a display of a search result. The figure which showed the other example of the search conditions input into the search screen. The figure which showed the other example of the query for audio | voice text search produced | generated in the content data search part. The figure which showed the other example of a display of a search result.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Video meta information production | generation part, 2 ... Content data storage part, 3 ... Video meta information storage part, 4 ... Content data search part, 5 ... Video meta information search part, 6 ... Word dictionary, 7 ... Playback apparatus, 8 ... Search request specifying unit, 9... Playback unit, 10... Network, 11.

Claims (6)

For each of the plurality of sections of content data divided into a plurality of sections, voice recognition means for recognizing voice data in the content data in the section;
For each of the plurality of sections of the content data, character recognition means for recognizing a telop included in video data in the content data in the section;
Determination means for determining a reliability of the voice recognition result using the voice recognition result in the voice recognition means and the character recognition result in the character recognition means;
Generate meta information consisting of a plurality of element data describing the characteristics of the content data based on the speech recognition result in the speech recognition means, the character recognition result in the character recognition means, and the determination result in the determination means Generating means to
A meta information generating method in a meta information generating apparatus comprising:
The voice recognition means, for each of the plurality of sections of the content data, to obtain a voice text that is a voice recognition result of the voice data of the section;
A second step in which the speech recognition means obtains a keyword included in the speech text and a category to which the keyword belongs from the speech text;
A third step in which the character recognition means obtains, for each of the plurality of sections of the content data, a telop text that is a character recognition result of the telop included in the video data of the section;
The determination unit determines that each of the plurality of sections has high reliability for the speech text in the section when the same text is included in both the speech text and the telop text obtained from the section, A fourth step of determining that the reliability of the speech text in the section is low when no homophone is included;
The generating means describes, for each of the plurality of sections, at least first element data describing an identifier of the section, second element data describing the speech text, and reliability of the speech text a third element data, the fifth to generate the meta-information and a fourth element data describing the said category belongs the keyword and the keyword when it contains the keywords in the audio text Steps,
A meta information generation method characterized by comprising: A meta information generating apparatus that generates meta information including a plurality of element data describing characteristics of content data including video data and audio data,
For each of the plurality of sections of the content data divided into a plurality of sections, a telop text that is a speech recognition result of the speech data of the section and a telop character recognition result of the telop included in the video data of the section A means of seeking
Means for obtaining a keyword contained in the speech text and a category to which the keyword belongs from the speech text;
For each of the plurality of sections, when both the speech text and the telop text obtained from the section include a homonym, it is determined that the reliability of the speech text in the section is high, and the homonym is included. Means for determining that the reliability of the speech text in the section is low when not,
For each of the plurality of sections, at least first element data describing an identifier of the section, second element data describing the speech text, and a third element describing the reliability of the speech text Means for generating the meta-information including data and fourth element data describing the keyword and the category to which the keyword belongs when the keyword is included in the speech text;
A meta information generating apparatus comprising:   3. The meta information generating apparatus according to claim 2, wherein the meta information corresponding to each of the plurality of sections includes fifth element data describing telop text obtained from the section. First storage means for storing content data including video data and audio data ;
Corresponding to each of the plurality of sections of the content data divided into a plurality of sections, each consisting of a plurality of element data describing the characteristics of the content data in one section of the plurality of sections A plurality of pieces of meta information data, wherein the plurality of element data includes a first element data describing an identifier of the section and a second text describing a speech text that is a speech recognition result of the speech data of the section. Fourth data describing element data, third element data describing either high or low reliability of the speech text, a keyword extracted from the speech text, and a category to which the keyword belongs Second storage means for storing the plurality of meta information data including element data ;
Search means;
A search method in a search device comprising:
When the first character string and the second character string are designated as search conditions, the search means has a high degree of reliability among the plurality of meta information data stored in the second storage means. Using the data as a search target, search the meta information including the fourth element data in which the category of the same sound as the first character string and the keyword of the same sound as the second character string are described. A first step to:
When only the second character string is designated as the search condition among the first character string and the second character string, the search means respectively have the same sound as or similar to the second character string. A second step for obtaining a plurality of third character strings having sounds;
The search means uses the meta information data stored in the second storage means as a search target, and from the search target, meta information including any of the plurality of third character strings in the speech text A third step of searching for
Fourth said search means, for searching the content data from said stored content data in the first storage means, the section corresponding to the identifier included in the first and third of the retrieved meta information in step And the steps
A search method characterized by comprising: First storage means for storing content data including video data and audio data;
A meta data corresponding to each of the plurality of sections of the content data divided into a plurality of sections, each of which is composed of a plurality of element data describing characteristics of the content data in one section of the plurality of sections. Information data, in the plurality of element data, a first element data describing an identifier of the section, a second element data describing a speech text as a speech recognition result of the speech data of the section; , Third element data describing either high or low reliability of the speech text, fourth element data describing a keyword extracted from the speech text and a category to which the keyword belongs, Second storage means for storing the plurality of meta information data including:
When the first character string and the second character string are designated as search conditions, meta information data having a high reliability among a plurality of meta information data stored in the second storage means is set as a search target. The first search for meta information data including the fourth element data describing a category of the same sound as the first character string and a keyword of the same sound as the second character string from the search target. Search means;
When only the second character string is specified as the search condition among the first character string and the second character string, a plurality of sounds each having the same sound as or similar to the second character string Means for obtaining a third character string;
The meta information data stored in the second storage means is set as a search target, and meta information data including any of the plurality of third character strings in the speech text is searched from the search target. Two search means;
Third search means for searching content data in a section corresponding to the identifier included in the meta information data searched by the first and second search means from the content data stored in the first storage means When,
A search device comprising: Means for obtaining, for each of the plurality of sections of the content data, speech text that is a speech recognition result of the speech data of the section and telop text that is a character recognition result of the telop included in the video data of the section;
Means for obtaining a keyword contained in the speech text and a category to which the keyword belongs from the speech text;
For each of the plurality of sections, when both the speech text and the telop text obtained from the section include a homonym, it is determined that the reliability of the speech text in the section is high, and the homonym is included. Means for determining that the reliability of the speech text in the section is low when not,
For each of the plurality of sections, at least first element data describing an identifier of the section, second element data describing the speech text, and a third element describing the reliability of the speech text Generating means for generating the meta information data including data and fourth element data describing the keyword and the category to which the keyword belongs when the keyword is included in the speech text;
Further comprising
6. The search apparatus according to claim 5, wherein the second storage unit stores the meta information data generated by the generation unit.

Images