JP5392780B2 - Content structured processing method, content structured processing apparatus, and content structured processing program - Google Patents

Description

  The present invention relates to a content structuring processing method, a content structuring processing apparatus, and a content structuring processing program for semantically structuring content using texts and sounds contained in media content as clues.

  Today, advances in digital technology and mass storage technology have made it possible to store a large amount of information such as text, sound, and images even in an individual. In order to effectively use such information as a resource, it is necessary to perform indexing to quickly reach desired information and structure to quickly grasp an outline of information.

  However, due to the large amount of information, it is difficult to perform such processing manually, and there is an urgent need for a technique for automatically recycling information using a computer.

  In particular, the semantic structuring technology that estimates the topic of each part of the content and divides the content into topics by focusing on the occurrence distribution of syntactic elements such as words contained in text and speech is a search. Many researches and developments have been made as basic techniques for performing higher-level information resource processing such as classification and summarization.

  When the topic appearing in the content can be predicted to some extent, a topic model can be constructed from prior knowledge and training data on the topic, and the given content can be semantically structured using the model (Patent Document 1). Non-Patent Document 1).

  However, it is often difficult to predict the topics that appear, such as content such as home videos and daily small meeting records in the company. In such casual content, it is not practical to prepare a model related to a topic in advance at a high cost. Therefore, in such a case, it is necessary to perform semantic structuring using only given contents.

  As a typical method when knowledge about a topic cannot be assumed, a semantic structure is created by dividing the content into small sections for analysis, defining similarities in these analysis sections, and putting together similar sections There is a method of making it.

  In many semantic structuring techniques, an analysis interval is expressed as a vector having the appearance frequency of a syntax element such as an appearing word as a feature. At that time, a stemming process for extracting a stem is generally performed in order to absorb word form changes such as utilization. Furthermore, stop word removal is also performed at the same time to remove function words such as particles and auxiliary verbs that are not useful for semantic analysis.

  In the determination of the similarity between the analysis sections, in Patent Document 2 and Non-Patent Document 2, the cosine of the expression vector of the analysis section is used, and when the similarity between the adjacent analysis sections is high, the section belonging to the same topic It is regarded.

  In this case, instead of considering only the local similarity between adjacent analysis intervals, consider the similarity between sets of multiple analysis intervals, and use the problem solving method of dynamic programming to optimize globally. A method for calculating a simple division has also been proposed (Non-Patent Document 5, Non-Patent Document 6).

  When performing analysis using speech contained in content, it is common to convert speech into text using a large vocabulary continuous speech recognition technology and apply the above-described text analysis technology.

  However, unlike read-out speech such as news speech, the current speech recognition processing technology does not provide sufficient speech recognition accuracy for free speech, and semantic structuring from text with low recognition accuracy also provides sufficient performance. I can't demonstrate it. In order to increase the accuracy of speech recognition, it is necessary to adapt vocabulary and language models to content, but for that purpose, there is a structural problem that knowledge about topics appearing in the content is required.

  Conventionally, in order to solve such problems, a speech is recognized as a code string of a subword unit such as a syllable, a phoneme, and a phoneme piece (Patent Document 3) without using large vocabulary continuous speech recognition. There is proposed a topic division processing method that analyzes the appearance distribution of a partial code string of instead of the distribution of words and performs semantic structuring of speech (Patent Document 4, Patent Document 5, Non-Patent Document 4). .

  Further, as the prior art documents related to the topic division of this type of multimedia content, the following documents can be referred to.

Japanese National Patent Publication No. 8-512148 JP 2004-157337 A Japanese Patent Laid-Open No. 2002-221984 JP 2006-251553 A JP 2007-316323 A

K. Ohtsuki et al. : Topic Extraction Based on Continuous Speech Recognition in Broadcast News Speech, IEICE TRANS. INF. & SYST. , Vol. E85D, no. 7, 2002. Seiichi Takao et al .: Topic segmentation and classification for news speech, IPSJ, 24, pp. 55-62, 1998. J. et al. Shawe-Taylor and N.M. Cristianini: Kernel methods for pattern analysis, Cambridge University Press, 2004. Takeshi Sadohara et al .: Topic segmentation using kernel principal component analysis of phonemes, Technical report of IEICE, 104, 726, pp. 37-41, 2005. Masao Uchiyama et al .: Field-independent text segmentation by statistical methods, natural language processing, 8: 4, pp. 19-36, 2001. P. Fragkou et al. A Dynamic Programming Algorithm For Linear Text Segmentation, Journal of Intelligent Information Systems, 23: 2, pp. 179-97, 2002.

  As described above, preprocessing such as a stemming process and a stop word removal process is necessary for semantic structuring of text. Normally, these processes are performed on any content using the same processing algorithm based on language knowledge given in advance. However, such a static processing algorithm does not always function effectively. For example, the words “speech” and “recognition” are content words, so they are not usually considered as stop words, but they are words that appear everywhere in the conference records that discuss speech recognition. Is not useful for semantic structuring.

  In this way, is preprocessing such as stemming processing and stopword removal processing useful for semantic structuring depending on given content, not static algorithms based on given language knowledge? It is necessary to change the algorithm dynamically in terms of whether or not.

  Also, when converting speech contained in content into text using large vocabulary continuous speech recognition technology, it is difficult to obtain sufficient recognition accuracy without language knowledge about vocabulary and words appearing in the content to be analyzed. In situations where even the topics included in the content are not known, it is difficult to obtain such prior knowledge, and the conventional method of converting to text once and performing semantic structuring using text analysis technology is semantic with sufficient accuracy. It cannot be structured.

  In the inventions described in Patent Document 4 and Patent Document 5 made by the present inventor with the aim of solving such problems, speech is converted to syllables, phonemes, phonemes, etc. without using large vocabulary continuous speech recognition. Disclosed is a method for semantic structuring of speech by recognizing a subword unit with a finer granularity than a word as a code string and analyzing the occurrence distribution of an arbitrary partial code string instead of the word occurrence distribution. doing.

  However, these conventional techniques cannot analyze discontinuous partial code strings including gaps, and drop / insert errors frequently occur when subword units are recognized. When frequency analysis is not performed, there is a problem that performance deteriorates.

  In addition, when recognizing a subword unit, a substitution error that mistakenly recognizes a subword unit as another subword unit with similar utterances frequently occurs. Instead, it is desirable to perform a soft matching process that evaluates the degree of similarity with a continuous amount according to the similarity of the corresponding subword units, but the conventional technique cannot perform the soft matching process due to the problem of the amount of calculation.

  In addition, these conventional technologies employ algorithms based on the greedy method when creating a semantic structure by gathering the analysis intervals, so it is caught by the local optimal solution and a global optimal solution can be obtained. There is a problem that it may not be possible.

  The present invention has been made to solve the above-described problems, and an object of the present invention is a content structure for semantically structuring content by using text and audio included in the media content as a clue. An object of the present invention is to provide an information processing method, a content structured processing device, and a content structured processing program.

  In order to achieve the above object, in the present invention, as basic data processing, the text and audio of the content are converted into a code string, and a pre-processing for a partial code string corresponding to stemming and stop word removal is given. Depending on the content, the analysis is performed dynamically in terms of whether or not it contributes to semantic structuring. When analyzing the appearance distribution of partial code sequences, discontinuous partial code sequences including gaps are also analyzed. The soft matching process targeted at is used. Furthermore, a content structuring processing method and a content structuring processing apparatus capable of obtaining a global optimal solution using dynamic programming, principal component analysis, etc. in the case of semantic structuring based on analysis And a content structuring program.

  Specifically, as a first aspect, the content structuring process according to the present invention inputs text or audio included in media content, executes data processing on the input text or audio by a computer, and makes the content semantic. A content structuring method for structuring a content, a content encoding process for converting text or speech into a code string such as words, syllables, phonemes and characters, and a minimum unit of analysis for the code string, Atomic segment extraction process to extract the atomic segment, similarity matrix calculation process to calculate the similarity between the atomic segments using the character string kernel and store it in memory, and to make the similarity between the atomic segments stand out Independence relaxation process and redundant component to remove any subcode sequences that are not useful in semantic structuring The steps to, is characterized in performing the process of the optimum structure calculation process of calculating the global optimal structure by a computer using a dynamic programming method or principal component analysis.

  As a second aspect of the present invention, the content structuring apparatus according to the present invention inputs text or audio included in media content, executes data processing on the input text or audio by a computer, A content structuring apparatus for semantically structuring content encoding means for converting text or speech into a code string such as words, syllables, phonemes and characters, and minimum analysis of the code string Atomic segment extraction means for extracting atomic segments as units, similarity matrix calculation means for calculating similarity between atomic segments and storing them in memory, and independence to make the similarity between atomic segments stand out Mitigation means, redundant component removal means for removing any partial code string that is not useful in semantic structuring, and It is characterized in further comprising an optimum structure calculating means for calculating a globally optimal structure using planning method and principal component analysis.

  As a third aspect of the present invention, the content structuring processing program according to the present invention inputs text or audio included in media content, executes data processing on the input text or audio by a computer, A content structuring processing program for semantically structuring, comprising: a content encoding step for converting text or speech into a code string such as a word, syllable, phoneme or character; and minimum analysis of the code string Atomic segment extraction step for extracting atomic segments as units, similarity matrix calculation step for calculating similarity between atomic segments and storing them in memory, and independence to make the similarity between atomic segments stand out Relaxation step and any subcode sequences that are not useful in semantic structuring A redundant component removing step of to, is characterized in that to execute the processing of the optimum structure calculation step of calculating the global optimal structure in a computer with dynamic programming and principal component analysis.

  According to the content structuring processing method, content structuring processing apparatus, and content structuring processing program of the present invention having the features as described above, text or speech included in the content is converted to a code string such as a word, syllable, phoneme, character, etc. It is possible to analyze the appearance distribution pattern of an arbitrary partial code string and convert the content into a semantic structure. For this reason, prior knowledge about topics included in the content, linguistic knowledge such as vocabulary, language model, stop word list, and the like are not required, and similar techniques already invented by the inventors (Patent Document 4, Patent Document) It is possible to realize semantic structuring processing with higher performance than 5).

It is a flowchart which shows an example of the content structuring process based on this invention. It is a figure explaining the structure of the content structure processing apparatus by this invention. It is a flowchart of the program in the case of performing the content structuring process by this invention by the process of a computer. It is a flowchart which shows the processing flow of the optimal structure calculation process in the content structuring process by this invention. It is a flowchart which shows the example of the smoothing process using the restriction | limiting of the granularity of the structure of an example of the process of the smoothing step of an optimal structure calculation process. It is a flowchart which shows the example of the smoothing process using the principal component analysis of another example of the process of the smoothing step of an optimal structure calculation process. It is a block diagram explaining the structural example which implements the content semantic structure processing apparatus by this invention as a semantic structure processing apparatus which presents and produces the summary of the recording of a daily small meeting. It is a figure which illustrates an example of the display screen shown on the display connected to the content structure presentation part (B8). It is a figure for demonstrating the implementation example of the semantic structure processing apparatus for a meeting content summary presentation.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a flowchart showing an example of content structuring processing according to the present invention. With reference to FIG. 1, the principle of content structuring processing according to the present invention will be described.

  In the content structuring processing according to the present invention, as basic processing, a content encoding process (P1) for converting text or speech included in media content into a code string such as a word, a syllable, a phoneme or a character, An atomic segment extraction process (P2) for extracting the atomic segment which is the minimum unit of analysis for the code string, and the similarity between the extracted atomic segments is calculated using a character string kernel and stored in the memory as a similarity matrix The similarity matrix calculation process (P3) to store, the independence relaxation process (P4) to reduce the diagonal components of the similarity matrix to make the similarity between atomic segments stand out, and semantically structured A redundant component elimination process (P5) that removes subcode sequences that are not useful due to predetermined criteria, and a globally optimal structure is calculated using dynamic programming or principal component analysis It performs each process of optimal structure calculation process (P6) that.

  Further details will be described. When executing the content structuring process, first, in the content encoding process (P1), when the input is text, encoding is performed in units of words and characters constituting the word, and the input is performed by voice. In some cases, encoding is performed in units of words, syllables, phonemes, phonemes using continuous speech recognition technology.

  Next, with this code sequence as an input, in the atomic segment extraction process (P2), if the text is input, for example, a partial sequence is extracted in sentence units, and if the input is speech, the silent section A partial sequence is extracted in units of utterances separated by, and this is used as an atomic segment.

今、原子セグメントｓの表現ベクトルを以下のようにｚとおく。

Subsequently, in the similarity matrix calculation process (P3), the similarity is calculated with respect to any two atomic segments by a method according to the Gap-weighted subsequence kernel described in Non-Patent Document 3. The similarity used here is defined as an inner product of the vectors after expressing each atomic segment as a vector having a feature corresponding to a partial character string having a length of P or less as an element. Therefore, the dimension of this vector is very large as follows.

Now, let the expression vector of the atomic segment s be z as follows:

ここで、｜ｓ｜はｓの文字数を表し、Ｉ_ｐはギャップを含むｓの任意の部分文字列のインデックスの集合であり、上記素性は、ｕの各出現を含まれるギャップ数の指数乗で重みづけしたものと考えられる。このように、単語ではなく、単語を構成する文字や音素等のサブワードユニットの出現頻度を分析する場合には、ステミング処理と同等の前処理が意味的構造化処理の内部で動的に行われることに注意されたい。 At this time, the component of z corresponding to the partial character string u of length p is defined as follows for each occurrence of u in s.

Here, | s | represents the number of characters of s, I _p is a set of indices of arbitrary substrings of s including gaps, and the above feature is the exponential power of the number of gaps including each occurrence of u. It is thought to be weighted. Thus, when analyzing the appearance frequency of subword units such as characters and phonemes constituting a word instead of words, preprocessing equivalent to stemming processing is dynamically performed within the semantic structuring processing. Please note that.

もちろん、このような内積は、表現ベクトルが非常に高次元であることを考えれば、素朴な計算法では現実的な時間に計算することはできない。しかし幸いなことに、非特許文献３に記載されたＧａｐ−ｗｅｉｇｈｔｅｄ ｓｕｂｓｅｑｕｅｎｃｅ ｋｅｒｎｅｌに準じた方法を用いることで効率良く計算することができる。具体的には、以下のような動的計画法を用いて原子セグメントｓ，ｔの表現ベクトルの内積を計算する。

ただし、ｓ（ｉ：ｊ）は符号列ｓのｉ番目からｊ番目までの部分文字列を表している。上記の計算は計算結果のテーブルを活用することでさらに効率化できて、Ｏ（ｐ｜ｓ｜｜ｔ｜）の計算量で計算可能であることが知られている。 Using the “expression vector” expressed as this vector, the similarity between arbitrary atomic segments s ₁ and s ₂ is defined as the inner product of the expression vectors as follows.

Of course, such an inner product cannot be calculated in a realistic time with a simple calculation method, considering that the expression vector has a very high dimension. Fortunately, however, the calculation can be performed efficiently by using a method according to Gap-weighted subsequence kernel described in Non-Patent Document 3. Specifically, the inner product of the expression vectors of the atomic segments s and t is calculated using the following dynamic programming method.

However, s (i: j) represents the i th to j th partial character strings of the code string s. It is known that the above calculation can be made more efficient by using a calculation result table and can be calculated with a calculation amount of O (p | s || t |).

ただし、上記計算が表現ベクトル間の内積になっていることを保証するためには、符号間の類似性行列Ａは、対称かつ半正定値でなければならない。この類似性行列は、例えば、符号の混同行列を用いて、次のように構成することが可能である。

ここで、Ｐ（Ｏ＝ｃ），Ｐ（Ｒ＝ａ｜Ｏ＝ｃ）は混同行列から計算される確率で、それぞれ、符号ｃの出現確率、符号ｃが符号ａとして誤認識される確率を表している。 It is also known that soft matching can be realized by replacing [a = b] appearing in the above mathematical expression with matrix elements A _{a, b} representing the similarity between codes as follows.

However, in order to guarantee that the above calculation is an inner product between expression vectors, the similarity matrix A between codes must be symmetric and semi-definite. This similarity matrix can be configured as follows using a code confusion matrix, for example.

Here, P (O = c) and P (R = a | O = c) are probabilities calculated from the confusion matrix, and the appearance probability of the code c and the probability that the code c is erroneously recognized as the code a, respectively. Represents.

  As described above, after the similarity matrix K between the atomic segments is calculated in the similarity matrix calculation process (P3), the independence relaxation process (P4) is similar to weaken the independence between the atomic segments. The diagonal component of the sex matrix K is appropriately reduced in the following manner.

If a partial code string appears only in a particular atomic segment s _i , such a partial code string contributes to increasing the independence of the atomic segment s _i relative to other atomic segments, and the similarity matrix K It has the effect of increasing only the _Kii component. The increase in such components means that the atomic segment is not similar to any other atomic segment, and in extreme cases, a very fine structure in which the content is divided into every atomic segment is output. End up. Therefore, a partial code string that appears only in a specific atomic segment considered to be noise from the viewpoint of semantic structuring is removed by the following procedure.

Now, φ _{(s i)} of all atoms segments except the expression vector _{φ (s 1), ...,} φ (s i-1), φ (s i + 1), ..., φ a _{(s n)} is spanned subspace U _i Distant calculates the projection to _{U i} of φ _{(s i),} replace phi a _{K ii} is the square norm of _{(s i)} to the square norm of the projective.

The projection of φ (s _i ) onto U _i is as follows using the eigenvalue λ _{j of the} matrix L obtained by removing the i-th row and i-th column of the similarity matrix K and the eigenvector v ^j having a norm of 1: Can be calculated.

次に、冗長成分除去過程（Ｐ５）では、ストップワード除去に相当する前処理として、部分文字列の平均的な出現パターンの影響を取り除く。そのために、例えば、重心を原点とする単純なセンタリングを行うことができる。この場合、類似性行列は以下のように変換すればよい。

あるいは、変換の線型性に配慮して、全ての原子セグメントの表現ベクトルの平均をとり、各原子セグメントに対して、この平均ベクトルの直交補空間への射影を新たな表現ベクトルとすることもできる。この場合、類似性行列は以下のように変換される。

By replacing the square norm of the projection thus calculated with K _ii , the following new similarity matrix is calculated in the independence relaxation process (P4).

Next, in the redundant component removal process (P5), the influence of the average appearance pattern of the partial character string is removed as preprocessing corresponding to stop word removal. Therefore, for example, simple centering with the center of gravity as the origin can be performed. In this case, the similarity matrix may be converted as follows.

Alternatively, in consideration of the linearity of the conversion, the expression vectors of all the atom segments can be averaged, and the projection of the average vector onto the orthogonal complement space can be used as a new expression vector for each atom segment. . In this case, the similarity matrix is converted as follows.

  Finally, in the optimal structure calculation process (P6), an optimal semantic structure is calculated using the similarity matrix K obtained in the redundant component removal process (P5). Hereinafter, the optimum structure calculation process (P6) will be described in more detail with reference to FIG.

  FIG. 4 is a flowchart showing the process flow of the optimum structure calculation process in the content structuring process. In the process of calculating the optimum structure, first, after the similarity matrix is loaded onto the memory in the similarity matrix reading step (S101), an arbitrary atomic segment is set to 1 in the content structure initialization step (S102). The initial value of the structure as one segment is stored in the memory.

  Next, in the smoothing parameter reading step (S103), parameters relating to the smoothness of the semantic structure granularity are read into the memory. At this time, confirmation and input by the user can also be performed. As the smoothing parameter, the number of content divisions can be given explicitly, and the upper and lower limits of the number of divisions can also be given. Alternatively, as described in detail in the smoothing step (S105), the average value of the size of each segment, the lower limit of the ratio of the smallest segment size to the largest segment, and the like are given to control the granularity of division. can do. Furthermore, when performing principal component analysis for the purpose of removing segments regarded as noise, the cumulative contribution rate, the ratio of the magnitude of the eigenvalue to the maximum eigenvalue, and the like can also be used as the smoothing parameter.

  Subsequently, the process of the minimum cost path calculation step (S104) is performed. In the process of this minimum cost path calculation step, an optimal segmentation calculation is performed in the sense described below, using an atomic segment series and its similarity matrix as inputs.

ここで、Ｔは全ての原子セグメント集合の分割である。各セグメントの表現ベクトルをそのセグメントに含まれる原子セグメントの表現ベクトルの和と考えると、Ｔ^＊は、可能な分割の内、セグメントの表現ベクトルの２乗ノルムの和を最大とする分割になる。 First, the case where there is no restriction on the number of divisions will be described. In this case, an optimal division T ^* defined as the following equation is calculated.

Here, T is the division of all atomic segment sets. Considering the expression vector of each segment as the sum of the expression vectors of the atomic segments included in the segment, T ^* is the division that maximizes the sum of the square norms of the segment expression vectors.

このとき、Ｃ（ｎ）が最小のコストを表し、最小コストパスは以下のようになる。

This optimization problem can be reduced to a so-called minimum cost path problem. The minimum cost path problem is a problem in which when the cost is defined in any two nodes of n + 1 nodes from 0 to n, the path with the lowest cost among the paths from 0 to n is obtained. It is possible to solve with O (n ² ) by the following dynamic programming method.

At this time, C (n) represents the minimum cost, and the minimum cost path is as follows.

もしも、平滑化パラメータとして分割の数Ｌが与えられた場合には、パス長さをＬに制限した最小コストパス問題を解けばよい。この場合最適分割は以下のような動的計画法で計算される。

Further, the optimum division can be obtained from the minimum cost path as follows.

If the number of divisions L is given as the smoothing parameter, the minimum cost path problem with the path length limited to L may be solved. In this case, the optimal partition is calculated by the following dynamic programming method.

In this case, the following is the optimal path.

In the same way, it is possible to obtain the optimum division with the maximum division value limited to L. The minimum cost path corresponding to the optimal partition is calculated by the following dynamic programming.

ただし、上記、長さの最小値を制限した動的計画法において、長さの制限のない最小コストパス問題が内包されていることに注意されたい。 Further, the optimum division with the minimum division value limited to L can be similarly obtained. The minimum cost path corresponding to the optimal partition is calculated by the following dynamic programming.

However, it should be noted that the above-described dynamic programming with a minimum length limit includes a minimum cost path problem with no length limit.

  As described above, as a smoothing parameter, when the number of divisions is not given, when the number of divisions is given, when the upper limit of the number of divisions is given, when the lower limit of the number of divisions is given, the minimum corresponding to each By solving the cost problem, the optimum division considering the restriction on the number of divisions is calculated in the minimum cost path calculation step (S105).

  In the smoothing step (S105), smoothing is performed on the optimal division obtained in this way to remove a segment that is too fine compared to other segments, as will be described later.

Further, with respect to the optimal division obtained in this way, a further higher-level structure can be calculated according to the user's request (S106). In this case, in the similarity matrix degeneration step (S107), each segment in the division is regarded as an atomic segment, and a new similarity matrix is calculated by degenerating a continuous row and a continuous column of the similarity matrix into one element. Is done. The i and j elements of the new similarity matrix represent the inner product of the i-th segment and the j-th segment, and are calculated as follows.

  Using the new similarity matrix, the optimum structure is calculated again in the minimum cost path calculation step (S104). In this case, a number smaller than the number of rows of the similarity matrix can be selected as the upper limit of the number of divisions so that an upper structure different from the current structure is output.

After repeating such steps, the process proceeds to the segment information providing step (S108). In the process of the segment information adding step (S108), information indicating the characteristics of the segment is added to each segment of each layer. For example, information on similarity to other segments is given. This information can be used, for example, for the purpose of displaying similar segments in similar colors. As segment similarity data, a cosine calculated as follows based on the similarity matrix can be used.

  In addition, when a label is already given to a certain segment, label information can be given to other similar segments with similarity.

  Finally, the content structure saving step (S109) is performed. In this processing, the content structure composed of the hierarchical structure of the content obtained so far and the information given to each segment of each hierarchy is stored in the main memory.

  Here, the smoothing step (S105) will be described in more detail using two specific methods. First, the case where a restriction regarding the granularity of the obtained segment is given as a result of the division will be described with reference to FIG.

FIG. 5 is a flowchart illustrating an example of the smoothing process using the structure granularity constraint of an example of the smoothing step process in the optimum structure calculation process. In this smoothing process, in the norm calculation step (S110), the square norm of the expression vector of each segment is calculated as follows for the similarity matrix K and the division T = T ₁ ,..., T _r . .

  Next, in the particle size criterion evaluation step (S111), it is checked whether or not the particle size criterion is met. When the lower limit of the ratio between the minimum value and the maximum value of the square norm is given as the granularity standard, if the value falls below this threshold, the segments are integrated. Alternatively, when an upper limit of the variance of the square norm is given, segment integration is performed when this threshold is exceeded.

  The integration of the segments is performed in a maximum length designated minimum cost path calculation step (S112), and a new division is calculated by using the dynamic programming that solves the above-described minimum cost path problem with the maximum path length limited. At this time, a value smaller than the original number of divisions is designated as the maximum value (maximum path length) of the number of divisions so that a coarser division than the original division is calculated. With the above procedure, the smoothing step (S105) when the granularity criterion is given as the smoothing parameter ends.

  In the process of the smoothing step (S105), the smoothing process can also be executed by performing a principal component analysis and removing a component that seems to be noise. This will be described with reference to FIG. 6 for smoothing processing using principal component analysis.

  FIG. 6 is a flowchart illustrating an example of a smoothing process using principal component analysis, which is another example of the process of the smoothing step in the optimal structure calculation process. In this case, since the representation vector of the atomic segment is very high-dimensional, normal principal component analysis using it as it is cannot be performed in a realistic time. Perform kernel principal component analysis to perform principal component analysis. Since kernel principal component analysis is described in detail in Non-Patent Document 3 and the like, description thereof is omitted here.

Referring to FIG. 6, in this process, first, in the similarity matrix degeneration step (S113), the inner product between the expression vectors of the T segments is _calculated from the similarity matrix K and the divisions T = T ₁ ,. Is calculated as follows.

Next, in the eigenvalue calculation step (S114), eigenvalues of this matrix are calculated, and using the smoothing parameters, the top m positive eigenvalues λ ₁ >...> Λ _m > 0 and the corresponding norm 1 Eigenvectors v ¹ ,..., V ^m are selected. As the smoothing parameter, the cumulative contribution rate, the ratio to the maximum eigenvalue, and the like are used, and the top m eigenvalues whose cumulative contribution rate exceeds the given threshold are selected, or the top m whose ratio to the maximum eigenvalue exceeds the threshold Or select eigenvalues.

Subsequently, the similarity matrix recalculation step (S115) is performed. In this process, using the eigenvalue and eigenvector selected in this way, a new similarity matrix having the inner product of the expression vectors of each segment from which the noise component has been removed as elements is calculated as follows.

  If the newly obtained similarity matrix is significantly different from the matrix obtained in the similarity matrix degeneration step (S113), this is determined (S116), and the process of the minimum cost path calculation step (S117) is performed. Do. By this process, the optimum division is calculated by the same method as the above-described minimum cost path step (S104) of FIG. Such processing steps are repeated until the similarity matrix hardly changes, and then the smoothing step (S105) ends.

  The principle of the content structuring process according to the present invention has been described above. However, the expression vector of the atomic segment is only used for explaining the principle, and in the actual calculation, only the similarity matrix K storing the inner product is used. Note that all calculations are done via Since the expression vector of the atomic segment has a very high dimension as described above, the use of such a high-dimensional vector for calculation explicitly contributes to a significant increase in efficiency of the calculation. In this sense, the present invention is a kind of kernel method described in Non-Patent Document 3, and shows one aspect of the novelty of the present invention in terms of calculation algorithms.

  FIG. 2 is a diagram for explaining the configuration of the content structuring apparatus according to the present invention. As shown in FIG. 2, the content structuring apparatus according to the present invention includes a content encoding means (M1), an atomic segment extraction means (M2), a similarity matrix calculation means (M3), an independence relaxation means (M4), It consists of redundant component removal means (M6) and optimum structure calculation means (M6), which takes multimedia content as input, performs semantic structuring of the content based on text or audio contained in the content, and finally Output semantic structure information to.

  The content encoding means (M1) converts text and speech included in the media content into a sequence of codes such as words, characters, syllables, phonemes, phonemes. The atomic segment extraction means (M2) receives the converted code sequence as input, divides it into a minimum unit of analysis, such as a sentence for text and a speech section for speech, and stores them in the main memory.

  In the similarity matrix calculation means (M3), the similarity between atomic segments is calculated using a character string kernel and stored in a table on the main memory as a similarity matrix. The independence relaxation means (M4) reads the similarity matrix from this table, reduces the diagonal component by the method described in the independence relaxation process (P4), and writes the value back into the table. In the redundant component removal means (M5), a similarity matrix is read from the table, and a new similarity matrix having the inner product of the expression vectors from which redundant components are removed as elements is read by the method described in the redundant component removal process (P5). Calculate and store in table on main memory. The optimal structure calculation means (M6) calculates the globally optimal content structure using dynamic programming or principal component analysis based on the similarity matrix stored on the main memory, and stores it on the main memory. Store.

  FIG. 3 is a flowchart of a program when the content structuring processing according to the present invention is executed by computer processing. As shown in FIG. 3, the computer executing the processing of this program performs a content encoding step (S1), an atomic segment extraction step (S2), a similarity matrix calculation step (S3), an independence relaxation step ( The process of each step of S4), redundant component removal step (S6), and optimal structure calculation step (S6) is executed. The computer in which this program is installed executes the processing of each step, thereby performing content encoding means, atomic segment extraction means, similarity matrix calculation means, independence relaxation means, redundant component removal means, and optimal structure calculation means. A functioning content structuring apparatus is configured.

  In the present invention, as described above, for the purpose of realizing a user interface that can quickly understand the outline of content by semantically structuring multimedia content using text and voice as clues. Yes. In addition, in order to realize a method that does not depend on knowledge about content, text and speech can be converted into code sequences without using linguistic knowledge such as vocabulary, language models and stop words contained in content, and prior knowledge about topics. After conversion, it is possible to analyze the appearance distribution of an arbitrary partial code string in a robust manner against dropout / insertion / replacement errors included in the code sequence, and to structure the content semantically. Next, a semantic structure processing apparatus for the purpose of presenting a summary of contents will be described.

  FIG. 7 is a block diagram illustrating a configuration example in which the content semantic structure processing device according to the present invention is implemented as a semantic structure processing device that presents and creates a summary of recordings of daily small-scale meetings. As shown in FIG. 7, the semantic structure processing apparatus includes a subword encoding unit (B1), an atomic segment extraction unit (B2), a similarity matrix calculation unit (B3), an independence relaxation unit (B4), and redundant component removal. Unit (B5), optimal structure calculation unit (B6), semantic structure information recording unit (B7), semantic structure presentation unit (B8), utterance interval recording unit (B9), utterance interval extraction unit (B10), speech feature extraction unit (B11).

  In the semantic structure processing device shown in FIG. 7, the utterance section extraction unit (B10) cuts out the utterance section from the speech input from the microphones of the conference participants, and the utterance section recording unit (B9) together with the speaker and the utterance time. Save to the external memory you have. At this time, in the case where the conference is being photographed by the video camera, the video image can be recorded in the utterance recording unit (B9) in synchronization with the utterance.

  At the same time, the extracted utterance section is sent to the speech feature extraction unit (B11), converted into acoustic feature quantities such as MFCC and LPC through normal framing processing and spectrum analysis, and then the subword encoding unit (B1). ), A sub-word code sequence is converted using a sub-word unit acoustic model using HMM or the like. Further, the atomic segment extraction unit (B2) stores the code string of the subword code obtained from one utterance in the main memory as one atomic segment.

  In the similarity matrix calculation unit (B3), the atomic segment on the main memory is read, the similarity is calculated for any two atomic segments by using the character string kernel, and the similarity matrix having this element as the main is calculated. Store in a memory table.

  In the independence relaxation unit (B4), the diagonal components of the similarity matrix are appropriately reduced in order to reduce excessive independence of each atomic segment. The redundant component removal unit (B5) calculates a new similarity matrix obtained by removing redundant components in the semantic structuring of the conference audio based on the previous similarity matrix, and calculates a new value as a main value. Write back to the memory table.

  The optimal structure calculation unit (B6) calculates the optimal semantic structure of the content using dynamic programming or principal component analysis while referring to the latest similarity matrix, and the semantic structure information recording unit (B7) Is stored in main memory or external memory.

  The semantic structure presentation unit (B8) refers to information such as the hierarchical structure held by the semantic structure information recording unit (B7), the feature amount of each segment of each layer, and visualizes the conference audio reflecting the semantic structure, Present to the user. At the same time, in response to a request from the user, the partial section desired by the user is reproduced based on the voice data held in the speech section recording unit (B9). Alternatively, when the user gives a label to a certain segment, gives a text to a certain utterance, or modifies the semantic structure, this change is made into semantic structure information through the semantic structure information recording unit (B7). To reflect. Further, when the user instructs recalculation of the semantic structure using different smoothing parameters, the semantic structure is recalculated through the optimum structure calculation unit (B6), and the semantic structure information obtained as a result is presented as the semantic structure. This is presented to the user by the part (B8).

  FIG. 8 is a diagram illustrating an example of a display screen presented on a display connected to the content structure presentation unit (B8). As shown in FIG. 8, this display screen is composed of window display areas of a hierarchy display section (B101) and a content playback section (B102).

  In the window display area of the hierarchy display section (B101), the structures included in the content are displayed hierarchically based on the semantic structure information. The depth of the displayed hierarchy can be selected by a slider, and a still image selected from the corresponding section of the content is displayed in the lowest segment.

  The user can select a window in an arbitrary hierarchy within the window display area displayed in the hierarchy display section (B101) by operating the mouse. When the user selects a certain level, the user interface processing unit that performs window display processing of the content playback unit (B102) reads the start time and end time corresponding to the selected level from the semantic structure information, and the corresponding part of the content To load. When the section playback button is pressed, the corresponding part is played back.

  Further, in the user interface processing unit that performs window display of the content playback unit (B102), when a phrase playback button is pressed, a phrase that characterizes the hierarchy is played back. Such a phrase can be extracted by calculating a partial section having an expression vector similar to the expression vector of the segment corresponding to the entire hierarchy. For example, the top three sections with high similarity are associated with the phrase playback buttons 1 to 3. When the phrase playback button 1 is pressed, the most similar partial section is played back. When the phrase playback button 2 is pressed, the second similar partial section is played back. When the phrase playback button 3 is pressed, 3 The subsection similar to the th is played.

  FIG. 9 is a diagram for explaining an implementation example of the semantic structure processing device for presenting the conference content summary. In FIG. 9, B113 is a computer in which an utterance interval recording unit (B9), an utterance interval extraction unit (B10), and an audio feature extraction unit (B11) are mounted. A microphone to be used and an external storage device for recording moving images and audio are connected.

  B112 is a computer on which the semantic structure presentation unit (B8) is mounted, and is connected to a user interface such as a display, a keyboard, a mouse, and a speaker. Although it can be mounted on the computer B 113, it can also be mounted on another computer connected by a communication line as shown in this figure.

  B111 includes a subword encoding unit (B1), an atomic segment extraction unit (B2), a similarity matrix calculation unit (B3), an independence relaxation unit (B4), a redundant component removal unit (B5), and an optimum structure calculation unit. (B6) is a computer on which the semantic structure information recording unit (B7) is mounted. Although computing resources can be installed on the user side, as shown in FIG. 9, it can also be constructed on a cloud computing environment on the Internet. In this case, the computer B 112 and the computer B 113 can provide a content analysis service via a predetermined API.

  According to the content semantic structure processing device of the present invention, the content is semantically structured based on text or audio included in the multimedia content, and the content structure presenting device is used to reflect the semantic structure. By selectively reproducing the partial hierarchies, for example, it is possible to grasp the outline of the content for one hour in a few minutes, and it is used as an interface device for multimedia content having a summary function.

Claims (3)

A content structuring method for inputting text or audio included in media content, executing data processing on the input text or audio by a computer, and semantically structuring the content,
A content encoding process that converts input text or audio into a code string;
An atomic segment extraction process for extracting an atomic segment as a minimum unit of analysis for the code string;
A similarity matrix calculation process in which the similarity between the extracted atomic segments is calculated by the string kernel and stored in the memory as a similarity matrix;
An independence relaxation process that reduces the diagonal components of the similarity matrix to highlight the similarity between atomic segments;
A redundant component removal process for removing an average appearance pattern of a partial code string that does not contribute to semantic structuring;
An optimal structure calculation process that calculates a globally optimal structure using dynamic programming;
A content structuring processing method characterized in that each processing is executed by a computer. A content structuring apparatus that inputs text or audio included in media content, executes data processing on the input text or audio by a computer, and semantically structures the content,
Content encoding means for converting input text or audio into a code string;
Atomic segment extraction means for extracting an atomic segment as a minimum unit of analysis for the code string;
A similarity matrix calculation means for calculating the similarity between the extracted atomic segments by using a character string kernel and storing it in a memory as a similarity matrix;
Independence mitigation means for reducing the diagonal component of the similarity matrix to make the similarity between atomic segments stand out,
Redundant component removing means for removing a partial code string that is not useful according to a predetermined criterion in semantic structuring;
A media content structuring processing apparatus comprising: an optimum structure calculating means for calculating an optimum structure by dynamic programming. A content structuring processing program for inputting text or audio included in media content, executing data processing on the input text or audio by a computer, and semantically structuring the content,
A content encoding step of converting the input text or sound into a code string;
An atomic segment extraction step of extracting an atomic segment as a minimum unit of analysis for the code string;
A similarity matrix calculation step of calculating the similarity between the extracted atomic segments by a string kernel and storing it in a memory as a similarity matrix;
An independence relaxation step for reducing the diagonal components of the similarity matrix to make the similarity between atomic segments stand out,
A redundant component removal step of removing a partial code string that is not useful according to a predetermined criterion in semantic structuring;
A content structuring processing program for causing a computer to execute processing with an optimal structure calculating step for calculating a globally optimal structure by dynamic programming.

Images