JP7682525B2 - Content similarity calculation system and content search and recommendation system - Google Patents

Description

The present invention relates to a technology for calculating content similarity, and in particular to a technology that is useful when applied to a content similarity system and a content search and recommendation system that calculates the similarity of media content that has a development or plot that changes over time.

In recent years, a huge amount of digital media content (e.g., novels, music, movies, videos, etc.) has been available all over the Internet, and users have more opportunities to enjoy this content. It is important to establish a method to effectively acquire and recommend content that matches the user's hobbies, tastes, and intentions. In particular, it is necessary to establish a method that searches and recommends content based on the meaning and impression of media content, as a type of so-called emotional search, rather than simply pattern matching.

Generally, these media contents have developments and plots that change over time, and their meanings and impressions change over time. For example, novels and other works may have similar developments and plots even if the expression methods and techniques differ, and it may be possible to infer the author's background from the development and plot. The semantic characteristics of media content are expressed not only by the characteristics of the media content itself, but also by changes in characteristics over time. When evaluating a work expressed as media content, it is important to examine not only the meaning and impression as a whole, but also the meaning and impression evoked by each scene.

In general time series data analysis, it is important to use an appropriate method to conceptually grasp the similarity between two pieces of time series data. Broadly speaking, there are two methods: one that directly calculates the similarity between time series data, and one that converts the time series data into frequency data and then calculates the similarity.

There are various known methods for directly calculating the former, such as Dynamic Time Warping (DTW), Edit distance with Real Penalty (ERP), Longest Common Subsequence (LCS), Edit Distance on Real sequence (EDR), and Fast Time Series Evaluation (FTSE). In addition, there are methods for converting the latter to frequency data and calculating the latter, such as a method using the Fourier transform (Non-Patent Document 1, Non-Patent Document 2, etc.) and a method using the Haar wavelet.

As a technique that takes into account changes in the meaning of content over time rather than simply pattern matching, for example, JP 2021-9633 A (Patent Document 1) describes an information retrieval method that analyzes the content of search target data to calculate an emotion vector in which each emotion item and its emotion amount are associated, calculates an emotion vector for an event requested by a user based on the user's response to a reference sample, and extracts search target data that shows an emotion vector close to the emotion vector calculated for the user by performing a comparative calculation between the emotion vector calculated for the search target data and the emotion vector calculated for the user. It is said that the story-like nature of the time series can be taken into account by including in the emotion items an item whose emotion amount is increased when it is determined that multiple specific conditions are satisfied in time series.

JP 2021-9633 A

R.Agrawal, C.Faloutsos, A.Swami, "Efficient similarity search in sequence databases", In International conference on foundations of data organization and algorithms, pp. 69-84, 1993. D. Rafiei, A. O. Mendelzon, "Efficient retrieval of similar time sequences using DFT", In 1998 International Conference on Data Organization (FODO'98), 1998.

For example, by applying conventional methods for calculating similarity of time-series data, including those described in Non-Patent Documents 1 and 2, to media content, it is possible to evaluate the similarity of the media content data itself. However, it is difficult to evaluate the similarity taking into account the semantic content, such as the transition of meaning and impressions over time in the media content.

On the other hand, the technology described in Patent Document 1 can take into account to a certain extent the change in the meaning of the content over time. However, even though it is a time series, it can only take into account the order and frequency of appearance of items, and cannot take into account the time interval at which the meaning changes.

The object of the present invention is to provide a content similarity calculation system that grasps the changes and transitions in the meaning and impression of media content on a time axis and in a chronological order, and calculates the similarity based on the development and plot of the media content, and a content search and recommendation system that uses the same.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

The following is a brief summary of the representative inventions disclosed in this application:

A content similarity calculation system according to a representative embodiment of the present invention is a content similarity calculation system that calculates the similarity between media contents whose contents change over time, and includes a semantic waveform extraction unit that extracts changes in the degree of influence of multiple semantic items for two or more input media contents and outputs them as semantic waveform data, a time-frequency domain conversion unit that obtains and outputs a semantic frequency spectrum from the semantic waveform data of each of the media contents by Fourier transform, and a similarity calculation unit that generates a semantic frequency spectrum vector for each of the media contents, whose elements are the data of each of the semantic frequency spectrums, and calculates and outputs the cosine similarity between the semantic frequency spectrum vectors.

The effects achieved by the representative inventions disclosed in this application can be briefly explained as follows:

In other words, according to a representative embodiment of the present invention, it is possible to grasp the changes and transitions in the meaning and impression of media content on a time axis and in a chronological order, and calculate the similarity based on the development and plot of the media content.

1 is a diagram showing an overview of a configuration example of a content similarity calculation system according to a first embodiment of the present invention; FIG. 2 is a diagram illustrating an overview of a content similarity calculation process according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an overview of a semantic waveform extraction process according to the first embodiment of the present invention. FIG. 2 is a diagram showing an overview of an example of extracting features in time series from media content in the first embodiment of the present invention. FIG. 2 is a diagram illustrating an overview of a time-frequency domain transform process according to the first embodiment of the present invention. FIG. 11 is a diagram showing an overview of a configuration example of a content search and recommendation system according to a second embodiment of the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings. In all drawings used to explain the embodiment, the same parts are generally given the same reference numerals, and repeated explanations will be omitted. However, there are cases where parts that have been described with reference numerals in one drawing will be referred to with the same reference numerals when explaining other drawings, although they will not be shown again.

(Embodiment 1)
＜Overview＞
The content similarity calculation system according to the first embodiment of the present invention grasps the time-series changes and transitions (semantic transitions) in the meanings and impressions of media content whose contents change over time, and extracts semantic waveforms that express these as waveforms.The semantic waveforms extracted for each of the multiple semantic items in the media content are then converted to the frequency domain by Fourier transform to obtain frequency spectra, which are then expressed as vectors.The information processing system then calculates the cosine similarity between the vectors obtained for each of the media contents, thereby calculating the similarity between the media content based on the semantic transitions.

<System Configuration>
1 is a diagram showing an overview of a configuration example of a content similarity calculation system 1 according to a first embodiment of the present invention. The content similarity calculation system 1 is, for example, a server system such as a server device or a virtual server constructed on a cloud computing service, or a computer system such as a PC (Personal Computer). A CPU (Central Processing Unit) (not shown) executes middleware such as an OS (Operating System), a DBMS (DataBase Management System), and a Web server program, which are expanded onto a memory from a recording device such as a HDD (Hard Disk Drive), and software running thereon, thereby realizing various functions related to similarity calculation of media content, which will be described later.

The content similarity calculation system 1 has various components, such as a content input unit 11, a semantic waveform extraction unit 12, a time-frequency domain conversion unit 13, and a similarity calculation unit 14, which are implemented as software.

The content input unit 11 has a function of accepting input of media content for which similarity is to be calculated. The input of a data file of the media content may be accepted directly via an input/output device (not shown), or may be accepted by uploading via a network 2 such as the Internet. The media content that is the subject of the similarity calculation does not necessarily have to be of the same type (for example, two novels, two movies, etc.). In this embodiment, when calculating the similarity, the data of the media content is not directly compared, but rather the semantic waveform is extracted and converted into the frequency domain for similarity calculation, so that similarity calculation can be performed even for different types of media content.

The semantic waveform extraction unit 12 has a function of extracting changes on a time axis/time series for each of multiple feature items from the media content input via the content input unit 11, and based on this, outputting semantic waveform data showing transitions on a time axis/time series for the degree of influence of each of multiple semantic items that represent the meaning and impression of the content. The functions and processing contents of the semantic waveform extraction unit 12 will be described later.

The time-to-frequency domain conversion unit 13 has a function of performing a Fourier transform on the semantic waveform data relating to each semantic item output from the semantic waveform extraction unit 12, converting the data from the time domain into data in the frequency domain, thereby acquiring and outputting a semantic frequency spectrum. The functions and processing contents of the time-to-frequency domain conversion unit 13 will also be described later.

The similarity calculation unit 14 has a function of calculating and outputting the similarity between the media contents for which similarity is to be calculated, based on the semantic frequency spectrum data output from the time-frequency domain conversion unit 13. There are no particular limitations on the calculation of similarity based on the semantic frequency spectrum data, but for example, a method can be used in which, for each media content to be calculated, a vector (semantic frequency spectrum vector) is generated whose elements are the semantic frequency spectrum data for each semantic item, and the cosine similarity between these vectors is calculated.

<Contents of content similarity calculation processing>
2 is a diagram illustrating an overview of the content similarity calculation process in the first embodiment of the present invention. For a plurality of media contents A (41a) and B (41b) to be calculated, which are input via the content input unit 11, the semantic waveform extraction unit 12 performs semantic waveform extraction processes (S1a, S1b) to obtain semantic waveforms A (42a) and B (42b).

The time-frequency domain conversion unit 13 performs time-frequency domain conversion processing (S2a, S2b) on the semantic waveform A (42a) and the semantic waveform B (42b), respectively, to obtain the semantic frequency spectrum A (43a) and the semantic frequency spectrum B (43b). The data of the semantic frequency spectrum A (43a) and the semantic frequency spectrum B (43b) are then vectorized by the similarity calculation unit 14 to obtain semantic frequency spectrum vectors, and a similarity calculation process (S3) is performed to calculate the cosine similarity between these vectors, thereby calculating the similarity 45 between the media content A (41a) and the media content B (41b).

The details of each process are explained below.

<Semantic Waveform Extraction Processing>
3 is a diagram illustrating an outline of the semantic waveform extraction process in the first embodiment of the present invention. In this embodiment, a plurality of semantic waveforms 42 are extracted for each semantic item from media content 41 that has a development or plot, i.e., whose meaning or impression changes over time t. The example in FIG. 3 shows that a semantic waveform 42 is extracted for each of two semantic items, "joy" and "sadness." Thus, in this embodiment, the number of semantic waveforms 42 extracted is the same as the number of semantic items (perspectives of meaning and impression) to be evaluated.

In the semantic waveform extraction process, first, the media content 41 is divided into multiple windows 51 separated by a predetermined time width (step 1). Here, the window size is set to the smallest time width that can express the meaning, and the target media content 41 is divided into multiple windows 51 for each set window size from the beginning (or from a specified position).

Then, multiple feature items are extracted from the media content 41 divided into multiple windows 51 (step 2). This makes it possible to obtain the time-series changes in the feature items appearing in each window 51. Note that the feature items here can be, for example, sentences, words, images, music, etc. that appear directly in the media content 41 and relate to meanings or impressions. After that, each feature item for each window 51 is converted into a semantic item (for example, "joy" or "sadness") (step 3).

Then, the magnitude and degree of the influence of each semantic item is quantified for each window 51 (step 4), and a semantic waveform 42 is obtained that shows the changes over time as a waveform (step 5). The example in Figure 3 shows an example of a semantic waveform 42 obtained for two semantic items, "joy" and "sadness," in media content 41.

In the above processing of steps 2 and 3, the method of converting feature items into semantic items can be, for example, an existing general sensibility/emotion analysis method such as negative/positive judgment or sentiment judgment for sentences and words. It is also possible to use a method of grasping the change in the share of topics (semantic items) over time in media content 41, for example, by using a topic model known as a method of statistical latent semantic analysis of sentences.

It is also possible to use the Media-lexicon Transformation Operator, a framework for extracting impression words from media content, proposed in the literature (T. Kitagawa and Y. Kiyoki, Y, "Fundamental framework for media data retrieval system using media lexicon transformation operator", Information Modeling and Knowledge Bases, vol.12, pp.316-326, 2001). The Media-lexicon Transformation Operator is a mechanism that uses research, reviews, statistics, etc. by experts in the field related to the target media content to extract impression words that humans get from that media content.

4 is a diagram showing an overview of an example of extracting features in a time series from media content 41 in the first embodiment of the present invention. In this embodiment, the above-mentioned Media-lexicon Transformation Operator, which is originally intended for static one-dimensional data, is extended to the time axis for use. For example, a word matrix Y is obtained from the correspondence between the appearance frequency of words ( _w1 , _w2 , ..., _wm ) related to feature items appearing in each window 51 in the media content 41 and the time ( _t1 , _t2 , _... ) of each window 51 (for example, grasped by the elapsed time from the start of _{the media content 41), and this is converted into a product of the conversion matrix T (prepared in advance as dictionary data) consisting of the relationship between the words (w1 , w2 ,} ..., _wm ) and semantic items ( _x1 , _x2 , ..., _xn ) indicating meanings and impressions, and a media content matrix X consisting of the relationship between the semantic items ( _x1 , x2, ..., xn) and the time (t1, _t2 , ...) of each window 51. The media content matrix X makes it possible to grasp the transition of each semantic item in time series.

<Time-frequency domain transformation processing>
5 is a diagram illustrating an overview of the time-frequency domain conversion process in the first embodiment of the present invention. Here, the data of the semantic waveform 42 of each semantic item (in the example of FIG. 5, "joy" and "sadness") is subjected to a discrete Fourier transform using a Fast Fourier Transform (FFT) algorithm to convert from the time domain to the frequency domain, thereby obtaining a semantic frequency spectrum for each semantic item (in the example of FIG. 5, a semantic frequency spectrum 43a for "joy" and a semantic frequency spectrum 43b for "sadness").

<Similar calculation processing>
In the similarity calculation process, for each media content 41, the semantic frequency spectrum for each semantic item obtained by the above series of processes is vectorized as a column of elements to obtain a semantic frequency spectrum vector 44. Then, the cosine similarity of the semantic frequency spectrum vector 44 for each media content 41 is calculated to obtain the similarity. The value of the cosine similarity may be used as is for the similarity, or this value may be compared with a predetermined reference value to determine whether or not the contents are similar, or to evaluate the degree of similarity in stages.

<Conclusion>
As described above, according to the content similarity calculation system 1 which is embodiment 1 of the present invention, it is possible to grasp the transitions in meaning and impression in the media content 41 on a time axis and in a chronological order, and to calculate the similarity between contents based on the development and plot of the media content 41.

In addition, when performing similarity calculations, the data of the media contents 41 are not directly compared, but rather the semantic waveform 42 is extracted from the media contents 41 and converted into the frequency domain to perform similarity calculations based on the semantic frequency spectrum 43. This makes it possible to perform similarity calculations even for different types of media contents 41, such as novels and movies.

Furthermore, when calculating the similarity, a simple method is used in which a semantic frequency spectrum vector 44 is generated for each media content 41 by vectorizing multiple semantic frequency spectra 43, and the cosine similarity between these vectors is calculated to obtain the similarity, making it possible to calculate the similarity between media contents 41 at low cost.

(Embodiment 2)
The content search and recommendation system according to the second embodiment of the present invention is an information processing system that uses the content similarity calculation system according to the first embodiment described above to search for and recommend media content.

<System Configuration>
6 is a diagram showing an overview of a configuration example of a content search and recommendation system according to a second embodiment of the present invention. The content search and recommendation system 3 is, for example, a server system such as a server device or a virtual server constructed on a cloud computing service, or a computer system such as a PC. Then, a CPU (not shown) executes middleware such as an OS, a DBMS, and a Web server program that are expanded from a recording device such as an HDD onto a memory, and software that runs on the middleware, thereby realizing various functions (described later) related to searching and recommending media content based on conditions specified by a user.

The content search and recommendation system 3 has various parts, such as a condition content input unit 31, a content extraction unit 32, a content similarity calculation unit 33, and a result content output unit 34, which are implemented as software. It also has a content database (DB) 35 that stores multiple media contents. The content search and recommendation system 3 extracts media contents similar to media contents specified by the user as search conditions from the media contents stored in the content DB 35, and outputs them as search results or recommended content.

The condition content input unit 31 has a function of accepting input of media content to be compared as a condition when searching for or recommending media content. The input of a data file of the media content may be accepted directly via an input/output device (not shown), or the input may be accepted by uploading via a network 2 such as the Internet.

The content retrieval unit 32 has a function of sequentially retrieving media content stored in the content DB 35 as search targets. It may be set to sequentially retrieve all media content, or it may narrow down the range of media content to be retrieved from the content DB 35 based on the contents, metadata, and other information of the media content of the search criteria inputted via the condition content input unit 31, and sequentially retrieve media content from among them.

The content similarity calculation unit 33 has a function of calculating the similarity between media contents by inputting the media contents of the search criteria inputted via the condition content input unit 31 and the media contents sequentially extracted from the content DB 35 by the content extraction unit 32. When implementing, for example, the content similarity calculation system 1 described in the above-mentioned first embodiment (more precisely, each unit implemented as software) can be used, and such a configuration is also adopted in this embodiment.

The result content output unit 34 has a function of outputting to the user as search results or recommended contents a certain number of media contents (or similarities equal to or greater than a certain value) from among the media contents in the content DB 35, in order of the highest similarity to the media content specified by the search criteria, based on the similarity information calculated by the content similarity calculation unit 33. The output method is not particularly limited, and may be displayed on a display (not shown) provided in the content search and recommendation system 3, or may be extracted as a data file or made available for the user to download via the network 2.

<Conclusion>
As described above, according to the content search and recommendation system 3 which is embodiment 2 of the present invention, it is possible to extract media content similar to media content specified by the user as a search condition from among media content stored in the content DB 35, and output the content as the search result or recommended content.

The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the gist of the invention. Furthermore, the above embodiments have been described in detail to explain the invention in an easy-to-understand manner, and the invention is not necessarily limited to having all of the configurations described. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a recording device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

In addition, in each of the above diagrams, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily show all of the control lines and information lines in the actual implementation. In reality, it can be assumed that almost all components are interconnected.

The present invention can be used in content similarity systems and content search and recommendation systems that calculate the similarity of media content that has a development or plot that changes over time.

1...Content similarity calculation system, 2...Network, 3...Content search and recommendation system,
11 ... content input unit, 12 ... semantic waveform extraction unit, 13 ... time-frequency domain conversion unit, 14 ... similarity calculation unit,
31: condition content input unit, 32: content extraction unit, 33: content similarity calculation unit, 34: result content output unit, 35: content DB,
41...media content, 41a...media content A, 41b...media content B, 42...semantic waveform, 42a...semantic waveform A, 42b...semantic waveform B, 43...semantic frequency spectrum, 43a...semantic frequency spectrum A, 43b...semantic frequency spectrum B, 44...semantic frequency spectrum vector, 45...similarity, 51...window

Claims (3)

A content similarity calculation system for calculating a similarity between media contents whose contents change over time, comprising:
a semantic waveform extraction unit for extracting changes in the degree of influence of a plurality of semantic items for each of two or more input media contents and outputting the changes as semantic waveform data;
a time-frequency domain transform unit that performs a Fourier transform on the semantic waveform data of each of the media contents to obtain and output a semantic frequency spectrum;
a similarity calculation unit that generates a semantic frequency spectrum vector having data of each of the semantic frequency spectra as elements for each of the media contents, calculates and outputs a cosine similarity between the semantic frequency spectrum vectors;
A content similarity calculation system comprising: 2. The content similarity calculation system according to claim 1,
The semantic waveform extraction unit
A content similarity calculation system which obtains a media content matrix whose product with the transformation matrix becomes the word matrix based on a word matrix showing the correspondence between the number of occurrences of words related to feature items in each window obtained by dividing the media content into a predetermined time width and the time of each window, and a transformation matrix showing the correspondence between the words related to the feature items and each of the semantic items, and obtains data for each of the semantic waveforms based on the media content matrix data. A content search and recommendation system for searching for media content similar to a specified second media content from a content storage unit in which a plurality of first media content whose contents change over time is stored, comprising:
A content similarity calculation unit comprising the content similarity calculation system according to claim 1;
a result content output unit that outputs, as a search result, the first media content whose similarity satisfies a predetermined condition based on the similarity between each of the first media content and the second media content calculated by the content similarity calculation unit;
A content search and recommendation system having the above.

Images