JP6712796B2 - Method and apparatus for learning between different language documents via image, cross-language document retrieval method and apparatus - Google Patents

Description

The present invention relates to cross-language document search.

Cross-language document search or cross-language document search is a technique for inputting a Japanese document and searching for related/similar English documents. In the conventional technology, a large amount of bilingual corpus (for example, a document set written in both Japanese and English) is required to train the system, and it is generally difficult to obtain such data. Therefore, it was not feasible.

Specifically, manually translating a large amount of documents requires a great deal of effort. It is also possible to crawl bilingual documents from the Web and use them as learning data, but many documents on the Web are closed in one language. Therefore, it is not easy to collect a sufficient amount of multilingual documents, especially in minor languages.

Therefore, we paid attention to multimedia information that has been abundantly present on the Web in recent years, especially document and image pairs. If two documents written in different languages both contain images and the image features are similar, it can be assumed that the text contained in the documents will be similar. In addition to the fact that images can be included in documents of any country, in addition to being able to understand the meaning and content included in images regardless of their native language, images have the advantage of being a universal expression. ..

Machine learning in traditional image recognition was very poor compared to machine learning in the field of natural language processing. However, in recent years, the accuracy of image recognition has rapidly approached that of humans due to a breakthrough in deep learning (Non-Patent Document 12).
Douglas J Carroll. 1968.Generalization of canonical correlation analysis to three or more sets of variables.In Proceedings of the 76th Annual Convention of the American Psychological Association, volume 3, pages 227-228. Jon Robers Kettenring. 1971. Canonical Analysis of Several Sets of Variables. Biometrika, 58(3):433-451. Michel Velden and Yoshio Takane. 2012. Generalized Canonical Correlation Analysis with Missing Values. Computational Statistics, 27(3):551-571. Jan Rupnik, Andrej Muhic, and Primo Skraba. 2012. Cross-Lingual Document Retrieval through Hub Languages. In Neural Information Processing Systems Workshop. Harold Hotelling. 1936. Relations between Two Sets of Variants. Biometrika, 28:321-377. David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical Correlation Analysis: an Overview with Application to Learning Methods. Neural Computation, 16(12):2639-2664. Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010. A New Approach to Cross-modal Multimedia Retrieval. Proceedings of the International Conference on Multimedia, pages 251-260. Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2014. A Multiview Embedding Space for Modeling Internet Images, Tags, and their Semantics.International Journal of Computer Vision, 106(2):210-233. Alexei Vinokourov, John Shawe-Taylor, and Nello Cristianini. 2002. Inferring a Semantic Representation of Text via Cross-Language Correlation Analysis. Advances in Neural Information Processing Systems, pages 1473-1480. Yaoyong Li and John Shawe-Taylor. 2004. Using KCCA for Japanese-English Cross-language Information Retrieval and Classification. In Learning Methods for Text Understanding Raghavendra Udupa and Mitesh M Khapra. 2010. Improving the Multilingual User Experience of Wikipedia Using Cross-Language Name Search. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 492-500. Hao Fang, Saurabh Gupta, Forrest Iandola, K. Rupesh Srivastava, Li Deng, Piotr Doll´ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the Devil in the Details: Delving Deep into Convolutional Nets. In Proceedings of the British Machine Vision Conference, pages 1-11. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2013.DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Proceedings of the International Conference on Machine Learning, pages 647-655. . Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the ACM International Conferenceon Multimedia, pages 675-678. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision. Christian Szegedy, Scott Reed, Pierre Sermanet, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. CoRRabs/1409.4842. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, pages 1097-1105. Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying Conditional Random Fields to Japanese Morphological Analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230-237. Cyrus Rashtchian, Peter Young, Micah Hodosh, Julia Hockenmaier, and North Goodwin Ave. 2010. Collecting Image Annotations Using Amazon's Mechanical Turk. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics Workshop on Creating Speech. and Language Data with Amazon's Mechanical Turk, pages 139-147. Florent Perronnin, Jorge S´anchez, and Thomas Mensink. 2010. Improving the Fisher kernel for large-scale image classification. In Proceedings of the European Conference on Computer Vision, pages 143-156. David G Lowe. 1999. Object recognition from local scale-invariant features.In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pages 1150-1157.

An object of the present invention is to propose a method and apparatus for learning between different language documents that indirectly use images, and to perform a different language document search using the learning model.

An image-mediated learning method between different language documents according to the present invention is
Prepare a first training data set consisting of a pair of first language documents and images,
Prepare a second training data set consisting of a pair of second language documents and images,
Extracting a first feature vector from the first language document and a second feature vector from the image in the first training data set;
In the second training data set, a third feature vector is extracted from the second language document, a second feature vector is extracted from the image,
By performing a generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, mapping of the first feature vector and the third feature vector is performed through the second feature vector.
It is a thing.

In one aspect, the first feature vector and the third feature vector are extracted using Bag of words.
Bag of words is one representative example for extracting the feature amount of a document, and is not limited to this.

In one aspect, the second feature vector is extracted using a convolutional neural network.
Examples of CNN include AlexNet, CaffeNet, GoogLeNet and VGG net.

In one aspect, the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced by the dimension reduction means.
A principal component analysis (PCA) is typically illustrated as a dimension reduction means.

In one aspect, the first training data set includes multimedia data obtained by crawling from a web in a first language,
The second training data set includes multimedia data obtained by crawling from the second language web.

In one aspect, further, a third training data set including a pair of first language document and second language document is prepared,
Extracting a first feature vector from the text of the first language and a third feature vector from the text of the second language in the third training data set;
In the generalized canonical correlation analysis, the first feature vector and the third feature vector extracted from the third training data set are further used.
This mode corresponds to the Few-Shot learning described later.

The learning device between different language documents mediated by images according to the present invention,
A first training data set consisting of a pair of first language documents and images;
A second training data set consisting of a pair of second language documents and images;
First feature vector extraction means for extracting a first feature vector from the first language document,
Second feature vector extraction means for extracting a second feature vector from the image,
Third feature vector extraction means for extracting a third feature vector from the second language document,
Generalized canonical correlation analysis means,
Equipped with
The generalized canonical correlation analysis means performs a generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, so that the second feature vector serves as a medium for the first feature vector and the first feature vector. Mapping the third feature vector,
It is a thing.

The cross-language document search method according to the present invention uses a learning model obtained by the learning method between different language documents,
In the learning model, the first projection coefficient from the first linguistic space (the space of the "first feature vector") to the canonical space, and the second projection space from the second linguistic space (the space of the "third feature vector") to the canonical space The second projection coefficient is specified,
Extracting a first feature vector from the first language query document,
The extracted first feature vector is projected onto the canonical space using the first projection coefficient to obtain the first projected feature vector,
Extracting a third feature vector from the second language target document candidate,
The extracted third feature vector is projected onto the canonical space using the second projection coefficient to obtain the third projected feature vector,
The target document is determined using the similarity between the first projected feature vector and the third projected feature vector.

A cross-language document search device according to the present invention uses a learning model obtained by a learning method between the different language documents,
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
Means for extracting a first feature vector from the first language query document;
A means for projecting the extracted first feature vector on a canonical space using a first projection coefficient to obtain a first projected feature vector;
Means for extracting a third feature vector from the second language target document candidate,
A means for projecting the extracted third feature vector on a canonical space using a second projection coefficient to obtain a third projected feature vector;
Means for determining a target document using the similarity between the first projected feature vector and the third projected feature vector;
Is equipped with.

In the present invention, in learning between different language documents, it is possible to perform learning even when there is no parallel translation corpus by interposing images (Zero-shot learning).
In the present invention, in learning between different language documents, it is possible to perform learning even when the bilingual corpus is small by mediating images (Few-shot learning).

A bilingual corpus is required even in a study (Non-Patent Document 4) that enables learning by using documents such as English as a hub in learning between different language documents.
Information is needed across two languages.
On the other hand, the present invention is characterized in that learning can be performed only with information closed in one language on the assumption that multimedia information is attached (see FIG. 1).
Since images are universal expressions, they are included in documents for each language (documents closed in one language), and they serve as a bridge (mediation) during learning.

The present invention focuses on multimedia information that has been abundant on the web in recent years, and uses a learning method that indirectly uses images, in combination with recent breakthroughs in image recognition technology Implement cross-language document retrieval without a corpus (or with a few bilingual corpora).

It is a conceptual diagram of image mediated learning. 3 illustrates a web document containing image data. 1 is a schematic view of a learning system between different language documents via an image according to the present invention. FIG. 1 is a schematic diagram of a cross-language document search system according to the present invention. It is a schematic diagram of a learning system between different language documents according to an embodiment of the present invention. It is a conceptual diagram which shows the projection of the feature vector of a Japanese document, the feature vector of an English document, and the feature vector of an image to the canonical space using generalized canonical correlation analysis. It is a conceptual diagram of the nearest neighbor search in the projected space (canonical space). It is a figure which illustrates the data set used for experiment. Five English sentences and five Japanese sentences corresponding to each English sentence are prepared for one image. Feature vectors extracted from five Japanese sentences and feature vectors extracted from one image form a pair, and feature vectors extracted from five English sentences and feature vectors extracted from one image form a pair. Is forming. The result of the search precision experiment 1 is shown. Search accuracy experiment 1 was performed by changing the number of samples in [train-E/I] and [train-I/J], and by changing the number of samples in [train-E/J]. I went. The result of the search precision experiment 2 is shown. Search accuracy experiment 2 is a comparison experiment using a plurality of types of image features.

[A] Outline of Image-Mediated Learning System An outline of the image-mediated learning system according to the present invention will be described with reference to FIG.
[A-1] Data used for learning First, a first training data set and a second training data set are prepared. The first training data set consists of a pair of first language documents and images. The second training data set consists of a pair of second language documents and images. In the experimental example described later, Japanese and English were used as the first language and the second language, but the first language and the second language can be selected from arbitrary different languages. In one aspect, the images included in the first training data set and the images included in the second training data set are completely different images so that no overlap occurs. In addition, it does not exclude that the same image is included.

In the first training data set, the first feature vector is extracted from the first language document and the second feature vector is extracted from the image. In the second training data set, the third feature vector is extracted from the second language document and the second feature vector is extracted from the image. Learning is performed using these feature vectors. The feature vectors extracted from the image are collectively referred to as the second feature vector, but the second feature vector obtained from the first training data set and the second feature vector obtained from the second training data set are different. ..

Although the UIUC Pascal Sentence Dataset (Non-Patent Document 19) was used in Examples described later, various data can be considered as a source of the training data set. In one aspect, the first training data set includes multimedia data obtained by crawling from a first language web and the second training data set is multimedia data obtained by crawling from a second language web. Contains media data. SNS such as Twitter, news articles, blog articles, etc. are rich in multimedia data and can be obtained by crawling. Moreover, the training data set is not limited to the data acquired from the Web. For example, a paper medium (book, advertisement leaflet) including an image such as a photograph and a document may be scanned to obtain a pair of image data and document data. Alternatively, a text/multimedia pair may be obtained from the video and subtitles of a television program.

[A-2] Extraction of Text Feature In one aspect, the first feature vector and the third feature vector are extracted using Bag of words (BoW). BoW is one typical example for extracting the feature amount of a document, and the following methods are exemplified as non-limiting methods other than BoW.
TF-IDF features;
Using the word vector obtained by Word2vec, adding all the words in the document and taking the average;
Document vector obtained from Paragraph2vec;
Features by N-gram language model;
BoW reduced dimensionally using LSI (Latent Semantic Indexing, LSA: also called Latent Semantic Analysis);
BoW is dimensionally reduced by LDA (Latent Dirichlet Allocation).

[A-3] Image Feature Extraction In one aspect, the second feature vector is extracted using a convolutional neural network (CNN). The convolutional neural network (CNN) is the most successful deep learning as image recognition (Non Patent Literatures 12 to 17). Examples of CNN include AlexNet, CaffeNet, GoogLeNet, and VGG net. CNN has various structures depending on the designer. CNN updates the weights that connect each layer so that classification is performed correctly when learning. That is, the learned neural network has fixed weights, and when an image is input to this neural network, the value of the activation function is calculated based on each weight. The result is used as a feature quantity. There is no limitation on which layer value of the plurality of layers is used. Those skilled in the art will understand that the layer near the final layer of the network is generally considered to give good characteristics, but the best layer can be selected by experiments and the like. It should be noted that the second feature vector that can be used in the present invention does not exclude the feature amount other than the CNN, and includes, for example, Fisher vector, Bag of Features (Bag of Visual Words), and the like.

[A-4] Dimension Reduction of Feature Vector In one aspect, the first feature vector, the second feature vector, and the third feature vector may be dimensionally reduced by the dimension reduction means. The dimension reduction of each feature vector is arbitrary, but considering the calculation time, it is desirable to reduce the dimension when the scale of data is large. A principal component analysis (PCA) is typically exemplified as the dimension reduction means, but an independent component analysis (ICA), LDA, or LSI may be used.

[A-5] Generalized canonical correlation analysis (GCCA)
The generalized canonical correlation analysis is a generalization of the canonical correlation analysis that handles two variable groups to handle three or more variable groups and maximizes the sum of correlations between multiple modalities. Data can be mapped to. By performing the generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, the first feature vector and the third feature vector are mapped through the second feature vector.

Learning using canonical correlation analysis (CCA) that handles two variable groups and application to CLDR are known (Non-Patent Documents 5 to 11). The GCCA is a generalized CCA for m (m=3 in this embodiment) modalities.

Representative examples of GCCA include those described in Non-Patent Document 1 (Carroll), Non-Patent Document 2 (Kettenring), Non-Patent Document 3 (Velden et al), and Non-Patent Document 4 (Rupnik et al). Are known. In Non-Patent Document 4, the method of Non-Patent Document 2 (also partially referred to in Non-Patent Document 1) is adopted. In the experiment described below, the GCCA of Non-Patent Document 2 was adopted, but it is understood by those skilled in the art that other GCCA may be adopted to realize the present invention.

[A-6] Image-mediated interlingual learning model
The GCCA determines the first projection coefficient from the first language space to the canonical space and the second projection coefficient from the second language space to the canonical space. That is, in the learning model obtained by the learning method between different language documents, the first projection coefficient from the first language space to the canonical space and the second projection coefficient from the second language space to the canonical space are defined. There is. The first feature vector extracted from the first language document is converted into the first projective feature vector by the first projective coefficient. The third feature vector extracted from the second language document is converted into the third projective feature vector by the second projective coefficient. The first projective feature vector and the third projective feature vector can be compared in a canonical space (broadly speaking, “latent space”) which is a common space or a joint space.

[A-7] Few-Shot Learning One ideal form of the present invention is Zero-Shot learning for realizing cross-language document retrieval without any bilingual corpus, but a small amount of bilingual corpus is used. You may perform the Few-Shot learning that you did. In this case, a third training data set consisting of a pair of first language document and second language document is prepared, and the first feature vector is extracted from the text of the first language in the third training data set, and the second language data is extracted. The third feature vector is extracted from the text, and the first feature vector and the third feature vector extracted from the third training data set are further used in the generalized canonical correlation analysis.

[A-8] Hardware Configuration The learning system according to the present invention is configured by one or a plurality of computers, and the computer includes a processing unit (CPU or the like) as a hardware, a storage unit (a hard disk, a RAM, ROM, etc.), input means, output means or display means, control programs for operating a computer as software, and the like. A first training data set including a first language document/image pair group and a second training data set including a second language document/image pair group are stored in the storage means. The means for extracting the features from the text data and the means for extracting the features from the image data are composed of processing means. The first feature vector extracted from the first language document, the second feature vector extracted from the image, and the third feature vector extracted from the second language document are stored in the storage means. The generalized canonical correlation analysis is executed by the processing means using the first feature vector, the second feature vector, and the third feature vector, and the generalized canonical correlation analysis is performed by using the second feature vector as a medium. Mapping of the first feature vector and the third feature vector is performed. Specifically, the first projection coefficient from the first linguistic space to the canonical space and the second projection coefficient from the second linguistic space to the canonical space are calculated, and the projection coefficient is stored in the storage means.

[B] Outline of Cross-Language Document Search System An outline of the cross-language document search system according to the present invention will be described with reference to FIG.
[B-1] Image-mediated interlingual learning model In the learning model obtained by the learning method between different language documents, the first projection coefficient from the first language space to the canonical space, and the second language space to the canonical space The second projection coefficient to space is specified. The first feature vector extracted from the first language document is converted into the first projective feature vector by the first projective coefficient. The third feature vector extracted from the second language document is converted into the third projective feature vector by the second projective coefficient. The similarity between the first language document and the second language document can be estimated from the similarity between the first projected feature vector and the third projected feature vector.

[B-2] Search When the first language query document is input to the search system, the first feature vector is extracted from the input text data. The extracted first feature vector is projected onto the canonical space using the first projection coefficient to obtain the first projected feature vector.

On the other hand, a third feature vector is extracted from the second language target document candidate, and the extracted third feature vector is projected onto the canonical space using the second projection coefficient to obtain the third projected feature vector. The third projection feature vector corresponding to each second language document is extracted in advance and stored in the storage unit, and the similarity described below is calculated using the prestored third projection feature vector. May be.

The target document is determined using the similarity between the first projected feature vector and the second projected feature vector. The degree of similarity is typically represented by the distance between vectors, and examples thereof include Eugrid distance, Mahalanobis distance, and Manhattan distance. Further, the cosine similarity may be used as the similarity. Typically, the candidate with the highest degree of similarity is output as the second language target document. Alternatively, a plurality of candidates having a high degree of similarity may be output as the second language target document, and may be ranked and displayed according to the degree of similarity.

How to set the second language target document candidate is not particularly limited. In one aspect, all second language documents available at the time the first language query document is entered are candidates. For example, all data acquired by crawling from the second language web may be targeted.

[B-3] Hardware Configuration The search system according to the present invention is configured by one or a plurality of computers, and the computer includes processing means (CPU etc.) as hardware, storage means (hard disk, RAM, ROM, etc.), input means, output means or display means, control programs for operating a computer as software, and the like. The user terminal is also composed of one or a plurality of computers, and the computer is provided with processing means, storage means, input means, output means or display means, control programs for operating the computer, and the like.

The search system and the user terminal are equipped with a transmission/reception means capable of exchanging information with each other via a computer network typified by the Internet, and are communicably connected to each other via a computer network typified by the Internet. There is. The search system is connected to an existing search engine via a computer network represented by the Internet. For example, a query screen is displayed on the screen of the user terminal, and the text data in the first language is input from the input means of the user terminal and transmitted as a search query to the search system. Note that you may upload documents instead of entering text data, and in the case of a recommendation system that automatically extracts and recommends similar sentences on the search system side, there is no interaction on the user side and the search result Is displayed on the user terminal. In one aspect, a plurality of second languages can be selected, and one or a plurality of second languages is designated. In the search system, the degree of similarity with the second language target document candidate is calculated based on the search query, and the search result can be browsed from the user terminal.

[C] Example
[C-1] Data Used In this embodiment, the different data divisions shown in Table 1 are used.
[train-E/I]: A learning document consisting of English text and image pairs
[train-I/J]: A learning document consisting of Japanese text and image pairs
[train-E/J]: A learning document consisting of English and Japanese text pairs
[test-E/J]: A test document consisting of a pair of English text and Japanese text
Each data division does not overlap. For example, the image data in [train-E/I] and the image data in [train-I/J] are different.

As shown in Table 1, an ID is defined for each modality of each division. For example, E1 represents an English feature in the [train-E/I] division. A typical CLDR based on the parallel corpora uses only [train-E/J] as training data and evaluates it using [test-E/J]. In the Zero-Shot learning scenario that does not use the [train-E/J] data in this experiment, only [train-E/I] and [train-I/J] are used as learning data. A few [train-E/J] samples are used in the Few-Shot learning scenario. In this specification, these learnings are collectively referred to as image-mediated learning.

[C-2] Schematic diagram of system A schematic diagram of the system according to the embodiment is shown in FIG. Learning is performed using three feature quantities of English, images, and Japanese. In learning, the first feature vector is extracted from the English text, the second feature vector is extracted from the image, the third feature vector is extracted from Japanese, and the obtained features are dimensionally reduced by principal component analysis (PCA). The reduced and reduced features are learned by GCCA.

In the test, the features obtained from the query Japanese text are dimensionally reduced by principal component analysis (PCA). As shown by the arrow in FIG. 5, the coefficients projected in low dimensions by PCA projection are learned by PCA in the learning phase. The reduced features are projected using the first projection coefficient obtained by GCCA, while the features obtained from the English text are dimensionally reduced by principal component analysis (PCA), and the reduced features are GCCA. Projection is performed using the second projection coefficient obtained by, and the nearest neighbor search from Japanese sentence to English sentence is performed in the joint space.

[C-3] Image feature extraction Image features are extracted using a convolutional neural network. In this embodiment, the CNN model that has been pre-learned using the ILSVRC2012 dataset (Non-Patent Document 15) and provided to Caffe (Non-Patent Document 16) is applied. In the experiment, the feature amount of the pool5/7x7 s1 layer of GoogLeNet (Non-patent document 17), the feature amount of the fc6 layer of VGG (Non-patent document 13), the fc6 layer of CaffeNet (Non-patent document 16, non-patent document 18) The feature quantity was used as the image feature vector.

[C-4] Text features As text features of English and Japanese, weighting by bag of words (BoW) and TF-IDF (term frequency-inverse document frequency) was used. The MeCab library is used for dividing Japanese sentences into words by morphological analysis (Non-Patent Document 18). In the experiment, pre-processing such as deletion of stop words and stemming was not performed, but these are optional.

[C-5] Generalized canonical correlation analysis (GCCA)
Using GCCA, data can be mapped to maximize the sum of correlations between multiple modalities (see Figure 6). In this embodiment, the GCCA of Non-Patent Document 2 is adopted. The calculation itself of GCCA is publicly known, and the GCCA described below is an example, and the GCCA used in the present invention is not limited.
If E, I, and J are English, image, and Japanese, respectively, the feature vector
At
Represents a canonical variable.
Here, X _k bar is the average of the feature vectors.
h _k is a projection coefficient.
k ∈ {E, I, J}, and a canonical variable (projection feature vector) can be calculated from each feature vector and the corresponding projection coefficient.

GCCA is a maximization problem
Is the constraint
Equation (1) is derived by solving
The projection coefficient h _k is calculated by maximizing the sum of the correlations of each pair of modalities obtained by solving the following generalized eigenvalue problem.
here,
And
Σ _ij is a covariance matrix of modalities i,j, where i,jε{E,I,J}. Σ _EJ is calculated by E ₃ and J ₃ , Σ _IJ is calculated by I ₂ and J ₂ , Σ _II is calculated by I ₁ and I ₂ , and Σ _JJ is calculated by J ₂ and J ₃ . Σ _EJ is different from Σ _EI and Σ _IJ, and in the case of Zero-Shot learning, the number of training samples may be 0. In that case, since it does not contribute to the maximization problem above, Σ _EJ becomes Filled with 0s.

The canonical axis is standardized as follows.
In addition, a regularization term is added to prevent overlearning. That is,
And α is a regularization parameter.

[C-5] Nearest Neighbor Search in Joint Space Given a query document in the first language, to search for related documents in the second language, which is another language, in the joint space, search for the query document and the candidate document in the joint space. Just calculate the distance. The feature vector (projection feature vector) in the joint space is
And h _k is obtained by GCCA.

For example, if the query document is Japanese and the target document is English, the nearest neighbor in the joint space can be calculated by the following formula.
Here, z ⁱ _E and z ^j _J are projective feature vectors of the target document and the query document, respectively, and d(·) is a distance function. In this example, the distance function is the Eugrid distance.

[D] Experiment
[D-1] Data set used in the experiment
The UIUC Pascal Sentence Dataset (Non-Patent Document 19) has 1000 images, each with 5 English annotations that describe the content. This data set was created for the study of the generation of documents from images, but for use in the image-mediated CLDR according to this embodiment, a Japanese translation corresponding to each English sentence was prepared ( (See FIG. 8). In this experiment, five sentences corresponding to each image are collectively treated as one text data. Therefore, in this setup, each document in the data set of 1000 documents consists of one image, corresponding English text, and Japanese text. Each symbol in the conceptual diagram of FIG. 6 represents "a feature vector extracted from five Japanese sentences", "a feature vector extracted from one image", and "a feature vector extracted from five English sentences". There is.

[D-2] Data was randomly extracted from each data division in the evaluation table 1 so as not to overlap. Experiments were performed by changing the sample sizes of [train-E/I] and [train-I/J]. Specifically, the number of samples was 100, 200, 300, 400. Furthermore, the number of samples of [train-E/J] was gradually increased from 0 to 100, and a Few-Shot learning scenario was created. The size of the test data [test-E/J] was set to 100.
According to this setup, image-mediated CLDR based on GCCA was performed and compared with the results of conventional CLDR based on CCA using only [train-E/J] data.

The performance was evaluated for the first Japanese-to-English retrieval accuracy in the test data. Given 100 test samples, the chance rate is 1%. In each trial, 50 trials were performed while randomly changing the data, and the average score was used. All features were reduced to 100 dimensions by PCA and α was set to 0.01.

As shown in FIG. 9, the experimental results show that the accuracy improves as the amount of text-image data increases in both the Zero-Shot learning scenario and the Few-Shot learning scenario. It is considered that further improvement in accuracy can be expected by increasing the amount of text-image data.

Table 3 summarizes the results of Zero-Shot learning scenarios (Zero-Shot learning accuracy). Image features were extracted by GoogLeNet and text features used bag-of-words (BoW) and TF-IDF.
As shown in Fig. 9, in both GCCA and CCA, the performance is improved by increasing the sample size of [train-E/J], but it can be expected that the sample size is in English text and Japanese text. The performance of CCA exceeds that of GCCA when it becomes large enough to be directly learned between texts. However, when the data amount of [train-E/J] is small, it exceeds the baseline of CCA of GCCA, and thus image-mediated learning is useful even in the Zero-Shot learning scenario.

[D-3] Effect of Image Feature The effect of the image feature performance in this embodiment was verified (FIG. 10, Table 4). As image features, we used three different features extracted using the following CNN.
GoogLeNet (Non-Patent Document 17) pool5/7x7 s1 layer feature quantity,
The feature amount of the fc6 layer of VGG (Non-Patent Document 13),
Features of fc6 layer of CaffeNet (Non-patent document 16, Non-patent document 18)

Furthermore, the Fisher vector (Non-Patent Document 20), which was widely used before the extraction of the image feature using the deep learning, was also tested. For the Fisher vector, the SIFT descriptor (Non-Patent Document 21) was reduced to 64 dimensions by principal component analysis, and a mixed Gaussian distribution using 64 elements was used. Four spatial grids were used for final feature extraction.

Table 4 shows the accuracy of Zero-Shot learning using a plurality of image features. The sample size of [train-E/I] and [train-I/J] is 400. The text features used were bag-of-words (BoW) and TF-IDF.
The order of accuracy when using each image feature is the order of known performance of the image features used, specifically GoogLeNet → VGG net → CaffeNet → FisherVector.
In this order, the recognition accuracy is high (Non-Patent Document 12). The fact that the ranks are similar in the image-mediated CLDR also means that a higher search accuracy can be obtained in the image-mediated CLDR if a better feature amount is used.

Claims (14)

A learning method executed by a computer having processing means,
Prepare a first training data set consisting of a pair of first language documents and images,
Prepare a second training data set consisting of a pair of second language documents and images,
The processing means extracts a first feature vector from the first language document and a second feature vector from the image in the first training data set;
The processing means extracts a third feature vector from the second language document and a second feature vector from the image in the second training data set;
The processing means performs a generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, so that the first feature vector and the third feature vector are mediated by the second feature vector. Do the mapping,
A learning method between different language documents through images. The learning method according to claim 1, wherein the first feature vector and the third feature vector are extracted using Bag of words. The learning method according to claim 1, wherein the second feature vector is extracted using a convolutional neural network. The learning method according to claim 1, wherein the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced. The first training data set includes multimedia data acquired by crawling from a web in a first language,
The second training data set includes multimedia data acquired by crawling from a second language web.
The learning method according to claim 1. Furthermore, prepare a third training data set consisting of a pair of first language document and second language document,
The processing means extracts a first feature vector from the text of the first language and a third feature vector from the text of the second language in the third training data set;
The processing means further uses a first feature vector and a third feature vector extracted from a third training data set in the generalized canonical correlation analysis,
The learning method according to claim 1. A first training data set consisting of a pair of first language documents and images;
A second training data set consisting of a pair of second language documents and images;
First feature vector extraction means for extracting a first feature vector from the first language document,
Second feature vector extraction means for extracting a second feature vector from the image,
Third feature vector extraction means for extracting a third feature vector from the second language document,
Generalized canonical correlation analysis means,
Equipped with
The generalized canonical correlation analysis means performs a generalized canonical correlation analysis using the first feature vector, the second feature vector, and the third feature vector, so that the second feature vector serves as a medium for the first feature vector and the first feature vector. Mapping the third feature vector,
A learning device between different language documents through images. The learning device according to claim 7, wherein the first feature vector extraction means and the third feature vector extraction means acquire Bag of words. 9. The learning device according to claim 7, wherein the second feature vector extraction unit is a convolutional neural network. The learning device includes principal component analysis means,
The learning device according to claim 7, wherein the first feature vector, the second feature vector, and the third feature vector are dimensionally reduced by a dimension reduction unit. The first training data set includes multimedia data acquired by crawling from a web in a first language,
The second training data set includes multimedia data acquired by crawling from a second language web.
The learning device according to claim 7. Furthermore, a third training data set consisting of a pair of first language documents and second language documents is provided,
In the generalized canonical correlation analysis, further, using the first feature vector and the third feature vector extracted from the third training data set,
The learning device according to claim 7. A cross-language document search method executed by a computer having processing means using a learning model obtained by the learning method between different language documents according to claim 1.
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
The processing means extracts a first feature vector from the first language query document,
The processing means projects the extracted first feature vector on a canonical space using a first projection coefficient to obtain a first projected feature vector,
The processing means extracts a third feature vector from the second language target document candidate,
The processing means projects the extracted third feature vector on a canonical space using the second projection coefficient to obtain a third projected feature vector,
The processing means determines a target document using the similarity between the first projected feature vector and the third projected feature vector,
Cross-language document retrieval method. A cross-language document search device using a learning model obtained by the learning method between different language documents according to claim 1.
In the learning model, a first projection coefficient from the first language space to the canonical space and a second projection coefficient from the second language space to the canonical space are defined,
Means for extracting a first feature vector from the first language query document;
A means for projecting the extracted first feature vector on a canonical space using a first projection coefficient to obtain a first projected feature vector;
Means for extracting a third feature vector from the second language target document candidate,
A means for projecting the extracted third feature vector on a canonical space using a second projection coefficient to obtain a third projected feature vector;
Means for determining a target document using the similarity between the first projected feature vector and the third projected feature vector;
A cross-language document retrieval device equipped with.