JP7660849B2 - Learning system, attribute value extraction system, learning method, and program - Google Patents

Description

This disclosure relates to a learning system, an attribute value extraction system, a learning method, and a program.

Conventionally, there are known techniques for extracting attribute values related to various items such as products or content. For example, Non-Patent Documents 1 and 2 describe a technique in which item data related to an item and a query including attributes related to the item are input to a model equivalent to a trained attribute value extraction model, and information equivalent to a named entity such as an attribute value output from the model is extracted.

Qifan Wang, Li Yang, Bhargav Kanagal, Sumit Sanghai, D.Sivakumar, Bin Shu, Zac Yu, and Jon Elsas. 2020. Learning to extract attribute value from product via question answering: A multi-task approach. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 47-55, Online. ACM. Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859, Online. Association for Computational Linguistics.

However, the accuracy of the attribute values output from the models in Non-Patent Document 1 and Non-Patent Document 2 may decrease depending on the query. For example, the technology in Non-Patent Document 1 assumes that an attribute name is input as a query. For example, the technology in Non-Patent Document 2 assumes that a definition and description of the object to be extracted is input as a query. For example, if a simple word is input as a query, the models in Non-Patent Document 1 and Non-Patent Document 2 may not be able to recognize the meaning of the query, and the accuracy of the attribute values output from the models may decrease.

One of the objectives of this disclosure is to improve the accuracy of attribute value extraction models.

The learning system according to the present disclosure includes a model storage unit that stores an attribute value extraction model for extracting a second attribute value from first data including a second attribute value related to a first item for estimation using a first query including at least one first attribute value associated with the first attribute related to the first item, a second query acquisition unit that acquires a second query in which at least a portion of a plurality of third attribute values associated with a second attribute related to a second item for learning is excluded, and a learning unit that performs learning on the attribute value extraction model based on second data including a fourth attribute value related to the second item and the second query.

This disclosure improves accuracy from attribute value extraction models.

FIG. 1 is a diagram illustrating an example of an overall configuration of a learning system. 11A and 11B are diagrams illustrating an example of attributes and attribute values acquired from product data. FIG. 13 is a diagram illustrating an example of an attribute value extraction model using a question answering model. FIG. 4 is a diagram illustrating an example of an attribute value extraction model according to the present embodiment. FIG. 13 is a diagram illustrating an example of a method for making an attribute value extraction model learn about incompleteness of knowledge. A functional block diagram showing an example of functions realized by the learning system. FIG. 2 is a diagram illustrating an example of a training database. FIG. 4 is a diagram illustrating an example of an attribute database. FIG. 2 is a flow chart showing an example of processing executed in the learning system. FIG. 2 is a flow chart showing an example of processing executed in the learning system. 13 is an example of a learning system and an attribute value extraction system according to a modified example.

[1. Overall configuration of the learning system]
An example of an embodiment of a learning system and an attribute value extraction system according to the present disclosure will be described. In this embodiment, an example is given of a learning system that also includes the functions of an attribute value extraction system, but the learning system and the attribute value extraction system may be separate systems. A case in which the learning system and the attribute value extraction system are separate systems will be described in a modified example below.

Figure 1 is a diagram showing an example of the overall configuration of a learning system. For example, the learning system 1 includes a server 10, a learning terminal 20, and an estimation terminal 30. Each of the server 10, the learning terminal 20, and the estimation terminal 30 can be connected to any network N such as the Internet or a LAN.

The server 10 is a server computer. The control unit 11 includes at least one processor. The storage unit 12 includes a volatile memory such as a RAM and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication.

The learning terminal 20 is a computer that learns the attribute value extraction model described below. For example, the learning terminal 20 is a personal computer, a smartphone, or a tablet terminal. The hardware configurations of the control unit 21, the memory unit 22, and the communication unit 23 are similar to those of the control unit 11, the memory unit 12, and the communication unit 13, respectively. The operation unit 24 is an input device such as a touch panel or a mouse. The display unit 25 is a liquid crystal display or an organic EL display.

The estimation terminal 30 is a computer that uses a trained attribute value extraction model described below. For example, the estimation terminal 30 is a personal computer, a smartphone, or a tablet terminal. The hardware configurations of the control unit 31, the memory unit 32, the communication unit 33, the operation unit 34, and the display unit 35 are similar to those of the control unit 11, the memory unit 12, the communication unit 13, the operation unit 24, and the display unit 25, respectively.

The programs stored in the storage units 12, 22, and 32 may be supplied via the network N. Each computer may also include at least one of a reading unit (e.g., a memory card slot) that reads a computer-readable information storage medium and an input/output unit (e.g., a USB port) for inputting and outputting data to and from an external device. For example, a program stored in an information storage medium may be supplied via at least one of the reading unit and the input/output unit.

The learning system 1 may include at least one computer. The computers included in the learning system 1 are not limited to the example in FIG. 1. For example, the estimation terminal 30 may exist outside the learning system 1. The learning system 1 may include only the server 10 and the learning terminal 20. The learning system 1 may include only either the server 10 or the learning terminal 20. The learning system 1 may include either the server 10 or the learning terminal 20, and another computer.

[2. Overview of the learning system]
In this embodiment, the processing executed by the learning system 1 will be described by taking as an example a case where attribute values according to product attributes are acquired from product data related to the product. A product is an object that is the subject of commercial transaction. In this embodiment, a product in an online shopping mall is taken as an example, but the product itself may be any product, for example, a product traded in an internet auction, an online flea market, or a real store.

Product data is information about the details of a product. For example, product data includes letters, numbers, other symbols, images, sounds, videos, or a combination of these. In this embodiment, products in an online shopping mall are used as an example, so any text entered by a store staff member (e.g., the title of the product) corresponds to the product data. For example, the contents of the product data can be viewed from the website or application of the online shopping mall.

Attributes are categories for classifying products. Attributes are sometimes called categories or genres. Attributes may be defined hierarchically. When attributes are defined hierarchically, the higher the attribute, the more abstract the meaning of the attribute, and the lower the attribute, the more specific the meaning of the attribute. Hereinafter, when simply describing an attribute, it means the name of the attribute. Attributes are expressed by letters, numbers, other symbols, or combinations of these. Attributes can be defined from any perspective as long as they can classify products from a specific perspective. For example, attributes are information such as brand, color, material, size, function, pattern, or place of origin.

An attribute value is a value that indicates a specific classification. Multiple attribute values are prepared in advance for an attribute. At least one of the multiple prepared attribute values is assigned to a product. The multiple prepared attribute values are candidates for the attribute values to be assigned to a product. Attribute values are expressed by letters, numbers, other symbols, or combinations of these. For example, when uploading product data for a product, a store staff member specifies an attribute and at least one of the multiple attribute values prepared for that attribute. The specified attribute and attribute value are associated with the product data for that product.

For example, various brand names are prepared as attribute values for the attribute "brand." Products of a certain brand are assigned an attribute value indicating the brand name of the brand of this product from among the multiple attribute values prepared for the attribute "brand." For example, various colors such as black, white, and yellow are prepared in advance as attribute values for the attribute "color." Products that look black are assigned an attribute value indicating "black" from among the multiple attribute values prepared for the attribute "color." The same is true for other attributes; any attribute value can be prepared in advance and assigned to products.

Figure 2 is a diagram showing an example of attributes and attribute values obtained from product data. For example, when the learning terminal 20 accesses a website of an online shopping mall, a product page P including product data is displayed on the display unit 25. For example, product data such as the store name, product title, product image, and detailed product description are displayed on the product page P. If the store staff specifies attributes and attribute values when uploading the product data, the specified attributes and attribute values are also displayed as product data on the product page P. The product attributes and attribute values are used as indexes during searches.

If appropriate attribute values according to the attributes of a product can be extracted from product data, various effects can be achieved. For example, the burden on store staff can be reduced because they no longer need to specify attribute values. For example, even if a store staff member mistakenly specifies an inappropriate attribute value, it can be replaced with an appropriate attribute value obtained from the product data. For example, new attribute values can be extracted from product data even if they are not currently available as attribute values. For example, it can also be used for marketing by gaining a deeper understanding of customer preferences or the marketplace.

The learning system 1 extracts attribute values from product data using an attribute value extraction model for extracting attribute values from product data. The attribute value extraction model is a model that uses machine learning. Various methods can be used for the machine learning itself. For example, any of the machine learning methods, such as supervised learning, semi-supervised learning, and unsupervised learning, can be used for the attribute value extraction model.

In this embodiment, an attribute value extraction model that uses a question answering model (QA model) based on a model called BERT is given as an example, but the attribute value extraction model may be any model that extracts attribute values using an entity, such as product data, and a query as input, and there is no restriction on the type. For example, the attribute value extraction model may be a so-called Transformer-based model other than BERT, or a neural network model that was used before the advent of Transformers.

Figure 3 is a diagram showing an example of an attribute value extraction model that uses a question-answering model. The question-answering model M1 and attribute value extraction model M2 in Figure 3 are conventional models, but in order to understand the basic mechanism, the conventional models will be explained. For details on the question-answering model M1 and the attribute value extraction model M2, please refer to Non-Patent Document 1 and Non-Patent Document 2 listed as prior art documents, or the references described therein. Note that in these documents, the models equivalent to the question-answering model M1 and the attribute value extraction model M2 may be called by other names.

For example, when a news article and a query are input, the question-answering model M1 outputs a response corresponding to the query. The question-answering model M1 has learned the relationship between the news article and the query, and the response corresponding to the query. The query input to the question-answering model M1 is a question about the content of the news article. The response output from the question-answering model M1 is an answer to the question, which is the query. The question-answering model M1 estimates the part of the news article that is appropriate as the answer, and outputs that part.

In the example of FIG. 3, a news article N10 about the lifting of the state of emergency in Japan and a query Q11 asking when the state of emergency will be lifted are input to the question-answering model M1. The question-answering model M1 infers that the part of the news article N10 that is "September 30th" is appropriate as the date when the state of emergency will be lifted, and outputs a response R12 that can identify this part. Such a question-answering model M1 can also be used to extract attribute values from product data.

For example, the attribute value extraction model M2 is a model created by reusing the question answering model M1. When product data and a query are input, the attribute value extraction model M2 outputs an attribute value as a response to the query. The attribute value extraction model M2 has learned the relationship between the product data and the query, and the attribute values contained in the product data. The query input to the attribute value extraction model M2 is the attribute of the product indicated by the product data. The query can be said to be a question that asks about a part of the product data that contains an attribute value corresponding to the attribute. The response output from the attribute value extraction model M2 is an attribute value corresponding to the attribute that is the query. The attribute value extraction model M2 estimates an appropriate part as an attribute value from the product data, and outputs that part.

In the example of FIG. 3, product data D20 relating to the characteristics of a bag and a query Q21 including the attribute "brand" are input to the attribute value extraction model M2. In FIG. 3, an image of the product is also input to the attribute value extraction model M2, but in reality, only the text portion of the product data D20 is input to the attribute value extraction model M2. The image portion of the product data D20 may be converted into a combination of numerical values such as a feature vector via a feature extractor based on a machine learning model such as a neural network, and then input to the attribute value extraction model M2.

For example, before the product data D20 is uploaded to an online shopping mall, the product data D20 and a query Q21 are input to the attribute value extraction model M2. The attribute value extraction model M2 estimates that the "BBB bag" portion of the product data is appropriate as the attribute value V22 (here, the brand name) corresponding to the attribute "brand" included in the query Q21, and outputs the inference. For example, other attributes such as the attribute "color", the attribute "material", the attribute "size", or the attribute "function" can also be input as a query to the attribute value extraction model M2.

For example, product attributes are often simple words, so the query input to the attribute value extraction model M2 may be shorter and less specific than the query input to the question-answering model M1. For this reason, the attribute value extraction model M2 may not be able to recognize the meaning of the query. If the attribute value extraction model M2 cannot recognize the meaning of the query, the accuracy of the output from the attribute value extraction model M2 will also decrease. In particular, in the case of an online shopping mall, when training data for the attribute value extraction model M2 is created based on product data, there tends to be more attributes for popular products and less training data for other products. Such sparseness of the training data is one of the factors that reduces the accuracy of the output from the attribute value extraction model M2.

Taking the attribute "function" as an example, a product such as an air conditioner may have functions such as "timer" and "energy saving." A product such as a bag may have attributes such as "waterproof," "mildew-proof," or "stain-resistant." For this reason, the same attribute "function" may have a variety of meanings. The attribute value extraction model M2 may not be able to recognize the meaning of a short, abstract query such as the attribute "function," and may not be able to output an appropriate attribute value. This is true for other attributes, and the accuracy of the output from the attribute value extraction model M2 may decrease depending on the quality of the query.

Therefore, in this embodiment, the query is expanded by using not only the product attributes but also attribute values that are pre-associated with the product attributes as the query. This makes the meaning of the query more specific, making it easier for the attribute value extraction model M2 to recognize the meaning of the query, which is believed to improve the quality of the query. If the query quality is improved, it is believed that the problem of sparseness of the training data mentioned above can also be solved.

Figure 4 is a diagram showing an example of an attribute value extraction model of this embodiment. The attribute value extraction model M3 is not a conventional technology, but a new technology. The attribute value extraction model M3 is similar to the attribute value extraction model M2 in that it is a model that outputs a response according to the query when product data and a query are input, but differs from the attribute value extraction model M2 in the content of the query. In the example of Figure 4, the product data D30 is the same as the product data D20, but the query Q31 includes not only attributes but also attribute values, and is therefore more extended than the query Q21.

For example, suppose that the product indicated by the product data D30 is a bag, and the product attribute input as query Q31 is "function". As mentioned above, the attribute "function" is a short and abstract attribute, and has various meanings related to products other than bags. For this reason, to enable the attribute value extraction model M3 to recognize that the meaning is a bag-related attribute, query Q31 is input that includes the attribute "function" and the attribute value "mold prevention" that is prepared in advance for the bag-related attribute "function". Note that, although omitted in FIG. 4, in this embodiment, all attribute values associated with the attribute "function" are included in query Q31, not just the attribute value "mold prevention".

The product in FIG. 4 is not a product with an anti-mold function, but the attribute value "anti-mold" included in query Q31 enables the attribute value extraction model M3 to recognize the meaning of the attribute "function". For example, the attribute value extraction model M3 can recognize that the attribute "function" included in query Q31 is not a function such as an air conditioner as described above, but a function related to a bag. Therefore, the attribute value extraction model M3 can output an attribute value V32 from the product data D30, such as "waterproof", which is a function name that has an expression similar in meaning or notation to the attribute value "anti-mold" included in query Q31.

In addition to the example in FIG. 4, when another expression that is semantically or notationally close to the attribute value "waterproof" to be extracted is given as a query, such as the attribute value "stain-resistant" (function to prevent stains), the attribute value extraction model M3 can output the attribute value "waterproof" from the product data. This is also true when other queries including short and abstract attributes, such as the attribute "kind" or "type", are input to the attribute value extraction model M3, not limited to the attribute "function".

As described above, in this embodiment, the query is expanded by using attribute values associated with the product attributes. This also applies to other attributes such as the attribute "brand", attribute "color", attribute "material", attribute "size", or attribute "function". For example, if it is possible to cover all the attributes and attribute values of products traded in online shopping malls, it is believed that it will be possible to handle any product.

However, since there is a wide variety of products in online shopping malls, it is not realistic to completely cover all attributes and attribute values. For example, among the products traded in online shopping malls, there are attributes and attribute values that the online shopping mall administrator does not fully understand. For this reason, in actual operation, it is necessary to use incomplete knowledge that does not completely cover all attributes and attribute values.

For example, if the attribute value extraction model M3 has not learned that the knowledge used in actual operations is incomplete, the attribute value extraction model M3 may estimate an attribute value assuming that the input query is completely correct. In this case, for example, the attribute value extraction model M3 may estimate an attribute value based on a simple string match with the attribute value contained in the query. In this case, it is considered that unknown attributes and attribute values cannot be handled. Therefore, in this embodiment, the attribute value extraction model M3 is intentionally made to learn the incompleteness of knowledge, thereby enabling it to handle actual operations.

Figure 5 is a diagram showing an example of a method for having the attribute value extraction model M3 learn about the incompleteness of knowledge. For example, the training database DB1 stores a large number of combinations of product data (product titles in Figure 5) for products to be learned by the attribute value extraction model M3, and the attributes and attribute values of the products. In the example of Figure 5, the product data for a battery-related product contains a character string such as "ABC Battery 12V 14AH SLA Rechargeable." "ABC Battery" is the attribute value of the attribute "Brand." "14AH" is the attribute value of the attribute "Nominal Capacity."

For example, the attribute database DB2 stores many pairs of attributes and attribute values that exist in the training database DB1. If there are 100 attribute values, such as "1 ah" to "100 ah", for the attribute "nominal capacity" stored in the training database DB1, pairs of the attribute "nominal capacity" and these 100 attribute values are stored in the attribute database DB2. It is assumed that the attribute database DB2 also stores many pairs of other attributes.

If attribute value extraction model M3 is trained using query Q40 that includes all 100 attribute values of the attribute "nominal capacity", there is a possibility that attribute value extraction model M3 will recognize that the knowledge input to it is complete. In this case, for example, attribute value extraction model M3 may focus only on attribute values and not on attributes, or may be unable to extract unknown attribute values. For example, if an unknown attribute is input as a query, or an attribute with few available attribute values is input as a query, the accuracy of attribute value extraction model M3 may decrease.

Therefore, in this embodiment, two techniques are used to make the attribute value extraction model M3 learn about the incompleteness of knowledge. In the first technique, rather than making the attribute value extraction model M3 learn all of the attribute values stored in the attribute database DB2, some are deliberately excluded and then made to learn by the attribute value extraction model M3. Hereinafter, the first technique is referred to as the knowledge dropout technique. In the knowledge dropout technique, the attribute value extraction model M3 can learn about the incompleteness of knowledge by intentionally reducing the number of attribute values that are actually usable. In other words, the knowledge dropout technique is a technique for intentionally reducing the amount of knowledge that is made to learn by the attribute value extraction model M3.

In the second technique, the attribute value extraction model M3 is trained without using any of the attribute values stored in the attribute database DB2. However, if no attribute values are used, there is a possibility that the accuracy will decrease as with the conventional attribute value extraction model M2 described in FIG. 3. Therefore, in this embodiment, for each product whose product data is stored in the training database DB1, two types of learning are performed: learning using the knowledge dropout technique, and learning in which all attribute values are removed and not used.

In the second technique, a special token that indicates the availability of an attribute value is included in the query. A token is a unit of a word. In the example of Figure 5, a token is a unit of a word separated by a space. A token may be a unit of a word separated by a special symbol instead of a space. A token may be a word that has some meaning, but looking at a particular token alone does not necessarily mean that it is a word whose meaning can be understood by a human.

Hereinafter, the special token used in the second method is referred to as a knowledge token. Furthermore, the second method is referred to as the knowledge token method. In the example of FIG. 5, the knowledge token is expressed as a character string such as [Seen] and "Unseen". When the knowledge token is [Seen], it means that the attribute value is available. When the knowledge token is [Unseen], it means that the attribute value is not available.

In this embodiment, it is assumed that an attribute stored in the attribute database DB2 always has a paired attribute value. For this reason, in the learning stage, a pseudo knowledge token of [Unseen] is associated with an attribute whose attribute value is actually available. In the example of FIG. 5, as in input I42 for the attribute value extraction model M3 during learning, the attribute "nominal capacity" actually has 100 attribute values, but the attribute value is pseudo-assigned to an [Unseen] knowledge token. In addition, as in input I41, a query containing a knowledge token of [Seen] using the knowledge dropout technique (a query containing an attribute value that was not excluded) is also learned by the attribute value extraction model M2.

As described above, in this embodiment, the knowledge dropout method and the knowledge token method are used to have the attribute value extraction model M3 learn about the incompleteness of knowledge. By having the attribute value extraction model M3 learn about the incompleteness of knowledge, attribute value extraction suitable for the actual operation environment can be performed, thereby improving the accuracy of the attribute value extraction model M3. The learning system 1 will be described in detail below.

[3. Functions realized by the learning system]
6 is a functional block diagram showing an example of functions realized by the learning system 1. In this embodiment, a case will be described in which a learning function related to learning the attribute value extraction model M3 is realized by the learning terminal 20. A case will be described in which an estimation function related to estimation using the learned attribute value extraction model M3 is realized by the estimation terminal 30. Hereinafter, in order to distinguish between the products, product data, attributes, attribute values, and queries used during estimation and learning, the products, product data, attributes, attribute values, and queries are given names as follows:

[Terms used in estimation]
First product: product to be processed by the trained attribute value extraction model M3 First data: product data of the first product First attribute: attribute of the first product First attribute value: attribute value associated with the first attribute First query: query used during estimation Second attribute value: attribute value extracted from the first data

[Terms used during study]
Second product: product used in learning the attribute value extraction model M3 Second data: product data of the second product Second attribute: attribute of the second product Third attribute value: attribute value associated with the second attribute Second query: query used during learning Fourth attribute value: attribute value included in the second data and serving as the correct answer during learning

[3-1. Functions realized by the server]
The model storage unit 100 is realized by the storage unit 12. The model storage unit 100 stores a trained attribute value extraction model M3. The trained attribute value extraction model M3 is an attribute value extraction model M3 for which training by a learning unit 204 described later has been completed. In this embodiment, a case will be described in which the estimation terminal 30 downloads and uses the trained attribute value extraction model M3 stored in the model storage unit 100, but the estimation terminal 30 may use the trained attribute value extraction model M3 online without downloading the trained attribute value extraction model M3. The model storage unit 100 may store a training database DB1 and an attribute database DB2 described later.

[3-2. Functions realized on the learning device]
The model storage unit 200 is realized by the storage unit 22. The third attribute value acquisition unit 201, the probability determination unit 202, the second query acquisition unit 203, and the learning unit 204 are mainly realized by the control unit 21. These functions are examples of learning functions.

[Model storage unit]
The model storage unit 200 stores data necessary for learning the attribute value extraction model M3. For example, the model storage unit 200 stores an attribute value extraction model M3 for extracting a second attribute value from first data including a second attribute value related to a first product using a first query including at least one first attribute value associated with a first attribute related to a first product for estimation. The first product is an example of a first item. Therefore, the description "first product" can be read as "first item."

An item is an object to which attributes and attribute values are assigned. An item may be a tangible object such as a product, or an intangible object such as a service or data. An item may be any object and is not limited to a product. For example, an item may be content related to accommodations, content related to restaurants, e-books, videos, music, websites, or other content. For example, an item may be an introduction to a service such as a financial service or a communication service, a post on a social networking site, a message such as an e-mail, or other document.

The first item is an item among the above items that is to be estimated by the trained attribute value extraction model M3. The first item is an item that is to be extracted as a second attribute value. The first item is an item for which a second attribute value has not yet been extracted from the first data. The first item may coincidentally be the same as the second item described below, but in principle, it is different from the second item.

The first data is data including some content related to the first item. The first data is data related to the details of the first item. The first data may be data according to the type of the first item, and is not limited to the title of the first product. For example, if the first item is an accommodation facility, the title or description of the accommodation facility or room may correspond to the first data. For example, if the item is an electronic book, the actual data portion of the electronic book may correspond to the first data. In this embodiment, the first data includes a first character string related to a description of the first product. The attribute value extraction model M3 is a model related to natural language processing for extracting a second attribute value from the first character string.

For example, the model storage unit 200 stores the attribute value extraction model M3 before learning. The attribute value extraction model M3 before learning is an attribute value extraction model M3 with parameters of initial values. After the learning unit 204 starts learning, the model storage unit 200 stores the attribute value extraction model M3 in the middle of learning. After learning is completed, the model storage unit 200 stores the learned attribute value extraction model M3. In addition to the attribute value extraction model M3, the model storage unit 200 also stores a training database DB1 and an attribute database DB2.

FIG. 7 is a diagram showing an example of the training database DB1. The training database DB1 is a database in which data that can be training data for the attribute value extraction model M3 is stored. The training data is data related to the relationship between the second data, the second query, and the fourth attribute value. For example, the training database DB1 is associated with the second data, the second attribute included in the second query, and the fourth attribute. The training data includes a pair of an input portion that is input to the attribute value extraction model M3 during learning, and an output portion that is to be output from the attribute value extraction model M3. In FIG. 7, only the second attribute of the second query is shown. The third attribute value included in the second query is filled in from the attribute database DB2.

The input portion of the training data has the same format as the data input to the trained attribute value extraction model M3. Therefore, the format of the first data and the first query input at the time of estimation is the same as the format of the second data and the second query input at the time of learning. In this embodiment, the first data and the first query, and the second data and the second query are both in character format, but they may be in any format, such as numbers, other symbols, or a combination of these and characters. For example, if character features are input to the attribute value extraction model M3 instead of characters, the input portion of the training data may include the features of the second data and the second query.

The output portion of the training data has the same format as the data output from the trained attribute value extraction model M3. Therefore, the format of the output during estimation is the same as the format of the output during learning. In this embodiment, a case where partial identification information described below is output will be described, but the second attribute value and the fourth attribute value themselves may be output. In this embodiment, a case where the second attribute value and the fourth attribute value are both in character format will be described, but these may be in any format, such as numbers, other symbols, or a combination of these and characters. For example, if the attribute value extraction model M3 does not output a character string indicating the attribute value, but rather outputs an ID or number that can identify the attribute value, the output portion of the training data may include an ID or number that can identify the fourth attribute value.

In this embodiment, a case will be described in which the first and second products correspond to products actually sold at an online shopping mall. That is, product data actually uploaded to an online shopping mall corresponds to the first and second data. For example, the first and second data are the titles, descriptions, or combinations of products sold at an online shopping mall. The first data includes characters entered by a user. Note that the training data may be created manually by an administrator of the online shopping mall, rather than being created based on products actually sold at an online shopping mall.

The second query is a question for inquiring about the attribute value of the second product. For example, the second query may include a second attribute and a third attribute value associated with the second attribute. The second attribute is an attribute associated with the second product. For example, the second attribute includes characters indicating an attribute specified by a store staff member. The second attribute may be specified by an online shopping mall manager, or a tool for extracting the second attribute may be used. In this embodiment, the third attribute value included in the second query is determined by a knowledge dropout method, so that only the second attribute to be included in the second query is shown in the training database DB1. The third attribute value to be included in the second query is obtained from the attribute database DB2 described below.

The fourth attribute value is the correct attribute value for the second product. The fourth attribute value may be any one of a plurality of third attribute values associated with the second attribute, or may be an attribute value that does not exist among the plurality of third attribute values. For example, the fourth attribute value includes characters indicating an attribute value specified by a store staff member. The fourth attribute value may not be specified by a store staff member, but may be specified by an online shopping mall manager, or may be extracted by a previously created attribute value extraction model M3. However, the fourth attribute value extracted from the previously created attribute value extraction model M3 may include an incorrect fourth attribute value, and therefore may be less accurate than an attribute value extraction model M3 trained based on manually created training data.

Figure 8 is a diagram showing an example of the attribute database DB2. The attribute database DB2 is a database in which pre-prepared attribute values are stored for each of a plurality of attributes. The attribute database DB2 can also be a dictionary of attributes and attribute values. For example, the attribute database DB2 stores a large number of pairs of attributes and attribute values. An attribute stored in the attribute database DB2 can be a first attribute or a second attribute. An attribute value stored in the attribute database DB2 can be a first attribute value or a third attribute value. The attribute database DB2 may also store frequencies, which will be described later.

In this embodiment, the attribute database DB2 is created based on the training database DB1. For example, the learning terminal 20 tally up combinations of the second attribute included in the input portion of the training data and the fourth attribute value that is the output portion corresponding to the input portion. Based on the tallying results, the learning terminal 20 stores pairs of the second attribute and the fourth attribute value in the attribute database DB2. The learning terminal 20 may store only the fourth attribute values that have a frequency equal to or greater than a threshold value for a certain second attribute in the attribute database DB2.

The attribute database DB2 can be created by any method and does not have to be created based on the training database DB1. For example, the learning terminal 20 may create the attribute database DB2 based on a product database related to products sold at an online shopping mall. For example, an administrator of the online shopping mall may manually create the attribute database DB2. For example, a list of attributes and attribute values that can be assigned to products sold at an online shopping mall may be used as the attribute database DB2.

[Third attribute value acquisition unit]
The third attribute value acquisition unit acquires at least one third attribute value included in the second query. For example, the third attribute value acquisition unit 201 acquires a plurality of third attribute values associated with the third attribute from an attribute database DB2 in which a plurality of third attribute values are stored. The attribute database DB2 is an example of a second database. Therefore, the description of the attribute database DB2 can be replaced with the second database. The second database may be called by another name as long as it is a database in which various third attribute values are stored.

In this embodiment, the third attribute value acquisition unit 201 acquires all the third attribute values stored in the attribute database DB2 in which a plurality of third attribute values are stored. For example, the third attribute value acquisition unit 201 refers to the attribute database DB2 and acquires all the third attribute values associated with the second attribute of a certain second product. Taking the battery in FIG. 5 as an example of the second product, the second attribute "nominal capacity" has 100 third attribute values such as "1 ah" to "100 ah" stored in the attribute database DB2, so the third attribute value acquisition unit 201 acquires all of these 100 third attribute values. The third attribute value acquisition unit 201 similarly acquires all the third attribute values for the other second attributes from the attribute database DB2.

The third attribute value acquisition unit 201 may acquire only a portion of the third attribute values stored in the attribute database DB2. For example, the third attribute value acquisition unit 201 may randomly acquire a predetermined number of third attribute values from among all the third attribute values. For example, the third attribute value acquisition unit 201 may acquire a predetermined number of third attribute values from among all the third attribute values in descending order of frequency in the training database DB1. For example, the third attribute value acquisition unit 201 may acquire third attribute values whose frequency in the training database DB1 is equal to or greater than a threshold value from among all the third attribute values.

For example, the third attribute value acquisition unit 201 may acquire the third attribute value from the training database DB1 or another database, rather than acquiring the third attribute value from the attribute database DB2. The third attribute value may be specified by an administrator of an online shopping mall. For example, when the administrator specifies the third attribute value one by one during learning of the attribute value extraction model M3, the third attribute value acquisition unit 201 may acquire the third attribute value specified one by one. When only one third attribute value is available, the third attribute value acquisition unit 201 may acquire only one third attribute value.

[Probability determination unit]
The probability determination unit 202 determines the probability that the third attribute value is excluded. This probability is the probability that the third attribute value included in the tentative second query will not be included in the final second query in the knowledge dropout method. This probability may be a common probability for all the third attribute values or may be determined randomly, but in this embodiment, it is assumed to be a probability according to the frequency of the third attribute value.

The frequency is the frequency with which the third attribute value is used in actual products. In this embodiment, the frequency of a certain third attribute value is the frequency with which the third attribute value appears in the training database DB1. The frequency can also be referred to as the number of occurrences in the training database DB1. Although omitted in FIG. 8, the frequency of the third attribute value is also stored in the attribute database DB2. For example, the learning terminal 20 tallies the frequency with which the third attribute value appears in the training database DB1, and stores it in the attribute database DB2.

For example, the probability determination unit 202 determines the probability of each third attribute value based on the frequency of the third attribute value in a training database DB1 in which multiple second data are stored. The training database DB1 is an example of a first database. Therefore, the reference to training database DB1 can be read as the first database. The first database may be called by another name as long as it is a database in which the third attribute values of various second products are stored. For example, a product database of an online shopping mall may correspond to the first database. In other words, the frequency in another database may be used.

For example, the probability determination unit 202 determines the probability of the third attribute value such that the higher the frequency of the third attribute value, the lower the probability of the third attribute value. Conversely, the probability determination unit 202 determines the probability of the third attribute value such that the lower the frequency of the third attribute value, the higher the probability of the third attribute value. In this way, it becomes less likely that a frequently used third attribute value will be excluded, making it easier to have the attribute value extraction model M3 learn frequently used third attribute values.

The data indicating the relationship between frequency and probability is stored in advance in the model storage unit 200. This data may be in any format, for example, in the form of a mathematical formula or a table. This data may be a model using machine learning, or may be part of a program. The probability determination unit 202 determines the probability of the third attribute value so that the probability is associated with the frequency of the third attribute value.

For example, the probability determination unit 202 may determine the probability of each third attribute value based on a predetermined exclusion rate and the frequency of the third attribute value. The exclusion rate is a default probability. The exclusion rate may differ depending on the third attribute value, but in this embodiment, the exclusion rate is the same for all third attribute values. The exclusion rate may be specified by the administrator of the online shopping mall, or may be dynamically determined based on an external database.

For example, if the exclusion rate is r and the frequency of a certain third attribute value is nv, the probability determination unit 202 determines the probability of this third attribute value to be r ^nv . Note that the calculation method of the probability may be other calculation methods and is not limited to the example of this embodiment. For example, the probability determination unit 202 may determine the value obtained by dividing the exclusion rate r by the frequency nv of a certain third attribute value as the probability of this third attribute value. For example, the probability determination unit 202 may determine the value obtained by subtracting the frequency nv of a certain third attribute value from the exclusion rate r as the probability of this third attribute value. For example, the probability determination unit 202 may determine the probability without using the exclusion rate r.

[Second query acquisition unit]
The second query acquisition unit 203 acquires a second query. In this embodiment, in order to express incompleteness of knowledge, third attribute values that are actually available for learning are intentionally excluded, so the second query acquisition unit 203 acquires a second query in which at least some of the third attribute values associated with the second attribute related to the second product for learning are excluded.

Excluding the third attribute value means not including the third attribute value in the second query. In other words, intentionally not including a third attribute value that can actually be used for learning in the second query corresponds to excluding the third attribute value. Although the third attribute value exists in the attribute database DB2, not including it in the second query corresponds to excluding the third attribute value. Exclusion of the third attribute value can also be referred to as disabling or deleting the third attribute value.

In this embodiment, the second query includes the second attribute, and at least some of the multiple third attribute values are excluded. That is, the second query includes both the second attribute and the third attribute value. The second query may not include the second attribute, but may include the third attribute values that are not excluded from the multiple third attribute values. If a certain number of third attribute values are available and the meaning of the second query can be understood to some extent from the third attribute values alone, the second query does not need to include the second attribute.

If the query Q40 in FIG. 5 corresponds to the second query, it includes a knowledge token, an attribute token corresponding to the second attribute, an attribute value token corresponding to the third attribute value, and an SEP token. The SEP token is a special token that indicates a separation. For example, there is an SEP token that indicates a separation between the second data and the second attribute (in FIG. 5, an SEP token between the title token and the knowledge token), an SEP token that indicates a separation between the second attribute and the third attribute value (in FIG. 5, an SEP token between the attribute token and the attribute value token), and an SEP token that indicates a separation between the third attribute values (in FIG. 5, an SEP token between the attribute value tokens). Note that the CLS token in FIG. 5 is a special token that is placed at the beginning. The hidden state corresponds to a memory for holding calculations inside the attribute value extraction model M3. The hidden state may hold information such as embedded expressions.

In the knowledge dropout method, the second query acquisition unit 203 excludes only a portion of the multiple third attribute values. The second query acquisition unit 203 acquires a second query including the remaining third attribute values that have not been excluded from the multiple third attribute values. For example, if there are k third attribute values (k is an integer of 2 or more) associated with a certain second attribute, then excluding 1 or more but less than k third attribute values from the k third attribute values corresponds to excluding only a portion of the multiple third attribute values. The number or proportion of the excluded third attribute values may be determined in advance or may change dynamically.

In this embodiment, the second query acquisition unit 203 determines, for each third attribute value, whether or not to exclude the third attribute value based on the probability associated with the third attribute value. The second query acquisition unit 203 excludes only a portion of the multiple third attribute values that are determined to be excluded based on the above probability. For example, the second query acquisition unit 203 determines, for each third attribute value, whether or not to exclude the third attribute value based on a probability associated with the third attribute value according to the frequency of the third attribute value.

In the example of the probability determination unit 202 described above, the probability of the third attribute value is r ^nv . The second query acquisition unit 203 determines, for each third attribute value, whether or not to exclude the third attribute value based on the probability r ^nv of the third attribute value. This determination can use various lottery algorithms. The second query acquisition unit 203 excludes the third attribute value determined to be excluded from the second query so that the third attribute value is not included in the second query. The second query acquisition unit 203 does not exclude the third attribute value from the second query so that the third attribute value not determined to be excluded is included in the second query.

In this embodiment, since all third attribute values are acquired for the time being, the second query acquisition unit 203 excludes only a portion of all third attribute values. The second query acquisition unit 203 acquires a second query including the remaining third attribute values that have not been excluded from all third attribute values. For example, the second query acquisition unit 203 acquires a tentative second query including all third attribute values for the time being. The second query acquisition unit 203 excludes the third attribute values that it has determined to be excluded from the tentative second query, and acquires a final second query.

For example, the second query acquisition unit 203 acquires a second query including a plurality of third attribute values. This second query is a tentative second query. The second query acquisition unit 203 may exclude some of the plurality of third attribute values included in the second query by replacing the some of the third attribute values with other values. The other values may be any predetermined value, for example, a predetermined character string. The other values may be called padding. The other values may be a character string that has no particular meaning. Here, the other values are assumed to be a character string such as [PAD].

For example, the second query after the excluded third attribute value is replaced with another value is the final second query. In the example of input I41 in FIG. 5, third attribute values such as "1ah", "4ah", and "5ah" are excluded. In the example of FIG. 5, a case is shown in which the excluded third attribute value is deleted from the tentative third query without being replaced with another value, but, for example, another value to replace the excluded "1ah" may be inserted before "2ah" in input I41. Similarly, another value to replace the excluded "4ah" and "5ah" may be inserted between "3ah" and "6ah".

The second query described above is the second query used in the knowledge dropout method. In the knowledge token method, the second query acquisition unit 203 acquires a first type of second query that includes a knowledge token of [Seen] indicating that the third attribute value is available and at least a part of the multiple third attribute values. The knowledge token of [Seen] is an example of first identification information. Therefore, the parts that describe the knowledge token of [Seen] can be read as first identification information.

The first identification information only needs to indicate that the third attribute value is available, and any information other than the knowledge token of [Seen] can be used. For example, it may be a character string other than [Seen], or may be a numerical value or other symbol. However, the token used as the first identification information is not included in the title, attribute name, and attribute value. When making a prediction using the trained attribute value extraction model M3, it is not known what title, attribute name, and attribute value will be input, so the first identification information is enclosed in [ ], etc., so that it can be known that it is the first identification information. The identifier indicating that it is the first identification information is not limited to [ ] and may be another symbol. The second query of the first type is a second query including the first identification information. The second query of the first type is a second query including the third attribute value. In the example of FIG. 5, the part of the input I41 after [Seen] corresponds to the second query of the first type.

In addition, in the knowledge token method, the second query acquisition unit 203 acquires a second query of a second type that includes second identification information indicating that the third attribute value is not available and in which all of the multiple third attribute values are excluded. The knowledge token of [Unseen] is an example of second identification information. Therefore, the description of the knowledge token of [Unseen] can be read as the second identification information.

The second identification information only needs to indicate that the third attribute value is not available, and any information other than the knowledge token of [Unseen] can be used. For example, it may be a character string other than [Unseen], or it may be a numerical value or other symbol. As with the first identification information, the second identification information is also made recognizable as the second identification information by enclosing it in [ ], for example. The second query of the second type is a second query that includes the second identification information. In the example of FIG. 5, the portion of the input I42 following [Unseen] corresponds to the second query of the second type. In the example of FIG. 5, the second query of the second type includes only the knowledge token and the second attribute.

The knowledge tokens [Seen] and [Unseen] may be inserted at any position, and the positions at which these knowledge tokens are inserted are not limited to the example in FIG. 5. For example, these knowledge tokens may be inserted after the second attribute or after the third attribute value. The knowledge tokens may be inserted at predetermined positions.

[Learning Department]
The learning unit 204 learns the attribute value extraction model M3 based on the second data including the fourth attribute value related to the second product and the second query. The second product is an example of a second item. Therefore, the section describing the second product can be read as the second item. The second item is an item that is used in learning the attribute value extraction model M3 among the items described in the section regarding the first item. The second item is an item for which the correct fourth attribute value has been identified. The second item is an item that is used as training data for the attribute value extraction model M3.

The second data is data including some content related to the second item. The second data is data related to details of the second item. The second data may be data according to the type of the second item, and is not limited to the title of the second product. For example, if the second item is an accommodation facility, the title or description of the accommodation facility or room may correspond to the second data. For example, if the item is an e-book, the actual data portion of the e-book may correspond to the second data. In this embodiment, the second data includes a second character string related to the description of the second product.

For example, the learning unit 204 learns the attribute value extraction model M3 so that when an input portion of training data is input, an output portion of training data is output. Learning is to adjust the parameters of the attribute value extraction model M3. The learning itself can use various algorithms according to the attribute value extraction model M3, and may take the form of fine tuning performed on a pre-trained Transformer-based model such as BERT, and algorithms such as backpropagation or gradient descent may be used when determining optimal parameters for other models.

In this embodiment, the learning unit 204 performs learning so that when the second data and the second query are input to the attribute value extraction model M3, the attribute value extraction model M3 outputs a start point Pb and an end point Pe that can identify the part of the fourth attribute value in the second data. The start point Pb and the end point Pe are examples of partial identification information. Therefore, the part that describes the start point Pb and the end point Pe can be read as partial identification information. In the example of FIG. 5, the part of the fourth attribute value in the inputs I41 and I42 is the part "100Ah". For example, when a character is used as a unit of tokens, learning is performed so that the part "1" becomes the start point Pb and the part "h" becomes the end point Pe. For example, when a word is used as a unit of tokens and "100Ah" is divided into "100" and "Ah", the part "100" becomes the start point Pb and the part "Ah" becomes the end point Pe. For example, if "100Ah" is divided into "100," "##A," and "##h," with subwords as the unit of tokens, the "100" part will be the starting point Pb, and the "##h" part will be the end point Pe. In this case, the "##" attached to a subword means that in the original string, the subword in question was connected to the previous subword. As shown in Figure 5, learning is performed so that slight variations in spelling, such as "AH," "ah," and "Ah," can be absorbed.

The partial identification information may be any information capable of identifying a portion of the second data that includes the fourth attribute value. For example, the partial identification information may be a pair of information indicating the number of the character immediately before the fourth attribute value and information indicating the number of the character immediately after the fourth attribute value. For another example, the partial identification information may include information capable of identifying not only the start point Pb and end point Pe of the fourth attribute value, but also characters in between. The attribute value extraction model M3 may output the fourth attribute value itself, rather than the start point Pb and end point Pe. In this case, the learning unit 204 learns the attribute value extraction model M3 so that, when the second data and the second query are input, the fourth attribute value included in the second data is output.

In the knowledge token method, the learning unit 204 performs a first learning on the attribute value extraction model M3 based on the second data and the second query of the first type. In this embodiment, the first learning is assumed to be the same as the learning in the knowledge dropout method. The learning unit 204 performs a second learning on the attribute value extraction model M3 based on the second data and the second query of the second type. The learning unit 204 performs two learnings for a certain second product: a first learning using the second query of the first type, and a second learning using the second query of the second type.

In this embodiment, the learning unit 204 performs the first learning and the second learning using multi-domain learning in which the availability of the third attribute value is regarded as a domain. In the multi-domain learning of this embodiment, the presence or absence of attribute value knowledge is regarded as a domain, and inputs including [Seen] or [Unseen] are artificially prepared as data for each domain, and the attribute value extraction model M3 for the two domains is learned. For example, the learning unit 204 performs the above multi-domain learning inspired by the multi-domain learning method of "Denny Britz, Quoc Le, and Reid Pryzant. 2017. Effective domain mixing for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 118-126, Copenhagen, Denmark. Association for Computational Linguistics.".

When a query of the same type as the first type is input by the first learning, the first domain corresponding to the first type is used in the estimation. When a query of the same type as the second type is input by the second learning, the second domain corresponding to the second type is used in the estimation. The first domain is estimated by focusing more on the attribute values in the query than the second domain. The second domain is estimated by focusing more on the attributes in the query than the first domain. The first domain is estimated by placing more importance on knowledge than the second domain. The learning unit 204 may use a method other than multi-domain learning to learn using the second query of the first type and the second query of the second type.

When the learning unit 204 completes learning the pre-learning attribute value extraction model M3 stored in the model storage unit 200, it records the learned attribute value extraction model M3 in the model storage unit 200. The learning unit 204 transmits the learned attribute value extraction model M3 to the server 10. When the server 10 receives the learned attribute value extraction model M3, it records the learned attribute value extraction model M3 in the model storage unit 100.

[3-3. Functions realized by the estimated terminal]
The model storage unit 300 is realized by the storage unit 32. The availability determination unit 301, the first query acquisition unit 302, and the second attribute value extraction unit 303 are mainly realized by the control unit 31. These functions are examples of estimation functions.

[Model storage unit]
The model storage unit 300 stores the trained attribute value extraction model M3. For example, when the estimation terminal 30 downloads the trained attribute value extraction model M3 from the server 10, the estimation terminal 30 records the trained attribute value extraction model M3 in the model storage unit 300. The model storage unit 300 also stores an attribute database DB2. This attribute database DB2 may also be downloaded from the server 10.

[Availability determination unit]
The availability determination unit 301 determines the availability of at least one first attribute value associated with the first attribute related to the first product for estimation. In this embodiment, depending on the first attribute, the first attribute value may not be available. For example, when an unknown first attribute is input as the first query, the first attribute value is not available. For this reason, the second attribute for learning is stored in the attribute database DB2 in principle, but the first attribute for estimation is not necessarily stored in the attribute database DB2. Furthermore, even if the first attribute is stored in the attribute database DB2, if the first attribute value does not exist, it is not necessarily associated with the first attribute.

For example, the availability determination unit 301 refers to the attribute database DB2 and determines whether or not the first attribute exists. If the availability determination unit 301 determines that the first attribute does not exist in the attribute database DB2, it determines that the first attribute value is not available. If the availability determination unit 301 determines that the first attribute exists in the attribute database DB2, it determines whether or not the first attribute value associated with the first attribute exists in the attribute database DB2.

When the availability determination unit 301 determines that the first attribute value associated with the first attribute does not exist in the attribute database DB2, it determines that the first attribute value is not available. When the availability determination unit 301 determines that the first attribute value associated with the first attribute exists in the attribute database DB2, it determines that the first attribute value is available. Note that the method of determining availability is not limited to the example of this embodiment. For example, even if a first attribute value associated with the first attribute exists, the availability determination unit 301 may determine that the first attribute value is not available if the number of first attribute values is less than a threshold value or the frequency of the first attribute value is less than a threshold value.

In the present embodiment, the first attribute is specified by a store staff member, but the first attribute may be acquired by other methods as long as it is acquired in some form. For example, the manager of an online shopping mall may specify the first attribute. For example, the first attribute may be acquired by estimating the first attribute from the first data based on a machine learning model that has learned the relationship between product data and attributes. In another example, another attribute related to the attribute specified by the store staff member for the first product may be acquired as the first attribute. The same is true for the first data, and it can be acquired by any method other than being specified by the store staff member. For example, the first data and the first attribute may be stored in the estimation terminal 30 or another computer, and the stored first data and first attribute may be acquired.

[First query acquisition unit]
The first query acquisition unit 302 acquires a first query including at least one first attribute value associated with a first attribute related to a first product for estimation. In this embodiment, the first query includes, in principle, the first attribute and at least one first attribute value. However, since the first attribute value is not always available, in this case, the first query may include the first attribute but not the first attribute value. When the first attribute value is available, the first query may include only one first attribute.

In the knowledge token method, when it is determined that the first attribute value is available, the first query acquisition unit 302 acquires a third type of first query including third identification information indicating that the first attribute value is available and at least one first attribute value. The third identification information differs from the first identification information in that it indicates the availability of the first attribute value, but is similar to the first identification information in other respects. For this reason, in this embodiment, the third identification information is the character string [Seen]. The third type differs from the first type in that it is the type of first query for estimation, but the format itself as data is similar to the first type. The details of the third identification information and the third type are similar to the details of the first identification information and the first type.

When it is not determined that the first attribute value is available, the first query acquisition unit 302 acquires a fourth type of first query including fourth identification information indicating that the first attribute value is unavailable. The fourth identification information differs from the second identification information in that it indicates the availability of the first attribute value, but is otherwise similar to the second identification information. For this reason, in this embodiment, the fourth identification information is the character string [Unseen]. The fourth type differs from the second type in that it is the type of first query for estimation, but the format itself as data is similar to the fourth type. Details of the fourth identification information and the fourth type are similar to the details of the second identification information and the second type.

In this embodiment, the knowledge dropout method is assumed to be used during learning, not during estimation. Therefore, when it is determined that the first attribute value is available, the first query acquisition unit 302 acquires the first query so that all available first attribute values are included in the first query. The first query may include only a portion of the available first attribute values. In this case, the first query acquisition unit 302 may acquire a first query including only the most frequent first attribute value. For example, the first query acquisition unit 302 may acquire a first query including a randomly selected first attribute value, a first query including all first attribute values whose frequency is equal to or greater than a threshold, or a first query including a predetermined number of first attribute values in descending order of frequency. For example, the first attribute value may be included in the first query in an embedded representation state.

[Second attribute value extraction unit]
The second attribute value extraction unit 303 extracts the second attribute value from the first data based on the first data including the second attribute value related to the first product, the first query, and the trained attribute value extraction model M3. For example, the second attribute value extraction unit 303 inputs the first data and the first query to the trained attribute value extraction model M3. The trained attribute value extraction model M3 calculates an embedding vector (embedded expression) of the first data and the first query, and outputs a start point Pb and an end point Pe of the second attribute value based on the calculated embedding vector. The second attribute value extraction unit 303 extracts a portion of the first data from the start point Pb to the end point Pe as the second attribute value.

In the knowledge token method, when it is determined that the first attribute value is usable, the second attribute value extraction unit 303 extracts the second attribute value from the first data based on the first data including the second attribute value related to the first product, the first query of the third type, and the trained attribute value extraction model M3. The second attribute value extraction unit 303 inputs the first data and the first query of the third type to the trained attribute value extraction model M3. The trained attribute value extraction model M3 outputs the start point Pb and end point Pe of the second attribute value according to these embedding vectors.

If it is not determined that the first attribute value is available, the second attribute value extraction unit 303 extracts the second attribute value from the first data based on the first data, the first query of the fourth type, and the trained attribute value extraction model M3. The second attribute value extraction unit 303 inputs the first data and the first query of the fourth type to the trained attribute value extraction model M3. The trained attribute value extraction model M3 outputs the start point Pb and the end point Pe of the second attribute value according to these embedding vectors. Note that, if the attribute value extraction model M3 outputs the second attribute value itself instead of outputting the start point Pb and the end point Pe, the second attribute value extraction unit 303 may obtain the second attribute value output from the attribute value extraction model M3.

4. Processing Executed by the Learning System
Figures 9 and 10 are flow diagrams showing an example of processing executed by the learning system 1. The processing in Figures 9 and 10 is executed by the control units 11, 21, and 31 operating in accordance with programs stored in the storage units 12, 22, and 32, respectively. When the processing in Figures 9 and 10 is executed, the training database DB1 and the attribute database DB2 are assumed to have been created in advance.

As shown in FIG. 9, the learning terminal 20 determines a second product to be learned from among the second products whose second data is stored in the training database DB1 (S1). The second product to be learned is the second product that is the subject of processing from S2 onwards. For example, from among the second products whose second data is stored in the training database DB1, a second product that has not yet been trained by the attribute value extraction model M3 is determined as the learning target.

The learning terminal 20 refers to the training database DB1 and acquires the second data and the second attribute of the second product to be learned (S2). The learning terminal 20 refers to the attribute database DB2 and acquires all the third attribute values associated with the second attribute acquired in S2 (S3). The learning terminal 20 acquires a tentative second query including the second attribute acquired in S2 and all the third attribute values acquired in S3 (S4). A knowledge token may be inserted at the point of S4, but it is assumed here that no knowledge token is inserted yet. An SEP token is inserted between the second attribute and the third attribute value, and between the third attribute values.

The learning terminal 20 determines, for each third attribute value, the probability that the third attribute value will be excluded based on the exclusion rate and the frequency of the third attribute value (S5). As described above, the exclusion rate is specified in advance. The frequency is assumed to be stored in the attribute database DB2. The learning terminal 20 determines, for each third attribute value, whether or not to exclude the third attribute value based on the probability of the third attribute value (S6). As a result of the determination in S6, it is determined that some of all third attribute values will be excluded. Note that there are cases where it is determined with an extremely low probability that all third attribute values will be excluded, but this point is not taken into consideration in this embodiment.

The learning terminal 20 acquires a second query of the first type by replacing the third attribute values determined to be excluded in S6 from the provisional second query acquired in S4 with other values (e.g., padding tokens) (S7). In S7, the knowledge token of [Seen] is also inserted into the second query. The learning terminal 20 acquires a second query of the second type from the provisional second query acquired in S4, in which all third attribute values have been excluded (S8). In S8, the knowledge token of [Unseen] is also inserted into the second query.

The learning terminal 20 uses multi-domain learning to learn the attribute value extraction model M3 based on the second data, the first type second query, and the second type second query acquired in S2 (S9). In S9, the learning terminal 20 learns the attribute value extraction model M3 so that when the second data of the second product to be learned and the first type second query are input, the start point Pb and end point Pe of the fourth attribute value in the second data are output. The learning terminal 20 learns the attribute value extraction model M3 so that when the second data of the second product to be learned and the second type second query are input, the start point Pb and end point Pe of the fourth attribute value in the second data are output.

The learning terminal 20 determines whether or not learning of the attribute value extraction model M3 is complete (S10). The determination in S10 may be performed based on a predetermined condition. For example, learning of the attribute value extraction model M3 may be completed when all of the second data stored in the training database DB1 have been used in learning, or learning of the attribute value extraction model M3 may be completed when a certain number of second data have been used in learning.

If it is not determined in S10 that the learning of the attribute value extraction model M3 is completed (S10:N), the process returns to S1. In this case, the next learning target, the second product, is determined and the learning of the attribute value extraction model M3 continues. If it is determined that the learning of the attribute value extraction model M3 is completed (S10:Y), the learning terminal 20 completes the learning of the attribute value extraction model M3 and transmits the learned attribute value extraction model M3 to the server 10 (S11). When the server 10 receives the learned attribute value extraction model M3 (S12), it records the learned attribute value extraction model M3 in the memory unit 12 (S13). Thereafter, the learned attribute value extraction model M3 becomes available from the estimation terminal 30.

Moving on to FIG. 10, the estimation terminal 30 downloads the trained attribute value extraction model M3 from the server 10 and records it in the memory unit 22 (S14). The estimation terminal 30 acquires the first data of the first product pre-stored in the memory unit 32 and the first attribute specified by the store staff (S15). The estimation terminal 30 refers to the attribute database DB2 and determines the usability of the first attribute value (S16).

If it is determined that the first attribute value is usable (S16: Yes), the estimation terminal 30 acquires all first attribute values associated with the first attribute (S17). The estimation terminal 30 acquires a third type first query including the first attribute acquired in S15 and all the first attribute values (S18). The estimation terminal 30 extracts a second attribute value from the first data based on the first data, the third type first query, and the learned attribute value extraction model M3 (S19), and the process ends. In S19, estimation is performed using the first domain of the attribute value extraction model M3.

If it is not determined in S16 that the first attribute value is usable (S16: Not possible), the estimation terminal 30 acquires a first query of a fourth type that does not include the first attribute value but does include the first attribute (S20). The estimation terminal 30 extracts a second attribute value from the first data based on the first data, the first query of the fourth type, and the learned attribute value extraction model M3 (S21), and the process ends. In S21, estimation is performed using the second domain of the attribute value extraction model M3.

According to the learning system 1 of the present embodiment, learning is performed on the attribute value extraction model M3 based on the second data including the fourth attribute value and the second query in which at least a part of the multiple third attribute values associated with the second attribute related to the second product is excluded. As a result, by excluding the third attribute value that is actually available for learning, the attribute value extraction model M3 can learn the incompleteness of knowledge, thereby improving the accuracy of the attribute value extraction model M3. For example, in actual operation, it is necessary to use incomplete knowledge such as the attribute database DB2 that is not completely covered. By having the attribute value extraction model M3 learn this incompleteness, even if an unknown first attribute is input or a first attribute with almost no first attribute value is input in the attribute database DB2, the attribute value extraction model M3 can extract the second attribute value from the first data. Furthermore, since the query can be expanded more than in the conventional attribute value extraction model M2, the accuracy of the attribute value extraction model M3 is also improved in this respect.

In addition, in the learning system 1, the first query includes the first attribute and at least one first attribute value. The second query includes the second attribute, and at least some of the multiple third attribute values are excluded. This allows not only the first attribute value but also the first attribute to be included in the first query, and allows the second attribute to be included in the second query, thereby achieving more effective query expansion. As a result, the accuracy of the attribute value extraction model M3 is improved.

The learning system 1 also excludes only a portion of the multiple third attribute values, and obtains a second query that includes the remaining third attribute values that were not excluded from the multiple third attribute values. This allows a certain number of third attribute values to be included in the second query, making it possible to have the attribute value extraction model M3 learn about the incompleteness of knowledge while realizing query expansion.

Furthermore, the learning system 1 determines, for each third attribute value, whether or not to exclude that third attribute value based on the probability associated with that third attribute value, and excludes only a portion of the multiple third attribute values that have been determined to be excluded based on the probability. This makes it easier to manage how many third attribute values are left unexcluded, allowing the administrator to exclude third attribute values as intended. For example, it is possible to prevent more third attribute values than the administrator anticipates from being excluded, or to prevent fewer third attribute values than the administrator anticipates from being excluded. As a result, the attribute value extraction model M3 can learn the unnaturalness of knowledge as intended by the administrator.

Furthermore, the learning system 1 determines, for each third attribute value, whether or not to exclude the third attribute value based on a probability associated with the third attribute value according to the frequency of the third attribute value. This makes it possible, for example, to make it more difficult for the third attribute value to be excluded the higher the frequency of the third attribute value, making it easier to have the attribute value extraction model M3 learn frequently occurring third attribute values.

Furthermore, the learning system 1 determines the probability of each third attribute value based on a predetermined exclusion rate and the frequency of the third attribute value. The exclusion rate makes it easier to manage how many third attribute values are left, so that the administrator can exclude third attribute values as intended.

The learning system 1 also excludes only a portion of all the third attribute values, and obtains a second query that includes the remaining third attribute values that have not been excluded from all the third attribute values. This allows the attribute value extraction model M3 to learn a certain number of third attribute values.

The learning system 1 also acquires a second query including multiple third attribute values, and replaces some of the multiple third attribute values included in the second query with other values, thereby excluding the some third attribute values. In this way, by including the other values in the second query, it is possible to cause the attribute value extraction model M3 to learn about the incompleteness of knowledge.

The learning system 1 also performs a first learning process on the attribute value extraction model M3 based on the second data and the second query of the first type, and performs a second learning process on the attribute value extraction model M3 based on the second data and the second query of the second type. This allows the attribute value extraction model M3 to learn the third attribute value while more efficiently learning about the incompleteness of knowledge. Furthermore, compared to creating an attribute value extraction model M3 for when the first attribute value is available and an attribute value extraction model M3 for when the first attribute value is not available separately, they can be consolidated into a single attribute value extraction model M3, reducing the management burden of the attribute value extraction model M3.

The learning system 1 also performs the first learning and the second learning by using multi-domain learning in which the availability of the third attribute value is regarded as a domain. This allows the estimation process to be switched in one attribute value extraction model M3 depending on the availability of the first attribute value at the time of actual estimation. This allows for a hybrid attribute value extraction model M3 that can handle both cases where the first attribute value is available and cases where it is not available.

Furthermore, the learning system 1 learns so that when the second data and the second query are input to the attribute value extraction model M3, the attribute value extraction model M3 outputs partial identification information capable of identifying the portion of the fourth attribute value in the second data. This makes it possible to identify which portion of the first data is the second attribute value during actual estimation, making it easier to extract the second attribute value from the first data.

In addition, in the learning system 1, the first item is a first product for estimation, and the second item is a second product for learning. This allows for accurate extraction of attribute values of products traded in online shopping malls.

The learning system 1 also extracts the second attribute value from the first data based on the first data including the second attribute value related to the first product, the first query including the first attribute value, and the trained attribute value extraction model M3. This enables query expansion, so that the second attribute value can be extracted with high accuracy.

Furthermore, when it is determined that the first attribute value is available, the learning system 1 extracts the second attribute value from the first data based on the first data including the second attribute value related to the first item, the first query of the third type, and the learned attribute value extraction model M3. When it is not determined that the first attribute value is available, the learning system 1 extracts the second attribute value from the first data based on the first data, the first query of the fourth type, and the learned attribute value extraction model M3. This makes it possible to accurately extract the second attribute value in both cases, when the first attribute value is available and when the first attribute value is not available.

5. Modifications
The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure.

Figure 11 is an example of a learning system and an attribute value extraction system in a modified example. As shown in Figure 11, the learning system 1 and the attribute value extraction system 2 may be separate systems. The attribute value extraction system 2 is a system that can use a trained attribute value extraction model created by the learning system 1. In the example of Figure 11, the learning system 1 includes a learning terminal 20. The attribute value extraction system 2 includes a server 10 and an estimation terminal 30. The functions of the server 10, the learning terminal 20, and the estimation terminal 30 are as described in the embodiment. The attribute value extraction system 2 may include only the estimation terminal 30 without including the server 10.

For example, in the embodiment, a case has been described in which both the knowledge dropout method and the knowledge token method are used, but the learning terminal 20 may learn the attribute value extraction model M3 using only the knowledge dropout method without using the knowledge token method. In this case, the first query and the second query do not include a knowledge token. For example, the learning terminal 20 may learn the attribute value extraction model M3 using only the knowledge token method without using the knowledge dropout method. In this case, the first query may include all third attribute values without excluding the third attribute values.

For example, the functions described as being realized by the learning terminal 20 may be realized by the server 10, the estimation terminal 30, or another computer, or may be shared among multiple computers. For example, the functions described as being realized by the estimation terminal 30 may be realized by the server 10, the learning terminal 20, or another computer, or may be shared among multiple computers.

[6. Notes]
For example, the learning system and the attribute value extraction system can be configured as follows.

(1)
a model storage unit that stores an attribute value extraction model for extracting a second attribute value from first data including a second attribute value related to a first item for estimation, by using a first query including at least one first attribute value associated with the first attribute related to the first item;
a second query acquisition unit that acquires a second query in which at least a part of a plurality of third attribute values associated with a second attribute related to a second item for learning is excluded;
a learning unit that performs learning on the attribute value extraction model based on second data including a fourth attribute value related to the second item and the second query;
A learning system including:
(2)
the first query includes the first attribute and the at least one first attribute value;
the second query includes the second attribute and excludes at least a portion of the third attribute values;
A learning system as described in (1).
(3)
The second query acquisition unit is
excluding only a portion of the plurality of third attribute values;
obtaining the second query including the remaining third attribute values that have not been excluded from the plurality of third attribute values;
A learning system according to (1) or (2).
(4)
The second query acquisition unit is
determining, for each of the third attribute values, whether to exclude the third attribute value based on a probability associated with the third attribute value;
excluding only a portion of the plurality of third attribute values that is determined to be excluded based on the probability;
A learning system as described in (3).
(5)
the learning system further includes a probability determination unit that determines, for each of the third attribute values, the probability of the third attribute value based on a frequency of the third attribute value in a first database in which a plurality of the second data are stored;
the second query acquisition unit determines, for each of the third attribute values, whether or not to exclude the third attribute value based on the probability associated with the third attribute value and corresponding to the frequency of the third attribute value;
A learning system as described in (4).
(6)
the probability determination unit determines the probability of each of the third attribute values based on a predetermined exclusion rate and the frequency of the third attribute value;
A learning system as described in (5).
(7)
The learning system further includes a third attribute value acquisition unit that acquires all of the third attribute values stored in a second database in which the plurality of third attribute values are stored,
The second query acquisition unit is
excluding only a portion of all the third attribute values;
obtaining the second query including the remaining third attribute values that have not been excluded from among all the third attribute values;
A learning system according to any one of (3) to (6).
(8)
The second query acquisition unit is
obtaining the second query including the third attribute values;
replacing a part of the plurality of third attribute values included in the second query with another value, thereby excluding the part of the third attribute values;
A learning system according to any one of (3) to (7).
(9)
Depending on the first attribute, the first attribute value may not be available;
The second query acquisition unit is
obtaining the second query of a first type including first identification information indicating that the third attribute value is available and at least a portion of the plurality of third attribute values;
obtaining the second query of a second type including second identification information indicating that the third attribute value is not available, the second query excluding all of the third attribute values;
The learning unit is
a first learning of the attribute value extraction model based on the second data and the second query of the first type;
performing second learning on the attribute value extraction model based on the second data and the second query of the second type;
A learning system according to any one of (1) to (8).
(10)
the learning unit performs the first learning and the second learning by using multi-domain learning in which availability of the third attribute value is regarded as a domain.
A learning system as described in (9).
(11)
the learning unit performs the learning such that, when the second data and the second query are input to the attribute value extraction model, the attribute value extraction model outputs partial identification information capable of identifying a portion of the fourth attribute value in the second data.
A learning system according to any one of (1) to (10).
(12)
the first item is a first commodity for estimation;
the second item is a second product for learning;
the first data includes a first string relating to a description of the first product;
the second data includes a second string relating to a description of the second product;
the attribute value extraction model is a model related to natural language processing for extracting the second attribute value from the first character string;
A learning system according to any one of (1) to (11).
(13)
An attribute value extraction system capable of using a trained attribute value extraction model created by the learning system according to any one of (3) to (8),
a first query acquisition unit that acquires a first query including at least one first attribute value associated with a first attribute related to a first item for estimation;
a second attribute value extraction unit that extracts the second attribute value from the first data based on first data including a second attribute value related to the first item, the first query, and the trained attribute value extraction model;
An attribute value extraction system comprising:
(14)
An attribute value extraction system capable of using a trained attribute value extraction model created by the learning system according to (9) or (10),
an availability determination unit that determines availability of at least one first attribute value associated with a first attribute related to a first item for estimation;
a first query acquisition unit that acquires a third type of first query including third identification information indicating that the first attribute value is available and the at least one first attribute value when it is determined that the first attribute value is available, and acquires a fourth type of first query including fourth identification information indicating that the first attribute value is not available when it is not determined that the first attribute value is available;
a second attribute value extraction unit that extracts the second attribute value from the first data based on first data including a second attribute value related to the first item, a first query of the third type, and the learned attribute value extraction model when it is determined that the first attribute value is available, and extracts the second attribute value from the first data based on the first data, the first query of the fourth type, and the learned attribute value extraction model when it is not determined that the first attribute value is available;
An attribute value extraction system comprising:

1 Learning system, 2 Attribute value extraction system, N Network, P Product page, 10 Server, 11, 21, 31 Control unit, 12, 22, 32 Memory unit, 13, 23, 33 Communication unit, 20 Learning terminal, 24, 34 Operation unit, 25, 35 Display unit, 30 Estimation terminal, P Product page, M1 Question answering model, M2, M3 Attribute value extraction model, Pb Start point, Pe End point, 100, 200 Model memory unit, 201 Third attribute value acquisition unit, 202 Probability determination unit, 203 Second query acquisition unit, 204 Learning unit, 300 Model memory unit, 301 Availability determination unit, 302 First query acquisition unit, 303 Second attribute value extraction unit, N10 News article, R12 Response, D20, D30 Product data, DB1 Training database, DB2 Attribute database, I41, I42 input, Q11, Q21, Q31, Q40 query, V22, V32 attribute value.

Claims (15)

a model storage unit that stores an attribute value extraction model for extracting a second attribute value from first data including a second attribute value related to a first item to be estimated, by using a first query including a first attribute related to the first item and at least one first attribute value associated with the first attribute;
a second query acquisition unit that acquires a second query including a second attribute related to a second item for learning and a third attribute value that has not been excluded from among a plurality of third attribute values associated with the second attribute;
a learning unit that causes the attribute value extraction model to learn training data including, as an input portion, second data including a fourth attribute value related to the second item and associated with the second attribute, and the second query, and including , as an output portion, information capable of identifying the fourth attribute value in the second data;
A learning system including: The second query acquisition unit is
excluding only a portion of the plurality of third attribute values;
obtaining the second query including the remaining third attribute values that have not been excluded from the plurality of third attribute values;
The learning system according to claim 1 . The second query acquisition unit is
determining, for each of the third attribute values, whether to exclude the third attribute value based on a probability associated with the third attribute value;
excluding only a portion of the plurality of third attribute values that is determined to be excluded based on the probability;
The learning system according to claim 2 . the learning system further includes a probability determination unit that determines, for each of the third attribute values, the probability of the third attribute value based on a frequency of the third attribute value in a first database in which a plurality of the second data are stored;
the second query acquisition unit determines, for each of the third attribute values, whether or not to exclude the third attribute value based on the probability associated with the third attribute value and corresponding to the frequency of the third attribute value;
The learning system according to claim 3 . the probability determination unit determines the probability of each of the third attribute values based on a predetermined exclusion rate and the frequency of the third attribute value;
The learning system according to claim 4 . The learning system further includes a third attribute value acquisition unit that acquires all of the third attribute values stored in a second database in which the plurality of third attribute values are stored,
The second query acquisition unit is
excluding only a portion of all the third attribute values;
obtaining the second query including the remaining third attribute values that have not been excluded from among all the third attribute values;
The learning system according to claim 2 . The second query acquisition unit is
obtaining the second query including the third attribute values;
replacing a part of the plurality of third attribute values included in the second query with another value, thereby excluding the part of the third attribute values;
The learning system according to claim 2 . Depending on the first attribute, the first attribute value may not be available;
The second query acquisition unit is
obtaining the second query of a first type including first identification information indicating that the third attribute value is available and at least a portion of the plurality of third attribute values;
obtaining the second query of a second type including second identification information indicating that the third attribute value is not available, the second query excluding all of the third attribute values;
The learning unit is
a first learning of the attribute value extraction model based on the second data and the second query of the first type;
performing second learning on the attribute value extraction model based on the second data and the second query of the second type;
A learning system according to any one of claims 1 to 7 . the learning unit performs the first learning and the second learning by using multi-domain learning in which availability of the third attribute value is regarded as a domain.
The learning system according to claim 8 . the learning unit performs the learning such that, when the second data and the second query are input to the attribute value extraction model, the attribute value extraction model outputs partial identification information capable of identifying a portion of the fourth attribute value in the second data.
A learning system according to any one of claims 1 to 7 . the first item is a first commodity for estimation;
the second item is a second product for learning;
the first data includes a first string relating to a description of the first product;
the second data includes a second string relating to a description of the second product;
the attribute value extraction model is a model related to natural language processing for extracting the second attribute value from the first character string;
A learning system according to any one of claims 1 to 7 . An attribute value extraction system capable of using a trained attribute value extraction model created by the learning system according to claim 2 ,
a first query acquisition unit that acquires a first query including at least one first attribute value associated with a first attribute related to a first item for estimation;
a second attribute value extraction unit that extracts the second attribute value from the first data based on first data including a second attribute value related to the first item, the first query, and the trained attribute value extraction model;
An attribute value extraction system comprising: An attribute value extraction system capable of using a trained attribute value extraction model created by the learning system according to claim 8 ,
an availability determination unit that determines availability of at least one first attribute value associated with a first attribute related to a first item for estimation;
a first query acquisition unit that acquires a third type of first query including third identification information indicating that the first attribute value is available and the at least one first attribute value when it is determined that the first attribute value is available, and acquires a fourth type of first query including fourth identification information indicating that the first attribute value is not available when it is not determined that the first attribute value is available;
a second attribute value extraction unit that extracts the second attribute value from the first data based on first data including a second attribute value related to the first item, a first query of the third type, and the learned attribute value extraction model when it is determined that the first attribute value is available, and extracts the second attribute value from the first data based on the first data, the first query of the fourth type, and the learned attribute value extraction model when it is not determined that the first attribute value is available;
An attribute value extraction system comprising: The computer
a second query acquisition step of acquiring a second query including a second attribute related to a second item for learning and a third attribute value that has not been excluded among a plurality of third attribute values associated with the second attribute;
a learning step of training data including, as an input portion, second data including a fourth attribute value related to the second item and the fourth attribute value related to the second attribute, and the second query, and including, as an output portion, information capable of identifying the fourth attribute value in the second data, using a first query including a first attribute related to a first item to be estimated and at least one first attribute value associated with the first attribute, to learn an attribute value extraction model for extracting the second attribute value from first data including the second attribute value related to the first item, by using the first query;
Learn how to do it . a second query acquisition unit that acquires a second query including a second attribute related to a second item for learning and a third attribute value that has not been excluded from among a plurality of third attribute values associated with the second attribute;
a learning unit that causes an attribute value extraction model that stores an attribute value extraction model for extracting the second attribute value from first data including a second attribute value related to the first item to learn training data including, as an input portion, second data including a fourth attribute value related to the second item , the fourth attribute value being related to the second attribute, the second query, the second data including the fourth attribute value different from the third attribute value , and the second query , and including, as an output portion, information capable of identifying the fourth attribute value in the second data, by using a first query including a first attribute related to a first item to be estimated and at least one first attribute value associated with the first attribute;
A program that makes a computer function as a

Images