JP6947768B2 - Estimator, estimation method and estimation program - Google Patents

Description

The present invention relates to an estimation device, an estimation method and an estimation program.

In recent years, there have been known techniques for realizing various classification processes such as language recognition and image recognition using a model such as DNN (Deep Neural Network) having neurons connected in multiple stages. For example, in such a technique, when a predetermined input information is input to the model, the weight coefficient between the nodes of the model is output so that the output information desired by the user is output as the output information corresponding to the input information. By modifying (connection coefficient), the model is made to learn the characteristics of the input information.

Japanese Unexamined Patent Publication No. 2017-162074

However, with the above-mentioned learning technique, it may be difficult to confirm what kind of features the model is learning.

For example, the trained model described above outputs output information according to the characteristics of the input information. However, among the features of the input information, such a model outputs the output information according to the features assumed by the user, or outputs the output information according to the features not assumed by the user. It is unknown whether it is outputting. Therefore, the model in which the above-mentioned learning is performed may output unexpected output information to the user.

The present application has been made in view of the above, and an object of the present application is to provide information indicating the features learned by the model.

The estimation device according to the present application identifies a plurality of classification targets for which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target when the target information regarding the classification target is input. It is characterized by having a unit and an estimation unit that estimates an index when the model classifies the classification object based on the target information of a plurality of classification objects specified by the specific unit.

According to one aspect of the embodiment, it is possible to provide information indicating the features learned by the model.

FIG. 1 is a diagram showing an example of estimation processing executed by the information providing device according to the embodiment. FIG. 2 is a diagram showing an example of estimation processing executed by the information providing device according to the embodiment based on the comparison result of different groups. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 4 is a diagram showing an example of a user database according to the embodiment. FIG. 5 is a diagram showing an example of a model database according to the embodiment. FIG. 6 is a flowchart showing an example of the flow of the estimation process according to the embodiment. FIG. 7 is a diagram showing an example of a hardware configuration.

Hereinafter, a mode for carrying out the estimation device, the estimation method, and the estimation program according to the present application (hereinafter, referred to as “the embodiment”) will be described in detail with reference to the drawings. Note that this embodiment does not limit the estimation device, estimation method, and estimation program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1. About an example of information providing device]
First, an example of the information providing device 10 will be described with reference to FIG. 1 as an example of the estimation device. FIG. 1 is a diagram showing an example of estimation processing executed by the information providing device according to the embodiment. In the example shown in FIG. 1, as an example of the estimation process, it is used from the user information which is various information about the user such as the user's psychographic attribute, demographic attribute, search history, purchase history, or browsing history. Regarding the model for classifying users, the flow of processing for estimating the index according to which index the model classifies users has been described, but the embodiment is not limited to this. As will be clarified in the explanation described later, the information providing device 10 can execute the estimation process for any classification target such as various contents other than the user.

The information providing device 10 is an information processing device that executes estimation processing, and is realized by, for example, a server device, a cloud system, or the like. For example, the information providing device 10 can communicate with external terminals 101, 102, 201, 202 used by various businesses via a network N (for example, see FIG. 6) such as the Internet. The information providing device 10 may be capable of communicating with an arbitrary number of external terminals 101, 102, 201, 202.

The external terminals 101, 102, 201, and 202 are information processing devices used by various businesses, and are realized by a server device, a cloud system, or the like. For example, the external terminals 101 and 102 (hereinafter, may be referred to as "external terminal 100") are information processing devices used by a business operator on the registration side that registers various types of information in the information providing device 10. .. Further, the external terminals 201 and 202 (hereinafter, may be referred to as "external terminal 200") are information processing devices used by a business operator who receives information from the information providing device 10. The external terminal 100 may operate as the external terminal 200, and the external terminal 200 may operate as the external terminal 100. That is, as the external terminals 100 and 200, any information processing device can be adopted as long as it is an information processing device used by a business operator different from the information providing device 10.

In addition, each business operator may be a business operator related to any business. For example, each business operator may be a manufacturing industry of arbitrary various products such as automobiles and groceries, or may be a business operator related to a sales business of these products. In addition, each business operator may be a business operator that provides various services.

[1-1. About the provision process provided by the information providing device]
Here, an example of the provision process executed by the information providing device 10 will be described prior to the explanation of the estimation process executed by the information providing device 10. For example, the information providing device 10 executes a providing process of providing information registered by each business operator to another business operator. That is, the information providing device 10 provides an information sharing service that promotes the aggregation and sharing of information owned by each business operator.

Hereinafter, an example of the flow of the provision process will be described. For example, the external terminal 100 collects user information by using various information collecting techniques (step S1). For example, the external terminal 100 collects user information # U1 which is user information of user U1 and user information # U2 which is user information of user U2. The external terminal 100 may collect user information of a user who is a service provider, or may collect user information collected through various information providing services. Subsequently, the external terminal 100 generates a model for classification based on user information (step S2). For example, the external terminal 100 generates a model for performing various arbitrary classifications in order to provide it for its own business.

For example, the external terminal 100 learns the model using the collected user information as learning data so as to output the output information of the content desired by the business operator when the user information is input. For example, when the external terminal 100 inputs user information, it learns a model so as to output output information indicating whether or not to provide its own advertisement to the user corresponding to the user information. May be done. Further, when the external terminal 100 inputs the user information of a predetermined type (for example, the history of the search query input by the user), the external terminal 100 indicates the user information of another type (for example, the hobby and taste of the user). The model may be trained so as to output information). Further, the external terminal 100 may learn the model so as to output output information indicating whether or not the user is interested in the company's product when the user information is input. That is, the external terminal 100 may learn the model used for any purpose as long as it learns the model by inputting the information about the user to be classified.

It should be noted that such a model can be realized by, for example, a so-called DNN (Deep Neural Network) in which a plurality of nodes are connected in multiple stages. For example, the external terminal 100 learns the model by various arbitrary learning methods such as backpropagation. In addition, such a model executes various classifications based on the characteristics of the input user information and outputs the execution result, so that not only the simple user classification but also the information about the user can be obtained. Processing such as estimation can be realized. In other words, the model can realize various processes by classifying the input information according to the index according to the purpose of use.

Subsequently, the external terminal 100 registers the generated model and user information in the information providing device 10 (step S3). The external terminal 100 may register only the user information without generating the model. Further, the external terminal 100 may register only the model. On the other hand, the information providing device 10 also collects various user information even in its own device (step S4). When standing up, the information providing device 10 collects user information via an information route different from that of the user who uses the external terminal 100. As a result, the information providing device 10 collects user information having contents different from those of the external terminal 100. For example, the information providing device 10 may collect information of a type that was not collected by the external terminal 100, or may collect information of the same type as that of the external terminal 100.

Subsequently, the information providing device 10 acquires the purpose of use of the model from the external terminal 200 (step S5). For example, the information providing device 10 acquires the input information and what kind of output information is desired to be output when the input information is input for the purpose of use. In such a case, the information providing device 10 extracts and generates a model for classification based on the user information according to the purpose of use (step S6). For example, the information providing device 10 may extract a model satisfying the purpose of use from the models registered by the external terminal 100, and the user information registered by the external terminal 100 and the use collected by the information providing device 10. A model that satisfies the purpose of use may be learned from the person information. Further, the information providing device 10 may generate a new model from the model registered by the external terminal 100. Then, the information providing device 10 provides the generated or extracted model to the external terminal 200 (step S7).

The information providing device 10 may provide the user information collected by the external terminal 100 to the external terminal 200. Further, the information providing device 10 may perform a process of providing various statistical information obtained by statistically processing the user information. Further, the information providing device 10 may learn the model by using the user information registered by the plurality of external terminals 100, that is, the user information individually collected by the plurality of businesses, and use these. Person information may be provided. Further, the information providing device 10 receives the consideration for the use of each user information and the model from the external terminal 200, and provides a part of the received consideration to the business operator who registered the used user information and the model. May be good.

Through such a provision process, the information providing device 10 can realize effective utilization of the information owned by each business operator. Further, when the characteristics of the user information are learned by DNN or the like, the classification process based on such characteristics and characteristics is demonstrated by the enumeration of the connection coefficients constituting the DNN, and thus is used as the learning data. The contents of user information can be hidden. As a result, it is possible to promote the effective use of user information while maintaining the privacy of each user.

[1-2. About the model]
In this way, in a model such as DNN, classification processing based on features is realized by a list of connection coefficients, so even if the model itself is analyzed, what kind of index the model performs classification processing is based on. Not clear. Therefore, it may be difficult to confirm what kind of features the model to be used is learning. In this way, if it is not clear what kind of features the model is learning and what kind of index the model is based on, it can be used for the purpose of use related to user safety. , Will be difficult.

Hereinafter, specific examples of indicators when the model classifies users will be described. In the following description, the index for classification may be described as "classification index". For example, the first business operator collects user attribute information as user information and collects whether or not each user has selected a predetermined advertisement. Then, the first business operator learns the model so that when the attribute information of the user is input to the model, the output information indicating whether or not the user has selected an advertisement belonging to a predetermined category is output. I do. As a result of such a learning process, the first business operator can generate a model that realizes a classification process of whether or not to like an advertisement belonging to a predetermined category based on the attribute information of the user.

However, unlike the model that operates on a rule basis, the model in which such learning is performed is classified based on the characteristics of the attribute information to determine whether or not the advertisement belonging to a predetermined category is liked. .. Therefore, it is difficult for the second business operator who wants to use this model to identify what kind of attribute information the user has to classify the advertisement belonging to the predetermined category as a favorite user.

A more specific example will be described. For example, if the attribute information of "users who have selected advertisements related to ramen" contains a lot of information such as "like ramen", the model refers to users who have attribute information such as "like ramen" as "ramen". It is also conceivable to learn to classify advertisements related to "users who can easily select". However, for example, if the attribute information of "users who have selected advertisements related to ramen" contains a lot of information such as "like hamburger" in addition to information such as "like ramen", the model is "hamburger is". There is a possibility that users who have attribute information such as "like" are learning to be classified as "users who can easily select advertisements related to ramen".

On the other hand, when attribute information including only information such as "I like ramen" or "I like hamburgers" is adopted as learning data, the learning data is biased. Therefore, for example, information such as "I like ramen" and information such as "I like hamburger" are not included in the attribute information, but the attribute information is similar to the attribute information of "users who selected advertisements related to ramen". It will not be possible to classify people as "users who can easily select advertisements related to ramen."

In this way, the model generated by each business operator and the information providing device 10 outputs the output information that the business operator wants to output to the model when the predetermined input information is input and the input information is input. Learning is done as if to do. In other words, the model is trained to output output information according to the purpose of use when predetermined input information is input. Therefore, although the model executes the classification process according to the purpose of use according to the characteristics of the input information, the classification process is performed based on the characteristics different from the characteristics assumed by the user who uses the model. There is a risk of

[1-3. About estimation processing]
Therefore, the information providing device 10 estimates the classification index when the trained model performs the classification processing by executing the following processing. More specifically, the information providing device 10 estimates the classification index of each model based on the similarity of the output information and the similarity of the user information (step S8). More specifically, the information providing device 10 outputs a plurality of classifications in which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target when the target information regarding the classification target is input. Identify the target. Then, the information providing device 10 estimates an index when the model classifies the classification target based on the target information of the specified plurality of classification targets.

For example, the information providing device 10 selects a model to be estimated from the registered or generated models, and inputs various user information to the selected model. More specifically, the information providing device 10 inputs not only the user information used for learning the selected model but also the user information independently collected by the information providing device 10 into the model. Then, the information providing device 10 identifies users having similar output information output by the model as users belonging to the similar group. That is, the information providing device 10 identifies the users whose models are classified into the same group.

Here, the user information of the users classified into the same group (hereinafter, referred to as "classified users") has at least a feature that causes the model to classify the classified users into the same group. It is considered to be included as commonality. Therefore, the information providing device 10 refers to the user information of the classified user and identifies the common user information. For example, the information providing device 10 specifies attributes, search histories, and the like that are commonly included in the user attributes of the classified users. Then, the information providing device 10 uses information indicating the specified attribute, search history, and the like as a classification index.

For example, when the information providing device 10 includes information such as "I like ramen" in common in the user information of the classification user, the user who "likes ramen" is used as a classification index of the model. Generate information such as "a model that makes it easy to classify into the same group." Further, for example, in the information providing device 10, when the user information of the classified user includes information such as "I like ramen" or "I like hamburger" in common, information such as "I like ramen". Identify the percentage of classified users that include "I like hamburgers" and the percentage of classified users that include information such as "I like hamburgers." Then, when the proportion of classified users including information such as "like hamburgers" is higher, the information providing device 10 generates information such as "a model that makes it easy to classify users who like" hamburgers "into the same group". ..

That is, the information providing device 10 identifies the user information having a high correlation with the output information among the user information. In other words, the information providing device 10 identifies and identifies the commonality that is presumed to be most relevant to the classification by the model when the user information of the classification users has a plurality of commonalities of different contents. Estimate the classification index of the model based on the content of sex.

The classification index of the model estimated in this way can be used as useful information when using the model. Therefore, the information providing device 10 provides the estimated classification index to the business operator who intends to use the model (step S9). As a result, the information providing device 10 not only facilitates the selection of the model according to the purpose of use, but can also present what kind of index the model performs the classification based on. As a result, in the information providing device 10, for example, when the business operator using the model and the business operator generating the model are different, that is, the business operator using the model learns what kind of learning data the model is based on. Even if you do not know if it was done, you can promote the use of the model and the use of the information sharing service.

[1-4. About estimation of classification index]
Here, in the above description, the information providing device 10 defines users having similar output information, that is, users classified into the same group as classified users, and based on the user information of the classified users, the information providing device 10 is used. The classification index of the classified model was estimated. Here, the information providing device 10 identifies the user information having a high correlation with the classification result by the model by using at least the user information of the classification user, and estimates the classification index from the specified user information. For example, the classification index may be estimated by any method. Therefore, in the following description, variations of processing in which the information providing device 10 estimates the classification index will be described.

[1-4-1. Estimate based on the similarity of classification users]
For example, in order to learn a model for estimating whether or not a user selects a predetermined advertisement according to the content of a search query, a user who has selected a predetermined advertisement or a similar advertisement and a user who has not selected the predetermined advertisement are not selected. Prepare the search history which is the history of the search query entered by the user, and when entering the search history of the user who selected the advertisement, output that the user selects the advertisement and select the advertisement. When the search history of a user who does not exist is input, the model is trained so that the user does not select an advertisement. However, when user information other than the search history is input during such learning, learning is made to estimate whether or not the user selects an advertisement based on the user information other than the search history. May be done. On the other hand, if user information other than the search history is not input, the classification accuracy may decrease or users without a search history may not be classified. Therefore, in order to ensure the classification accuracy when using the model, in the learning of the model, learning by inputting a plurality of types of user information is often performed.

Therefore, the information providing device 10 may estimate whether the model uses an unintended classification index based on the commonality of the user information of the classification users. For example, the information providing device 10 may output similar output information by a model trained to output output information according to the characteristics of a predetermined type of information among the target information as a model. The index for classifying the classification target by the model may be estimated based on the commonality of the information of the types other than the predetermined type among the target information of the specified classification target.

For example, the information providing device 10 specifies the user information as learning data when learning the model specified as the estimation target, and also specifies the type of the user information that is emphasized among the user information as the assumed type. .. Further, the information providing device 10 collects user information other than the user information used as learning data, and inputs the collected user information into the model to identify the classified user. Then, the information providing device 10 is a type or combination of types of user information having a high correlation with the output information among the user information of the specified classified users, and is a type or type combination different from the assumed type. Identify the combination.

For example, in the information providing device 10, when the user information used as learning data includes gender, search history, and purchase history, and the search history is emphasized when learning the model, the gender and purchase search history , And identify the correlation between each of the purchase history and the output information. Then, when the correlation between the gender and the output information or the correlation between the purchase history and the output information is higher than the correlation between the search history, the information providing device 10 learns a feature not assumed by the model. You may output as an estimation index that you are doing this or that you are learning the characteristics of gender or purchase history.

Further, the information providing device 10 may specify a combination of a plurality of types of user information as a classification index by the model. For example, in the information providing device 10, when the user information of the classified user includes gender, search history, and purchase history, the information providing device 10 includes a pair of gender and search history, a pair of gender and purchase history, and a search history and purchase history. The correlation with the output information is specified for each pair of and, and for each pair of gender, search history, and purchase history. For example, the information providing device 10 correlates the similarity between the user information of each classification user and the similarity of the output information of each classification user for each combination of attributes (that is, each similarity). (Similarity) is calculated. Then, the information providing device 10 may output the set having the highest calculated score as an estimation index. Further, when the information providing device 10 has a group other than the set of the assumed type that has a higher correlation with the output information than the set of the assumed type, the information providing device 10 provides information that the set is an estimation index. It may be output.

[1-4-2. Estimate based on comparison with other users]
Further, the information providing device 10 provides user information of classified users classified into a certain group and user information of classified users classified into another group (hereinafter, referred to as "comparative users"). The estimation index may be estimated based on the comparison result with. For example, the information providing device 10 specifies a classification target for which similar output information is output from the target information by the model as the first group, and specifies a classification target for which the target information has a predetermined similarity as the second group. .. Then, the information providing device 10 estimates an index when the model classifies the classification target based on the comparison between the target information of the classification target belonging to the first group and the target information of the classification target belonging to the second group. ..

Here, the classification target belonging to the second group may be classified based on the similarity of the target information itself. That is, the second group to be compared may be classified by a model that is a white box (for example, rule-based), unlike the model that is an estimation target that is a black box. More specifically, the information providing device 10 is a model different from the model to be estimated as the second group, and is a model for classifying the classification target according to whether or not the target information satisfies a predetermined condition. It is sufficient to specify the classification target classified into the same group by. Here, the classification target classified into the first group and the classification target classified into the second group may or may not have overlap.

For example, FIG. 2 is a diagram showing an example of estimation processing executed by the information providing device according to the embodiment based on the comparison result of different groups. For example, the information providing device 10 inputs each user information into a model to be estimated, and sets users having similar output information output by the model as a first group, that is, a group of classified users (step S10). ). For example, the information providing device 10 inputs the user information # U1 to # U5 of the users U1 to U5 into the model, respectively, and obtains the output information # U1 to # U5. Then, the information providing device 10 mutually identifies the similarity of each output information # U1 to # U5, and extracts users having high similarity (for example, users U1, U3, U5) as a first group.

Subsequently, the information providing device 10 sets users with similar user information as a second group on a rule basis (step S11). For example, the information providing device 10 specifies user information of the same predetermined type from the user information # U1 to # U5, and the user corresponding to the specified user information (for example, user U2). , U4, U6) are specified as comparative users. Then, the information providing device 10 has a comparative user as a second group.

Then, the information providing device 10 estimates the classification index when the model classifies from the comparison between the user information of the first group and the user information of the second group (step S12). For example, the information providing device 10 has the commonality of the user information # U2, # U4, and # U6 of the second group among the commonality of the user information # U1, # U3, and # U5 of the first group. The same commonality may be used as a classification index, and different commonalities may be used as a classification index.

The information providing device 10 may specify the commonality of the user information of the first group, and specify the users of the second group based on the specified commonality. For example, in the information providing device 10, when a predetermined search query is included in the search history in most of the user information of the first group, the user whose search history includes the predetermined search query among the users. May be extracted as the second group. Then, when the information providing device 10 has a discrepancy between the user belonging to the first group and the user belonging to the second group, or when the user information of the user belonging to the first group and the user information of the user belonging to the first group are combined with the second group. When the commonality with the user information of the user to which the user belongs is equal to or less than a predetermined threshold value, among the commonality of the user information belonging to the first group, "a predetermined search query is included as the search history". A commonality other than the commonality may be used as an estimation index.

[1-4-3. Estimate using correlation]
The information providing device 10 may estimate an index when the model classifies the classification target based on the correlation between the target information of the classification target classified in the same group and the output information output by the model. good. For example, the information providing device 10 inputs the user information of each user into the model and obtains the output information. For example, the information providing device 10 obtains a value output by the model (that is, a value indicating the probability that the input user information and the corresponding user are classified into a predetermined group).

Subsequently, the information providing device 10 digitizes various types of information included in the user information by using a predetermined method. For example, the information providing device 10 sets a numerical value that identifies various types of information, such as "0" if the gender is male and "1" if the gender is female, so that the user information can be converted into a numerical value for each type. replace. Subsequently, the information providing device 10 associates each type and the output information with individual axial directions in the multidimensional space, and on the multidimensional space corresponding to each type of numerical value and the numerical value of the output information. Plot the position. Then, the information providing device 10 may execute such a process for each user, and based on the plotted positions, may investigate the type having a high correlation with the output information among the various types. That is, the information providing device 10 may estimate the type of target information whose correlation with the output information satisfies a predetermined condition as an index when the model classifies the classification target.

The information providing device 10 projects, for example, a multidimensional space leaving a dimension corresponding to a predetermined one or a plurality of types, and whether or not the plot position after projection satisfies a predetermined condition (for example, output). The correlation between the output information and the set of each type or type may be estimated based on whether the information has a linear change or whether it has a Bernoulli distribution. That is, the information providing device 10 may estimate a set of information types whose correlation with the output information output by the model satisfies a predetermined condition as an index when the model classifies the classification target.

Further, the information providing device 10 may confirm the correlation by limiting the user information to be input to the model. For example, the information providing device 10 refers to user information and identifies a user who has common information of the type corresponding to the purpose of use of the model. For example, when the model classifies "people wearing glasses" and other people, the information providing device 10 selects users who have the attribute of "wearing glasses" from among the users. Identify. Then, the information providing device 10 inputs the user information of the specified user into the model, and if the output information of the model is common, even if the "person wearing glasses" is specified as an estimation index of the model. good.

[1-4-4. Index estimation for each dimension]
Here, when the model is trained so as to perform other class classification, the model may output a multidimensional quantity such as a vector. In such a case, it is not always clear what kind of user information the value of each dimension corresponds to. Therefore, the information providing device 10 may estimate the correlation with the user information for each value of each dimension. That is, the information providing device 10 identifies a plurality of classification targets for which vectors having the same value of a predetermined dimension are output from the target information by a model that outputs a vector as output information, and sets the target information of the specified classification target. Based on this, the index corresponding to the value of a predetermined dimension in the vector output by the model may be estimated.

For example, the information providing device 10 generates a vector to be output information from the user information of each user, and identifies the user information corresponding to the vector having the same value in a predetermined dimension. Then, the information providing device 10 specifies a type having commonality in the specified user information, and identifies that the specified type is a type corresponding to a value of a predetermined dimension in the vector to be output information. Information indicating the correspondence between the specified type and a predetermined dimension may be used as a classification index. Further, the information providing device 10 may obtain a correlation with the user information for each set of each dimension.

[1-4-5. Evaluation using models]
A plurality of models registered by a plurality of businesses are registered in the information providing device 10. Therefore, the information providing device 10 may estimate the estimation index of each model by comparing the user information of the classified users classified by different models. For example, the information providing device 10 is based on a comparison result between the user information of the first group classified using the model to be estimated and the user information of the second group classified using another model. The estimation index of the model to be estimated may be specified.

[2. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the information providing device 10 described above will be described. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 3, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the external terminals 100 and 200.

The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. In addition, the storage unit 30 stores the user database 31 and the model database 32. Hereinafter, an example of information registered in the user database 31 and the model database 32 will be described with reference to FIGS. 4 and 5.

Various information about the user is registered in the user database 31. For example, FIG. 4 is a diagram showing an example of a user database according to the embodiment. In the example shown in FIG. 4, in the user database 31, "user ID" and "user information" are registered in association with each other. Further, in the user database 31, types of information such as "attribute information", "search history", and "purchase history" are registered as "user information". In addition to the information shown in FIG. 4, various types of user information such as the history of advertisements selected by the user and the history of contents viewed by the user are registered in the user database 31. It may be, and other information about the user may be registered.

Here, the "user ID" is an identifier that identifies the user. Further, the "attribute information" is various attribute information such as a user's demographic attribute and psychographic attribute. The "search history" is a history of search queries entered by the user. The "purchase history" is a history of transaction targets purchased by the user. For example, in the example shown in FIG. 4, the user database 31 has the user ID "U1", the attribute information "attribute information # U1", the search history "search history # U1", and the purchase history "purchase history # U1". Are registered in association with each other. Such information is the user information of the user indicated by the user ID "U1", the attribute indicated by the attribute information "attribute information # U1", the history of the search query indicated by the search history "search history # U1", and the history of the search query indicated by the search history "search history # U1". Purchase history Indicates that the transaction target history and the like indicated by "purchase history # U1" are registered.

In the example shown in FIG. 4, conceptual values such as "attribute information # U1", "search history # U1", and "purchase history # U1" are described, but in reality, various attributes and search queries are described. Information such as the history of the transaction and the character string or numerical value indicating the transaction target will be registered.

Various information about the model is registered in the model database 32. For example, FIG. 5 is a diagram showing an example of a model database according to an embodiment. In the example shown in FIG. 5, in the model database 32, information such as "model ID", "registrant", "learning data", "classification purpose", and "estimated index" are registered in association with each other. In addition to the information shown in FIG. 5, various information related to the model, such as the registration date and time of the model, may be registered in the model database 32. Further, in the example shown in FIG. 5, although the description is omitted, the data constituting the DNN such as the connection relationship of the nodes and the connection coefficient is registered in the model database 32 as the data of the model itself.

Here, the "model ID" is an identifier for identifying the model. The "registrant" is an identifier for identifying the business operator who registered the model. Further, the "learning data" is user information or output information used for learning the model indicated by the associated "model ID", that is, information indicating learning data. Further, the "classification purpose" is information indicating the purpose of classification performed by the model indicated by the associated "model ID". For example, the "classification purpose" is information indicating what kind of output information is output when learning is performed when what kind of user information is input. The "classification information" may be, for example, information indicating which data of the learning data is a positive example and which data is a negative example. Further, the "estimated index" is an estimated index estimated for the model indicated by the associated "model ID".

For example, in the example shown in FIG. 5, the model database 32 contains the model ID “M1”, the registrant “business operator # 1”, the learning data “learning data # M1”, the classification purpose “classification purpose # M1”, and the estimation. The index "estimated index # M1" is registered in association with each other. Such information is a model in which the model indicated by the model ID "M1" is registered by the operator indicated by the registrant "business operator # 1", and is learned by the learning data indicated by the learning data "learning data # M1". Indicates that Further, such information indicates that the model indicated by the model ID "M1" is learned for the purpose of classification purpose "classification purpose # M1" and the estimated estimation index is "estimation index # M1". show.

In the example shown in FIG. 5, conceptual values such as "business operator # 1", "learning data # M1", "classification purpose # M1", and "estimated index # M1" are described, but in reality, they are described. , The model database 32 contains a character string or numerical value for identifying a business operator, a set of information indicating learning data or user information serving as learning data and corresponding output information, a character string indicating a classification purpose, or a positive value. An identifier indicating learning data as an example or a negative example, a character string indicating an estimation index, and the like will be registered.

The explanation will be continued by returning to FIG. The control unit 40 is, for example, a controller, and is an example of various programs (notification programs) stored in a storage device inside the information providing device 10 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Is realized by executing RAM as a work area. Further, the control unit 40 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 40 includes a reception unit 41, a generation unit 42, a specific unit 43, an estimation unit 44, and a providing unit 45. The reception unit 41 accepts model registration. For example, the reception unit 41 receives a model generated by a business operator from an external terminal 100, learning data used for learning, and a classification purpose. In such a case, the reception unit 41 registers the model data in the model database 32 in association with the information indicating the business operator, the information indicating the learning data, and the classification purpose. The model to be registered may be one learned by using the user information collected by the business operator, and the business operator has learned by using the user information managed by the information providing device 10. It may be a model.

In addition, the reception unit 41 accepts the registration of user information. For example, the reception unit 41 acquires user information from various log servers and the like, and registers the acquired user information in the user database 31.

The generation unit 42 generates a model. For example, when the generation unit 42 receives the classification purpose from the external terminal 200, the generation unit 42 refers to the model database 32 and searches for a model having the same classification purpose. Then, when there is a model having the same classification purpose, the generation unit 42 provides the searched model to the external terminal 200. The generation unit 42 receives the user information of the user to be classified from the external terminal 200, classifies the user from the received user information using the searched model, and displays only the classification result in the external terminal 200. May be provided to. That is, the generation unit 42 may provide only the result of the classification process using the model to the business operator while keeping the model having the same classification purpose secret. Further, when the generation unit 42 receives the user's designation from the external terminal 200, the generation unit 42 identifies the user information of the designated user from the user information managed by the information providing device 10, and the specified user. Users may be classified by inputting information into the model.

On the other hand, the generation unit 42 generates a model when there is no model having the same classification purpose. For example, the generation unit 42 provides the classification purpose to a predetermined model creator such as a data scientist, and specifies a set of user information and output information used for learning the model, that is, designation of training data. Accept. For example, when the purpose of classification is "whether or not to select a predetermined advertisement", the generation unit 42 uses the user information of the user who has selected the predetermined advertisement from the model creator as a correct example, and determines the predetermined advertisement. User information of users who have not selected advertisements will be accepted as a negative example. The generation unit 42 may automatically select such a positive example and a negative example by using various character analysis techniques and the like. Then, the generation unit 42 generates the model by letting the model learn the characteristics of the user information, and registers the generated model in the model database 32. The generation unit 42 may use the model generated in this way for use by the business operator.

When the target information regarding the classification target is input, the identification unit 43 identifies a plurality of classification targets for which similar output information is output from the target information by a model that outputs output information indicating the classification of the input classification target. .. For example, the specific unit 43 selects a plurality of users for which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target when the target information regarding the user to be classified is input. Identify. More specifically, the specific unit 43 outputs output information indicating the classification of the user when the attribute information indicating the attribute of the user or the action information indicating the behavior of the user is input as the target information regarding the user. Identify multiple users for whom similar output information is output from attribute information or behavior information.

For example, the specific unit 43 refers to the model database 32 at a predetermined timing (for example, at regular intervals) and selects a model in which an estimation index is not registered. In such a case, the specifying unit 43 specifies the type of user information to be input to the model from the learning data of the selected model and the like. Then, the specific unit 43 acquires the output information for each user by inputting the specified type of information among the user information registered in the user database 31 into the selected model.

Subsequently, the specifying unit 43 compares the output information with each other, and identifies users having similar output information, that is, users classified into the same group by the model as classified users. For example, the identification unit 43 may specify a user whose output information difference is within a predetermined range as a classification user. That is, the specific unit 43 selects a plurality of classification targets to which similar output information is output by a model trained to output output information according to the characteristics of a predetermined type of information among the target information. It will be specified.

Note that the specific unit 43 may execute arbitrary processing as long as the users classified into the same group according to the model are classified users. For example, when a vector is output as output information, the specific unit 43 calculates an arbitrary similarity of the output information such as an Euclidean distance, a Hamming distance, and a cosine distance between the vectors, and the calculated similarity is within a predetermined range. The user corresponding to the vector group that fits within may be a classified user. Further, the specific unit 43 may use, for example, a user corresponding to a vector group in which a value of a predetermined dimension falls within a predetermined range among the vectors as a classification user.

Further, when estimating the classification index based on the comparison of the user information of different groups, the specifying unit 43 identifies the users whose user information has a predetermined similarity as the comparative user. For example, the specific unit 43 refers to the user database 31 and refers to the user information of each user. Then, the identification unit 43 identifies a group of users whose user information satisfies a predetermined condition as comparative users. For example, in the specific unit 43, the user information of the type corresponding to the classification purpose of the model to be estimated by the classification index is a user who satisfies the condition corresponding to the classification purpose (for example, "gender" is "female"). Identify users and users whose "attribute information" includes "like ramen") as comparative users. That is, the identification unit 43 is a model different from the model, and identifies the classification targets classified into the same group by the model that classifies the classification targets according to whether or not the target information satisfies a predetermined condition.

The estimation unit 44 estimates an index when the model classifies the users based on the user information of the identified classification users. That is, the estimation unit 44 estimates an index when the model classifies the classification target based on the target information of the classification target classified into the same group by the model. For example, the estimation unit 44 estimates an index when the model classifies the users based on the commonality of the user information of the classified users. For example, the estimation unit 44 refers to the user information of the classified users and identifies the information in which the ratio of common users exceeds a predetermined threshold value. To give a more specific example, when the user information of all the classified users includes information such as "I like ramen", the estimation unit 44 uses the information "I like ramen" as an estimation index. do.

The estimation unit 44 may estimate the classification index based on the commonality of the information of the types other than the type in which the model has learned the characteristics among the user information of the classification users. For example, the estimation unit 44 estimates the type of user information that the model is presumed to have learned the features from the purpose of classifying the model. For example, the estimation unit 44 identifies the user information used in the learning data, and specifies the type of user information having the highest commonality among the information included in the specified user information as the learning type. Further, the estimation unit 44 calculates the commonality for each type or group of types of user information of the classification user, and whether or not there is a type or group of types having a higher commonality than the commonality of the learning types. Is determined. Then, when the estimation unit 44 has a type or a set of types having a commonality higher than the commonality of the learning types, the type or a set of the types having the highest commonality may be used as an estimation index. Further, the estimation unit 44 may estimate that the model is a model in which appropriate learning has been performed for the purpose of classification when the learning types have the highest commonality.

Further, the estimation unit 44 may estimate an index when the model classifies the classification target based on the comparison between the user information of the classification user and the user information of the comparison user. For example, when the commonality of the user information of the classified user and the commonality of the user information of the comparative user are similar, the estimation unit 44 uses the rule used when identifying the comparative user. It may be used as an estimation index. Further, when the commonality of the user information of the classification user and the commonality of the user information of the comparison user are not similar, the estimation unit 44 sets a feature other than the feature assumed for the purpose of classification by the model. It may be determined that it has been learned.

Further, the estimation unit 44 may estimate an index when the model classifies the classification target based on the correlation between the user information of the classification user and the output information. For example, the estimation unit 44 digitizes each user information and output information of the classified users, plots them in individual axial directions, and based on the plot results, the correlation with the output information of the user information is determined. The highest type or set of types may be specified. Further, the estimation unit 44 specifies the correlation between the user information of the classified user and the output information, specifies the correlation between the user information of the comparative user and the output information, and resembles the specified correlation. Classification indicators may be estimated based on gender. For example, if the correlations are not similar, the estimation unit 44 may determine that the model has learned features other than those assumed for the purpose of classification.

Further, the estimation unit 44 identifies the type of user information that has a high correlation with each dimension by plotting the user information and the value of each dimension of the vector that becomes the output information in the individual axial directions. You may. Further, the estimation unit 44 estimates the type and type set of the user information corresponding to each dimension as a classification index by specifying the correlation between the predetermined dimension of the vector to be the output information and the user information. You may.

The estimation unit 44 will register the estimated classification index as an estimation index in the model database 32.

The providing unit 45 provides the estimation index to the business operator. For example, the providing unit 45 may provide an estimation index of the model to a business operator who wants to use the model. Further, when the estimation index and the classification purpose do not correspond to each other, the providing unit 45 may provide the estimation index to the business operator who generated the model.

[3. Processing procedure]
Next, an example of the flow of the estimation process by the information providing device 10 according to the embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the flow of the estimation process according to the embodiment.

For example, the information providing device 10 selects a model to be processed (step S101), and uses the selected model to generate output information from the user information of each user (step S102). Then, the information providing device 10 identifies users having similar generated output information as users belonging to the first group, that is, classified users (step S103). Subsequently, the information providing device 10 identifies a user whose user information satisfies a predetermined condition as a user belonging to the second group, that is, a comparative user (step S104). Then, the information providing device 10 estimates an index when the selected model classifies the user information based on the comparison result between the user information of the first group and the user information of the second group (step S105). ), End the process.

[4. Modification example]
The information providing device 10 described above may be implemented in various different forms other than the above-described embodiment. Therefore, another embodiment of the information providing device 10 will be described below.

[4-1. Applicable target]
In the above example, the information providing device 10 estimates the estimation index of the model for classifying the users based on the user information. However, the embodiment is not limited to this. For example, the information providing device 10 may estimate the above-mentioned estimation index for a model that classifies an arbitrary classification target such as advertisement classification and various contents distributed to users. For example, the information providing device 10 targets an advertisement as a classification target, targets the type, name, price, manufacturer, seller, etc. of the transaction target to be the advertisement, and preferentially uses the advertisement from the input target information. The above-mentioned estimation process may be performed on the model for estimating the person.

Further, the information providing device 10 uses the information corresponding to the classification target as the target information, and if the classification index is estimated from the target information of the classification target classified in the same group, an arbitrary number of type information is used as the target information. However, information of arbitrary contents may be used as the target information. For example, the information providing device 10 includes a user's gender, demographic attribute, psychographic attribute, search query history, behavior history on various networks, posting history which is a history of information posted by the user, purchase history, and the like. Arbitrary information about the user may be adopted as the user information.

[4-2. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically dispersed / physically distributed in any unit according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[4-3. program〕
Further, the information providing device 10 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. 7. FIG. 7 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Has.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various calculations, such as a RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, and a scanner, and is realized by, for example, USB.

The input device 1020 includes, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[5. effect〕
As described above, the information providing device 10 outputs similar output information from the target information by the model that outputs the output information indicating the classification of the input classification target when the target information regarding the classification target is input. Identify multiple classification targets. Then, the information providing device 10 estimates an index when the model classifies the classification target based on the target information of the specified plurality of classification targets. As a result of such processing, the information providing device 10 can estimate the classification index that the model has really learned, so that it can provide information indicating the characteristics that the model has learned.

Further, the information providing device 10 estimates an index when the model classifies the classification target based on the commonality of the target information regarding the specified classification target. Therefore, the information providing device 10 can estimate the information that the model has learned the features.

Further, the information providing device 10 is a plurality of models to which similar output information is output by a model trained to output output information according to the characteristics of a predetermined type of information among the target information. The index for classifying the classification target by the model is estimated based on the commonality of the information of the types other than the predetermined type among the target information of the specified classification target. Therefore, the information providing device 10 can estimate whether or not the model is learning a feature that is not assumed by the business operator or the like who has learned the model.

Further, the information providing device 10 specifies the classification target for which similar output information is output from the target information by the model as the first group, and also identifies the classification target for which the target information has a predetermined similarity as the second group. , The index when the model classifies the classification target is estimated based on the comparison between the target information of the classification target belonging to the first group and the target information of the classification target belonging to the second group. For example, the information providing device 10 is a second group, which is a model different from the model, and is classified into the same group by a model that classifies the classification target according to whether or not the target information satisfies a predetermined condition. Identify the target. Therefore, the information providing device 10 can provide information indicating the features learned by the model more accurately.

Further, the information providing device 10 estimates an index when the model classifies the classification target based on the correlation between the specified target information of the classification target and the output information output by the model. For example, the information providing device 10 uses, among the target information, the type of information whose correlation with the output information satisfies a predetermined condition as an index when the model classifies the classification target. Further, the information providing device 10 uses a set of information types whose correlation with the output information output by the model satisfies a predetermined condition as an index when the model classifies the classification target. Therefore, the information providing device 10 can accurately estimate the features learned by the model.

Further, the information providing device 10 identifies a plurality of classification targets for which vectors having the same value of a predetermined dimension are output from the target information by a model that outputs a vector as output information, and uses the specified classification target information as the target information. Based on this, the index corresponding to the value of a predetermined dimension in the vector output by the model is estimated. Therefore, the information providing device 10 can estimate which target information of the target information corresponds to each dimension of the vector that becomes the output information.

Further, the information providing device 10 is a plurality of users for which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target when the target information regarding the user to be classified is input. And estimate the indicators by which the model classifies users. For example, the information providing device 10 uses a model that outputs output information indicating the classification of the user when the attribute information indicating the attribute of the user or the action information indicating the behavior of the user is input as the target information regarding the user. , Identifies multiple users for whom similar output information is output from attribute information or behavioral information. Therefore, the information providing device 10 can estimate what kind of features the user information has learned by the model.

The embodiments of the present application have been described in detail with reference to the drawings, but this is an example, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other forms described above.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the control unit can be read as a control means or a control circuit.

10 Information providing device 20 Communication unit 30 Storage unit 31 User database 32 Model database 40 Control unit 41 Reception unit 42 Generation unit 43 Specific unit 44 Estimating unit 45 Providing unit 100, 200 External terminal

Claims (13)

When the target information related to the classification target is input, a specific part that identifies a plurality of classification targets for which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target, and
An estimation device, characterized in that the model has an estimation unit that estimates an index for classifying the classification object based on target information of a plurality of classification objects specified by the specific unit. The first aspect of claim 1, wherein the estimation unit estimates an index when the model classifies the classification object based on the commonality of target information regarding the classification object specified by the specific unit. Estimator. The specific unit is a plurality of models to which similar output information is output by the model in which learning is performed so as to output output information according to a feature of a predetermined type of information among the target information. Identify the classification target of
The estimation unit estimates an index when the model classifies the classification target based on the commonality of information of a type other than the predetermined type among the target information of the classification target specified by the specific unit. The estimation device according to claim 1 or 2. The specific unit specifies a classification target for which similar output information is output from the target information by the model as a first group, and also identifies a classification target for which the target information has a predetermined similarity as a second group. ,
The estimation unit estimates an index when the model classifies the classification target based on the comparison between the target information of the classification target belonging to the first group and the target information of the classification target belonging to the second group. The estimation device according to claim 1, wherein the estimation device is used. As the second group, the specific unit is a model different from the model, and is classified into the same group by a model that classifies the classification target according to whether or not the target information satisfies a predetermined condition. The estimation device according to claim 4, wherein the target is specified. The estimation unit estimates an index when the model classifies the classification target based on the correlation between the target information of the classification target specified by the specific unit and the output information output by the model. The estimation device according to any one of claims 1 to 5. The claim is characterized in that the estimation unit uses, among the target information, the type of information whose correlation with the output information satisfies a predetermined condition as an index when the model classifies the classification target. 6. The estimation device according to 6. The claim is characterized in that the estimation unit uses a set of information types whose correlation with the output information output by the model satisfies a predetermined condition as an index when the model classifies the classification target. 7. The estimation device according to 7. The specific unit identifies a plurality of classification targets for which vectors having the same value of a predetermined dimension are output from the target information by a model that outputs a vector as the output information.
The claim is characterized in that the estimation unit estimates the index corresponding to the value of the predetermined dimension among the vectors output by the model based on the target information of the classification target specified by the specific unit. The estimation device according to any one of 1 to 8. The specific unit outputs output information indicating the classification of the classification target when the target information regarding the user to be classified is input. A plurality of users for which similar output information is output from the target information by the model. Identify and
The estimation device according to any one of claims 1 to 9, wherein the estimation unit estimates an index when the model classifies the users. When the attribute information indicating the attribute of the user or the action information indicating the behavior of the user is input as the target information regarding the user, the specific unit outputs output information indicating the classification of the user. The estimation device according to claim 10, wherein a plurality of users who have output similar output information from the attribute information or the behavior information are specified by a model. It is an estimation method performed by the estimation device.
When the target information related to the classification target is input, a specific process for identifying a plurality of classification targets for which similar output information is output from the target information by a model that outputs output information indicating the classification of the classification target, and
An estimation method comprising an estimation step of estimating an index when the model classifies the classification target based on the target information of a plurality of classification targets specified by the specific step. When the target information related to the classification target is input, the specific procedure for identifying multiple classification targets for which similar output information is output from the target information by the model that outputs the output information indicating the classification of the classification target, and
An estimation program for causing a computer to perform an estimation procedure for estimating an index when the model classifies the classification object based on the target information of a plurality of classification objects specified by the specific procedure.

Images