JP7722665B2 - Method for training a learning model for estimating advertising effectiveness, method for estimating advertising effectiveness, and program - Google Patents

Description

The present invention relates to technology for estimating advertising effectiveness, and more specifically to technology for improving the accuracy of ranking advertising effectiveness.

Advertisements on advertising media such as Google (registered trademark) and Yahoo! (registered trademark) have a hierarchical structure, and are created in the following order, starting from the highest level: 1: Account, 2: Campaign, 3: Ad Group, 4: Ad.

Advertising media such as Google (registered trademark) and Yahoo! (registered trademark) have an optimization function that prioritizes the delivery of ads within an ad group that are expected to perform better than other ads. This function allows ads with statistically better performance to be delivered first as data accumulates. However, because this optimization function relies on the accumulation of data, there is a problem in that ads with poor performance may be delivered until sufficient data has been accumulated. Against this background, technology has been developed to predict the effectiveness of ad delivery in order to prioritize ads that are expected to perform better than other ads within an ad group. Indicators of ad delivery effectiveness include the number of impressions, number of clicks, click-through rate, and delivery amount.

For example, Japanese Patent Application Laid-Open Publication No. 2021-182340 (Patent Document 1) and "Estimating Delivery Priority of Ad Creatives Using Rank Learning" (Non-Patent Document 1) propose technology for determining delivery priority for ad creators by predicting the average daily delivery amount, which is one of the indicators of the quality of ad creations, before delivery. Specifically, based on the idea that determining delivery priority does not require strict estimation of the actual numerical value of the effectiveness of ads, but rather requires only knowing the order of advertising effectiveness for a given group of ads, a method is proposed that uses rank learning technology to recommend ad creations in ascending order of quality. Note that this method calculates the score of ad creatives using pairwise rank learning.

Japanese Patent Application Laid-Open No. 2021-182340

"Estimating the Delivery Priority of Ad Creatives Using Rank Learning" (Yuki Iwasaki, Kazuteru Taniguchi: Proceedings of the 2020 National Conference of the Japanese Society for Artificial Intelligence, Vol. JSAI2020, 34th (2020), Session ID: 1H4-OS-12b-04, Published: June 19, 2020)

Prediction models for ad delivery effectiveness using conventional methods such as those described above learn the rankings of all ads in a dataset. However, in advertising operations, it is not necessary to know the relative merits of all ads; it is sufficient to determine the relative merits of some of the ads that you want to compare. Conditions for "some of the ads you want to compare" include delivery settings such as ad groups, image sizes, and user attributes. For example, if you can determine the relative merits of ads that are included in the same ad group and have the same image size, it becomes possible to determine the relative merits of ad images displayed in similar positions within the ad group, and decide which ad images to deliver.

In this way, to determine the advertising images to be delivered, it is only necessary to compare the merits and demerits of a portion of the advertisements that one wishes to compare. However, the methods described in Patent Document 1 and Non-Patent Document 1 determine the merits and demerits of all advertisements. As is commonly known as Simpson's paradox, the correlation in a population does not match the correlation in groups into which the population is divided. For this reason, conventional methods that learn the "merits and demerits of all advertisements" have the problem of low prediction accuracy for the "merits and demerits of the delivery effectiveness of a portion of the advertisements that one wishes to compare."

For example, consider the example of distribution effectiveness between advertisements shown in Figure 1. This figure is intended to simply explain an example in which the correlation between feature values and advertising effectiveness in the total population does not match the correlation between feature values and advertising effectiveness in groups obtained by dividing the total population.

Figure 1(a) shows four example advertisement images (advertisement IDs 1 to 4), and Figure 1(b) is a scatter plot showing the correlation between the features of these four advertisement images and the effectiveness of the advertisements. Advertisement IDs 1 and 2 are advertisement images aimed at selling "cars," and belong to the same group (group A) in the sense that they both sell "cars." On the other hand, advertisement IDs 3 and 4 are advertisement images aimed at selling "houses," and belong to the same group (group B) in the sense that they both sell "houses." In this example, the effectiveness of the advertisements is the click-through rate (CTR), and the feature of the advertisements is the font size of the characters. Within group A, the feature (font size) of the advertisement with advertisement ID 2 is larger than that with advertisement ID 1, and within group B, the feature (font size) of the advertisement with advertisement ID 4 is larger than that with advertisement ID 3.

In this example, in group A, ad ID 2, which has larger features than ad ID 1, has a higher CTR, and in group B, ad ID 4, which has larger features than ad ID 3, has a higher CTR, so there is a positive correlation between font size and CTR within the same group. On the other hand, when comparing ad ID 2 and ad ID 3, which belong to different ad groups, ad ID 3, which has smaller ad features, has a higher CTR than ad ID 2, which has larger ad features, so there is a negative correlation between font size and CTR for the four ad images as a whole. In this way, if learning is done as a whole without taking ad groups into consideration, the opposite correlation (negative correlation) will be learned instead of the positive correlation that we originally wanted to learn, resulting in a problem of reduced rank prediction accuracy within the group we want to rank.

In light of these problems, the inventors came up with the idea that performing rank learning for each ad group would make it possible to accurately estimate the quality of ad creations, and thus developed the present invention.

In order to solve the above problem, a learning method for a learning model for estimating advertising effectiveness according to the present invention is characterized by including the following steps A to E.
Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data;
Step B: grouping all advertisements included in the plurality of advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements;
Step C: selecting one rank learning group from the plurality of rank learning groups;
Step D: calculating a loss function value for the selected one rank learning group;
Step E: A step of ranking learning in which rank learning is performed within the selected one rank learning group, while rank learning between different rank learning groups is not performed.

In one aspect, steps D and E are steps of performing rank learning and correcting the learning model so as to minimize the loss function value of the selected rank learning group.

In a preferred embodiment, the method further includes step F, in which steps C to E are performed for all of the plurality of rank learning groups, and the execution of step F corrects the learning model for all of the learning data.

In one aspect, step F includes a substep of setting the sum of the loss function values of each of the rank learning groups as the loss function value of the learning model.

For example, the conditions characterizing the advertisement may include at least one of the following: delivery settings, image size, delivery date and time, and ad group.

Also, for example, the loss function value is calculated using a pairwise or listwise loss function.

The advertising effectiveness estimation method of the present invention includes the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are yet to be delivered, based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for advertising effectiveness estimation of the present invention; and determining a delivery priority based on the calculated score for each advertisement.

Furthermore, the program according to the present invention is a program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, the method being characterized by including the following steps A to E:
Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data;
Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements;
Step C: selecting one rank learning group from the plurality of rank learning groups;
Step D: calculating a loss function value for the selected one rank learning group;
Step E: A step of ranking learning in which rank learning is performed within the selected one rank learning group, while rank learning between different rank learning groups is not performed.

In one aspect of the program according to the present invention, steps D and E are steps of performing rank learning and correcting the learning model so as to minimize the loss function value of the selected rank learning group.

In a preferred embodiment, the method further includes step F, in which steps C to E are performed for all of the plurality of rank learning groups, and the execution of step F corrects the learning model for all of the learning data.

In one aspect, step F includes a substep of setting the sum of the loss function values of each of the rank learning groups as the loss function value of the learning model.

For example, the conditions characterizing the advertisement may include at least one of the following: delivery settings, image size, delivery date and time, and ad group.

Also, for example, the loss function value is calculated using a pairwise or listwise loss function.

Furthermore, the program of the present invention is a program for causing a computer to execute a method for estimating advertising effectiveness, the method including the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are yet to be delivered, based on conditions characterizing the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness of the present invention; and determining a delivery priority based on the calculated score for each advertisement.

In this invention, rank learning to estimate advertising effectiveness is performed for each group to be ranked, thereby improving the accuracy of rank prediction within the group to be ranked.

FIG. 10 is a diagram for simply explaining an example in which the correlation between the feature amount and the advertising effectiveness in a population does not match the correlation between the feature amount and the advertising effectiveness in a population divided from the population. FIG. 2 is a block diagram for conceptually explaining the hierarchical structure of advertisements. FIG. 10 is a diagram showing an example of a plurality of advertising accounts that constitute learning data to be acquired. FIG. 10 is a diagram conceptually illustrating an example in which all advertisements contained in multiple advertising accounts acquired as learning data are grouped into multiple rank learning groups according to conditions that characterize the advertisements. 1 is a flowchart for explaining an outline of processing executed in a learning method for a learning model for estimating advertising effectiveness according to the present invention. 10 is a flowchart illustrating an example of a procedure for completing one learning model for all acquired advertising data. 1 is a flowchart illustrating an example of ranking prediction using a learning method for a learning model for estimating advertising effectiveness according to the present invention.

Below, with reference to the drawings, we will explain the learning method for a learning model for estimating advertising effectiveness, the method for estimating advertising effectiveness, and the program related to the present invention.

Figure 2 is a block diagram that conceptually explains the hierarchical structure of advertising. As shown in this diagram, advertising has a hierarchical structure, where an "advertising account" is a collection of "campaigns," each "campaign" is a collection of "ad groups," and each "ad group" is a collection of "ads." An "advertisement" is an image that is actually displayed on advertising media, and one from the "ad groups" that is considered appropriate for the media is displayed.

In the example shown in this diagram, the "Ad Account" is a collection of two "Campaigns," with "Campaign 1" being a collection of "Ad Group A" and "Ad Group B," "Ad Group A" being a collection of "Ad a," "Ad b," and "Ad c," and "Ad Group B" being a collection of "Ad d," "Ad e," and "Ad f." The same is true for "Campaign 2," with "Campaign 2" being a collection of "Ad Group C" and "Ad Group D," "Ad Group C" being a collection of "Ad g," "Ad h," and "Ad i," and "Ad Group D" being a collection of "Ad j," "Ad k," and "Ad l."

In the training method for a learning model for estimating advertising effectiveness according to the present invention, first, all advertising data (advertising data) contained in multiple advertising accounts consisting of one or more advertising groups is obtained as training data.

Figure 3 shows an example of one or more advertising accounts (data) that make up the learning data to be acquired. In the example shown in this figure, data from all advertisements contained in each of the advertising accounts of companies A to N will be acquired as learning data.

In the learning method for a learning model for estimating advertising effectiveness according to the present invention, all advertisements contained in one or more advertising accounts acquired as learning data are grouped into multiple rank learning groups based on conditions characterizing the advertisements.

Figure 4 is a conceptual diagram showing an example of grouping all advertisements contained in one or more advertising accounts acquired as learning data into multiple rank learning groups based on conditions that characterize the advertisements. In the example shown in this figure, data on all advertisements contained in each advertising account of companies A to N is acquired as learning data, and this learning data is grouped into four "learning groups" (A to D), with each "learning group" being a collection of multiple "advertisements."

Figure 5 is a flowchart outlining the processing executed in the learning method for a learning model for estimating advertising effectiveness according to the present invention. As shown in this figure, as described above, in step S101, all advertising data is acquired as learning data (dataset). Then, in step S102, the acquired learning data is divided into multiple rank learning groups based on conditions characterizing the advertisements.

Next, one rank learning group is selected from the multiple rank learning groups (step S103), a loss function value is calculated for the selected rank learning group (step S104), and rank learning is performed within the selected rank learning group based on this loss function value (step S105), and the learning model is corrected (step S106).

In the present invention, the rank learning in step S105 above is performed within one selected rank learning group, while rank learning between different rank learning groups is not performed. As a result, the superiority or inferiority of advertisements within the same ad group is learned, but the superiority or inferiority of advertisements between different ad groups is not learned, thereby resolving the above-mentioned "Simpson's Paradox" problem.

In the example shown in Figure 5, rank learning in step S105 is performed to correct the learning model so as to minimize the loss function value of one selected rank learning group.

As such, the training method for a learning model for estimating advertising effectiveness according to the present invention includes the steps of: acquiring data on all advertisements included in one or more advertising accounts consisting of one or more advertising groups as training data (Step A); grouping all advertisements included in the one or more advertising accounts into multiple rank learning groups based on conditions characterizing the advertisements (Step B); selecting one rank learning group from the multiple rank learning groups (Step C); calculating a loss function value for the selected rank learning group (Step D); and performing rank learning within the selected rank learning group while not performing rank learning between different rank learning groups (Step E). In Step E, rank learning is performed within the selected rank learning group, but rank learning between different rank learning groups is not performed. By adopting this configuration, the superiority or inferiority of advertisements within the same advertising group is learned, but the superiority or inferiority of advertisements between different advertising groups is not learned, thereby resolving the problem of "Simpson's Paradox."

As mentioned above, in one aspect of the present invention, rank learning may be performed to minimize the loss function value of a selected rank learning group, thereby correcting the learning model.

By modifying the learning model as described above, it is theoretically possible to optimize the learning model for a single selected rank learning group. However, if the single selected rank learning group does not contain a sufficiently large amount of training data, the learning model that is supposed to have been "optimized" may not be optimal. Furthermore, a learning model that is "optimized" for a single selected rank learning group is generally not "optimized" for the entire dataset obtained as training data.

Therefore, in the learning method for a learning model for estimating advertising effectiveness according to the present invention, in order to obtain an "optimized" learning model for the entire data set acquired as learning data, the following steps can be included in addition to the steps shown in Figure 5, thereby completing an "optimized" learning model for the entire acquired data set, i.e., a single learning model for all acquired advertising data.

Figure 6 is a flowchart illustrating an example of the procedure for completing a single learning model for all acquired advertising data.

After "learning within the selected learning group" is completed using the procedure shown in Figure 5, another learning group is selected (step S201). Then, a loss function value within this selected learning group is calculated (step S202), and based on this loss function value, rank learning is performed within the selected rank learning group (step S203), and the learning model is modified (step S204). Note that in the example shown in Figure 6, rank learning in step S203 is performed to modify the learning model so as to minimize the loss function value of the selected rank learning group.

Following step S204, it is determined whether "an unprocessed learning group exists" (step S205), and if "an unprocessed learning group exists" (Yes), the processing from step S201 onwards is repeated. If "an unprocessed learning group does not exist" (No), the learning of the learning model is terminated.

As such, in order to obtain a learning model that is "optimized" for the entire data set obtained as learning data, the learning method for learning models for advertising effectiveness estimation according to the present invention preferably includes a step (step F) of performing steps C to E described above for all of the multiple rank learning groups into which the learning model has been grouped, and performing step F may modify the learning model for the entire learning data. Even if the learning model obtained in this way is not "optimized" for each individual "rank learning group," it is "optimized" for the entire data set obtained as learning data, and there is no problem that the supposedly "optimized" learning model may not be optimal if there is not enough learning data contained in a selected rank learning group.

Note that step F described above may include a substep of setting the sum of the loss function values of each rank learning group as the loss function value of the learning model.

In the present invention, the conditions that characterize an advertisement include, for example, at least one of the following: distribution settings, image size, distribution date and time, and advertisement group.

Furthermore, in the present invention, the loss function value is calculated, for example, using a pairwise or listwise loss function.

By using this learning method for a learning model for estimating advertising effectiveness according to the present invention, advertising effectiveness can be estimated (ranking prediction).

Figure 7 is a flowchart illustrating an example of ranking prediction using the learning method for the learning model for advertising effectiveness estimation according to the present invention.

First, create a ranking group for which you want to rank advertising effectiveness (step S301), and then calculate the score of each advertisement in the ranking group using the trained learning model (step S302). Then, predict the ranking based on the calculated score and determine (rank) the delivery priority (step S303).

In this way, the advertising effectiveness estimation method of the present invention is a method for estimating advertising effectiveness that includes the steps of creating a ranking group consisting of multiple advertisements that have been delivered or are yet to be delivered, based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for advertising effectiveness estimation of the present invention; and determining the delivery priority based on the calculated score for each advertisement.

The method of training a learning model for estimating advertising effectiveness according to the present invention can be executed by a computer using a program.

That is, the program according to the present invention is a program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, the method being characterized by including the following steps A to E.
Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data;
Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements;
Step C: selecting one rank learning group from the plurality of rank learning groups;
Step D: calculating a loss function value for the selected one rank learning group;
Step E: A step of ranking learning in which rank learning is performed within the selected one rank learning group, while rank learning between different rank learning groups is not performed.

In one aspect of the program according to the present invention, steps D and E are steps of performing rank learning and correcting the learning model so as to minimize the loss function value of the selected rank learning group.

In a preferred embodiment, the method further includes step F, in which steps C to E are performed for all of the plurality of rank learning groups, and the execution of step F corrects the learning model for all of the learning data.

In one aspect, step F includes a substep of setting the sum of the loss function values of each of the rank learning groups as the loss function value of the learning model.

For example, the conditions characterizing the advertisement may include at least one of the following: delivery settings, image size, delivery date and time, and ad group.

Also, for example, the loss function value is calculated using a pairwise or listwise loss function.

Furthermore, the program of the present invention is a program for causing a computer to execute a method for estimating advertising effectiveness, the method including the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are yet to be delivered, based on conditions characterizing the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness of the present invention; and determining a delivery priority based on the calculated score for each advertisement.

Below, in the following examples, we will verify the effectiveness of rank learning that takes into account the structure of advertisements according to the present invention by comparing it with rank learning that does not take into account the structure of advertisements. The average rank correlation coefficient was used as an indicator to evaluate effectiveness.

[Create data]
The data acquired as learning data includes the "impressions,""imagesize" (image width (image_width) and image height (image_height)), "date,""image" (image_id), and "ad group ID (ad_group_id)" of each advertisement.

Ten months' worth of data was extracted from the accumulated data and data was created that took into account the ad structure. The specific procedure is as follows: Of the 10 months' worth of data, the first eight months' worth of data were used as training data, and the following two months' worth of data were used as test data, and the data was divided into training data and test data. This data was then divided into monthly data sets, and the resulting monthly data sets were grouped by records with the same "ad group ID (ad_group_id)," "image width" (image_width), and "image height" (image_height). The reason for considering "image width" (image_width) and "image height" (image_height) when grouping is because the relative effectiveness of images of the same size is important when determining ad submission priorities. Next, groups with only one image (unique number of image_ids) were removed, and groups containing two or more images were extracted. The "impressions" (impressions) for each "image" (image_id) within each group were then totaled. This creates multiple groups with "image" (image_id) and "impressions" (impressions) as columns. "image" (image_id) and "impressions" (impressions) are then used for learning and testing.

On the other hand, the procedure for creating data that does not take into account the ad structure is as follows. First, prepare training data for "data that takes into account the ad structure." Then, ignore "group" and add up the "impressions" for each "image" (image_id). This creates data with columns for "image" (image_id) and "impressions." Then, use "image" (image_id) and "impressions" for training.

[Learning method]
"Impressions" is used as an index for rank learning. The learning procedure taking into account the structure of advertisements is as follows. First, as described above, training data is prepared that has multiple groups with "image" (image_id) and "impressions" as columns. Next, the groups are extracted one by one in order, and two rows of records are randomly extracted from each extracted group. Then, pairwise learning is performed with the extracted records. As in Non-Patent Document 1, RankNet is used as the loss function. When records have been extracted once from all groups, one learning epoch is considered to be complete. This learning epoch is repeated 10 times.

On the other hand, the learning procedure for not taking into account the ad structure is as follows. First, as described above, prepare learning data with the columns "image" (image_id) and "impressions" (impressions). Next, randomly extract two records. Then, perform pairwise learning with the extracted records. When the same number of records as the number of groups in "learning taking into account the ad structure" have been extracted, one learning epoch is completed. This learning epoch is repeated 10 times.

The testing procedure is as follows: After 10 epochs of training, the average rank correlation coefficient for the test data is calculated and recorded. To calculate the rank correlation coefficient, first calculate the model output (score) for all images, then calculate the rank correlation coefficient between that score and the indicators (impressions) being trained for each group and average them.

Table 1 shows the results of the learning experiment. The values shown in this table are the average rank correlation coefficients for the test data after 10 epochs of learning.

As can be seen from these results, the average rank correlation coefficient for the test data is higher for training that takes ad structure into account (0.2349) than for training that does not (0.1026). This result shows that training that takes ad structure into account can predict rankings more accurately than training that does not take ad structure into account.

As described above, rank learning that takes into account the structure of advertisements according to the present invention improves rank prediction accuracy compared to rank learning that does not take into account the structure of advertisements. As a result, it becomes possible to more accurately estimate the quality of advertisement creation within a group whose advertising effectiveness is to be ranked.

As explained above, the present invention employs a method for performing rank learning of ad creation delivery priorities on a group-by-group basis, with ads within the same group being extracted as pairs for learning. Such group-by-group learning improves rank prediction accuracy compared to predicting rank within the entire dataset. As a result, it becomes possible to more accurately estimate the quality of ad creations within groups for which it is desired to rank advertising effectiveness.

This invention improves the accuracy of rank prediction within a group for which you want to rank advertising effectiveness.

Claims (12)

A method for learning a learning model for estimating advertising effectiveness, comprising the following steps A to E:
Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data;
Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements;
Step C: selecting one rank learning group from the plurality of rank learning groups;
Step D: calculating a loss function value for the selected one rank learning group;
Step E: a step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups;
Steps D and E are steps of performing rank learning using the loss function value of the selected one rank learning group to correct the learning model;
The method for learning a learning model further comprises step F of executing steps C to E for all of the plurality of rank learning groups obtained by grouping, and the execution of step F corrects the learning model for all of the learning data. The learning method for a learning model described in claim 1, wherein steps D and E are steps of performing rank learning to correct the learning model so as to minimize the loss function value of the selected rank learning group. The learning method for a learning model described in claim 1 or 2, wherein step F includes a substep of setting the sum of the loss function values of each of the rank learning groups as the loss function value of the learning model. The learning method for a learning model described in claim 1 or 2, wherein the conditions characterizing the advertisement include at least one of delivery settings, image size, delivery date and time, and ad group. The learning method for a learning model according to claim 1 or 2, wherein the loss function value is calculated using a pairwise or listwise loss function. creating a ranking group consisting of a plurality of advertisements that have been delivered or have not yet been delivered, based on conditions that characterize the advertisements;
Calculating a score for each of the plurality of advertisements included in the ranking group using a learning model trained by the method according to claim 1 or 2;
determining a delivery priority based on the calculated score for each advertisement;
How to estimate advertising effectiveness. A program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, the method comprising:
Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data;
Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements;
Step C: selecting one rank learning group from the plurality of rank learning groups;
Step D: calculating a loss function value for the selected one rank learning group;
Step E: a ranking learning step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups;
Steps D and E are steps of performing rank learning using the loss function value of the selected one rank learning group to correct the learning model;
The program further comprises a step F of executing steps C to E for all of the plurality of rank learning groups obtained by grouping, wherein execution of step F corrects a learning model for all of the learning data. 8. The program according to claim 7 , wherein steps D and E are steps of performing rank learning to correct a learning model so as to minimize a loss function value of the selected one rank learning group. The program described in claim 7 or 8, wherein step F includes a substep of setting the sum of the loss function values of each of the rank learning groups as the loss function value of the learning model. 9. The program according to claim 7, wherein the conditions characterizing the advertisement include at least one of a distribution setting, an image size, a distribution date and time, and an advertisement group. The program described in claim 7 or 8, wherein the loss function value is calculated using a pairwise or listwise loss function. A program for causing a computer to execute a method for estimating advertising effectiveness, the method comprising:
creating a ranking group consisting of a plurality of advertisements that have been delivered or have not yet been delivered, based on conditions that characterize the advertisements;
Calculating a score for each of the plurality of advertisements included in the ranking group using a learning model trained by the method according to claim 1 or 2;
determining a delivery priority based on the calculated score for each advertisement;
program.