JP7387703B2 - Device, method and program for predicting gaze area of advertising video - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (Part 1) Publication date September 13, 2021 Publication Fuzzy System Symposium Lecture Proceedings 37, 67-70 (Part 2) Event date September 13, 2021 to September 2021 March 15th Meeting name, venue 37th Fuzzy System Symposium/FSS2021 (held online)

The present invention relates to a technique for predicting a gaze area of an advertising video.

The user's perspective is an important discussion in the production stage of advertising videos, and until now, the only way to objectively confirm the user's gaze area was to use direct aggregate evaluation using eye tracking, which is difficult from a cost and time cost perspective. Direct measurements were often abandoned. Ahead of the spread of 5G, digital advertising agencies are gradually starting to handle TV advertising, and it is becoming increasingly important to objectively discuss user gaze. There is a high demand for an environment that can be clearly checked.

On the other hand, Patent Document 1 discloses a technique for generating an attention map that expresses a gaze region of an input image using a trained neural network (trained model).

JP 2021-22368 Publication

DROSTE Richard, JIAO Jianbo, NOBLE J. Alison: Unified image and video saliency modeling, European Conference on Computer Vision, Springer Cham, pp. 419-435, 2020, URL: https://arxiv.org/pdf/2003.05477.pdf SANDLER Mark, HOWARD Andrew, ZHU Menglong, ZHMOGINOV Andrey, CHEN Lian-Chieh: MobilenetV2: Inverted residuals and linear bottlenecks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018 JIANG Ming, HUANG Shengsheng, DUAN Juanyong, ZHAO Qi: Salicon: Saliency in context, IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp.1072-1080, 2015 WANG Wenguan, SHEN Jianbing, XIE Jianwen, CHENG Ming-Ming, LING Haibin, BORJI Ali: Revisiting Video Saliency Prediction in the Deep Learning Era, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43, 1, pp.220- 237, 2019 MATHE Stefan, SMINCHISESCU Cristian: Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 37, 7, pp.1408-1424, 2015

Typically, a trained model is generated by machine learning using a learning data set containing only still images or a training data set containing only moving images. However, when predicting the gaze area of an advertising video using such a normal trained model, it was found that high prediction accuracy could not be obtained. Furthermore, the method of directly evaluating the gaze area using eye tracking has the problem of requiring a great deal of effort as described above.

The present invention has been made to solve the above problems, and an object of the present invention is to easily and accurately predict the viewing area of an advertising video.

In order to solve the above problem, the present inventor focused on the fact that advertising videos have a characteristic that there are natural video regions and video regions that are artificial and partly still images, such as pop-up and application screens. However, they discovered that by applying a model that has a mechanism that allows simultaneous analysis of still images and videos, it is possible to predict the gaze area with high precision for the first time.

The present invention includes the following aspects.
Item 1.
A device for predicting a gaze area of an advertising video, the device comprising:
a prediction unit that predicts a gaze area of the advertising video by inputting the advertising video into a machine-learned model;
The device is characterized in that the learned model is generated by machine learning using learning images including moving images and still images and information on a gaze area in each learning image as a learning data set.
Item 2.
2. The method according to item 1, further comprising a map generation unit that generates a saliency map that visualizes the predicted gaze area.
Item 3.
A method for predicting a gaze area of an advertising video, the method comprising:
the step of predicting a gaze area of the advertising video by inputting the advertising video into a machine-learned model;
A method characterized in that the learned model is generated by machine learning using learning images including moving images and still images and information on a gaze area in each learning image as a learning data set.
Item 4.
4. The method according to item 3, further comprising the step of generating a saliency map that visualizes the predicted gaze region.
Item 5.
A program that causes a computer to operate as each part of the device according to item 1 or 2.

According to the present invention, the gaze area of an advertising video can be easily and highly accurately predicted.

1 is a functional block diagram showing a schematic configuration of an analysis system according to an embodiment of the present invention. This is an example of a saliency map. This is the configuration of the UNISAL model. This is a schematic configuration of the UNISAL model. (A) to (C) are patterns of the UNISAL model in the example. This is an example of a saliency map in an example.

Embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

FIG. 1 is a functional block diagram showing a schematic configuration of an analysis system 100 according to an embodiment of the present invention. The analysis system 100 includes an analysis device 1 and a user terminal 2. The analysis device 1 is provided on the cloud and is connected to the user terminal 2 via a communication network such as the Internet.

The analysis device 1 is a device that analyzes images, and particularly has a function of predicting a human gaze area in an advertising video. The analysis device 1 can be configured with a general-purpose computer, and its hardware configuration includes a processor such as a CPU or GPU (not shown), a main storage device such as a DRAM or SRAM (not shown), and an HDD or SSD. It is equipped with an auxiliary storage device 10.

The auxiliary storage device 10 stores a learned model M generated by machine learning and various programs for operating the analysis device 1. The learned model M is generated by machine learning using learning images including moving images and still images and information on the gaze area in each learning image as a learning data set. That is, unlike a normal trained model, this is a model that has been machine learned using both moving images and still images. Although the machine learning method is not particularly limited, for example, deep learning or the like can be used.

The analysis device 1 includes a communication section 11, a prediction section 12, and a map generation section 13. The communication unit 11 is an interface for the analysis device 1 to communicate with other devices. The prediction unit 12 and the map generation unit 13 are realized in software by the processor of the analysis device 1 reading a predetermined program into the main storage device and executing it.

The user terminal 2 can be configured with a general-purpose computer, and its hardware configuration includes a processor such as a CPU or GPU (not shown), a main storage device such as a DRAM or SRAM (not shown), and an HDD or SSD. It is equipped with an auxiliary storage device 20. The auxiliary storage device 20 stores an advertisement video A created by a user or the like.

The user terminal 2 includes a communication section 21 and a display section 22. The communication unit 21 is an interface for the user terminal 2 to communicate with other devices. The display unit 22 is composed of, for example, a liquid crystal display device.

When predicting the gaze area in the advertising video A, the communication unit 21 transmits the advertising video A to the analysis device 1 . The advertisement video A can be transmitted, for example, by uploading it using a web browser.

When receiving the advertising video A from the user terminal 2, the communication unit 11 of the analysis device 1 inputs the advertising video A to the prediction unit 12.

The prediction unit 12 predicts the gaze area of the advertising video A by inputting the advertising video A into the trained model. Specifically, the prediction unit 12 decomposes the advertising video A into frames, digitizes the brightness of each pixel, etc., and inputs the numeric values into the learned model M. The trained model M outputs a predicted value of the degree of attention at each coordinate in the advertising video A based on the input numerical values. The prediction unit 12 inputs the output data of the trained model M to the map generation unit 13.

The map generation unit 13 generates a saliency map that visualizes the gaze area predicted by the prediction unit 12. Specifically, the map generation unit 13 generates a saliency map by converting the frames decomposed by the prediction unit 12 into a video in a heat map format. The manner in which the gaze area is visualized is not particularly limited, but it can be visualized by coloring the gaze area, as in the saliency map shown in FIG. 2, for example. In this saliency map, the red areas are areas where the gaze of people watching the video is predicted to be concentrated.

The saliency map generated by the map generation unit 13 is transmitted to the user terminal 2 via the communication unit 11. Upon receiving the saliency map, the communication unit 21 of the user terminal 2 outputs the saliency map to the display unit 22, and the saliency map is displayed on the display unit 22.

(Brief Summary)
Advertising videos have a natural video area and a video area that is artificial and partly still images, such as pop-ups and application screens, so it is difficult to use models that are machine learned using only videos or only still images. Machine learning models cannot predict the gaze area with high accuracy. Therefore, in this embodiment, we use a model that has been machine learned using both moving images and still images, that is, a model that has a mechanism that allows simultaneous analysis of still images and moving images, to predict the gaze area of advertising videos. Is going. As a result, the gaze area of an advertising video can be easily and highly accurately predicted without using labor-intensive eye tracking techniques.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various changes can be made without departing from the spirit thereof.

In order to verify the analysis accuracy of the gaze area of an advertising video, a test was conducted using a trained model developed by combining multiple open data as the trained model M shown in FIG. Specifically, a UNISAL model was used as the trained model. Note that the original paper on the UNISAL model (Non-Patent Document 1) discusses only from the perspective of whether still images and videos can be analyzed with one model, and this makes it difficult to analyze still images such as advertising videos. There is no discussion that it functions superiorly to videos that have a partial structure.

FIG. 3 shows the configuration of the UNISAL model shown in FIG. 3(a) of Non-Patent Document 1, and FIG. 4 is a simplified block diagram of this UNISAL model. In the UNISAL model, input moving images operate as continuous still images (frames), and a saliency map is output via an encoder 3, an RNN 4, and a decoder 5. The encoder 3 has the functions of a spatial filter, MobileNetV2 (non-patent document 2), and offset, the RNN 4 has the function of a temporal filter, and the decoder 5 has the functions of expansion, fusion, and smoothing.

A data path directly connecting the encoder 3 and decoder 5 represents the transmission of still image information. Data calculations through the RNN4 are in charge of processing motion information, and the entire network functions even with a graph structure in which each line is cut. In addition, a part that learns to adapt to a dataset during learning and a domain adaptation technology that learns to adapt to all datasets are partially introduced, and the stars in Figure 3 are used. They have been introduced in the marked areas.

In this example, the basic properties of what kind of output the UNISAL model outputs for the video of the dataset were qualitatively evaluated. The UNISAL model calculates the time-series elements of the video in the RNN4 part, and by bypassing this, it can operate in the same state as making predictions using only still image information. By constructing saliency map prediction that considers motion information, a prediction model that considers only motion information, a prediction model that handles both information, and a model that is trained without domain adaptation, and evaluating their performance, We verified from a quantitative perspective using evaluation indicators whether the hypothesis that a model that utilizes both video information is the most appropriate is valid.

Specifically, as a UNISAL model control,
(1) Domain adaptation applied to the parts marked with an asterisk in Figure 3 is turned off (Figure 5 (A), the structure is the same as Figure 4)
(2) Motion information only, which cuts off the information directly transmitted from the encoder 3 to the decoder 5 and uses only the output of the RNN 4 (Figure 5 (B))
(3) One in which the output of RNN 4 is cut and motion information is turned off, operating only on the direct path from encoder 3 to decoder 5 (Figure 5 (C))
(4) All functions activated (Figure 5 (A))
We created four patterns, and compared these four patterns with those that were newly learned and those that used trained models. The seven model numbers used for this comparison. 1 to 7 are shown in Table 1.

In a pattern where the domain is a still image, all other datasets are moving images, so the domain is fixed to the still image dataset SALICON (Non-Patent Document 3). For those whose learning state is default, the model downloaded from GitHub published by Non-Patent Document 1 is applied as is, and for those whose learning state is new, a new model has been created. . For those whose dataset is Full, the still image dataset SALICON and the video datasets DHF1K (Non-patent Document 4), Hollywood-2 (Non-patent Document 5), and UCFSports (Non-patent Document 5) are learned respectively. The ones with 3 videos are ones that were trained using video datasets other than SALICON.

In order to quantitatively evaluate the impact on performance of changing model control, we used five indicators: AUC-J, SIM, S-AUC, CC, and NSS using DHF1K under the same conditions as Non-Patent Document 1. I made a comparison. Each index of NSS, CC, SIM, AUC-J, and S-AUC is the index used in Non-Patent Document 1. NSS is an index for evaluating the similarity of saliency maps, and is an index widely used mainly in papers on models for predicting saliency maps.

Table 2 shows the results for each index. The maximum difference in each index was 0.023 for AUC-J, 0.104 for SIM, 0.025 for S-AUC, 0.077 for CC, and 0.513 for NSS, respectively.

Each model No. Table 3 shows the performance ranking for each index from 1 to 7.

Model no. Model 1 uses the model published on GitHub with default settings, and shows excellent and stable performance overall, even from the average score. Model No. using only still image information. Model No. 2 has the lowest performance, and model No. 2 uses only motion information of the video. Regarding 4, the performance was not very good, at best 4th place.

An example of one video from the DHF1K dataset is shown in FIG. While the ground truth saliency map is focused on the butterfly's head, the still image output spans the entire area. Overall, there was a similar relationship between the still image model and ground truth in DHF1K, and the more the number of subjects increased, the more the output of the model predicted using only still image information tended to disperse. For this reason, even for advertising videos that are composed of various parts, in UNISAL control, it is more suitable to make predictions using a default setting model than to make predictions based only on still image information. Model No. that made predictions using only motion information. Rather than using Model No. 4, which is the default model. Since Model No. 1 can stably observe overall better scores in Table 3, the system for predicting the gaze area of advertising videos uses Model No. 1, which uses both videos and still images. 1 is suitable.

Based on the above results, model No. We developed a UI for an advertising video analysis tool using 1, and evaluated the output by video directors who actually produce advertising videos. We actually made predictions for 5 advertising videos that were actually distributed and 10 advertising video drafts (prototype videos before providing them to clients), and the director analyzed the results. While analyzing this, we were able to make predictions that were quite close to the ``actual value,'' and we were able to use a very creative presentation method even when loading something that contained many objects (and only for a moment). There was a difference in the results, and it has reached a level that can be used in actual combat.The sensation may be similar to that of the human skin.'', ``In Fantz's preferential gaze experiment, newborns and infants were exposed to visual stimuli. When assessing the time spent gazing at the face, print, bow target, red, white, and yellow circles, both infants and newborns found that face > print = bow target. It is known that red = white = yellow, and people are most interested in things where the arrangement of eyes, nose, and mouth resembles a face, but the same phenomenon has also been confirmed. It was praised as ``reproduced.'' Therefore, model no. We were able to confirm that 1 can be applied to the business.

Also, model no. Business application 1 has the following advantages.
1. Survey can be completed in a few minutes 2. 3. Temporary editing data can be used to refine the UI/UX perspective. The above three advantages of being able to refine the screen before shooting, depending on the preparation of the V-con data, are sufficiently useful for efficiently producing better advertising videos.

As described above, in the performance evaluation of control changes using DHF1K, it was confirmed that among the UNISAL models, the model that uses both motion information and still image information using domain adaptation exhibits the most stable performance. Ta. In addition, in the evaluation analysis by the video director of the advertising video evaluation system using a predictive model that uses all the information, the output of the predictive model is similar to the Fantz preference gaze experiment data, so it is not practical for business applications. I was able to confirm that there is.

1 Analysis device 10 Auxiliary storage device 11 Communication unit 12 Prediction unit 13 Map generation unit 2 User terminal 20 Auxiliary storage device 21 Communication unit 22 Display unit 3 Encoder 4 RNN
5 Decoder 100 Analysis system A Advertising video M Learned model

Claims (5)

A device for predicting a gaze area of an advertising video, the device comprising:
a prediction unit that predicts a gaze area of the advertising video by inputting the advertising video into a machine-learned model;
The device is characterized in that the learned model is generated by machine learning using learning images including a moving image region and a still image region, and information on a gaze region in each learning image as a learning data set. The apparatus according to claim 1, further comprising a map generation unit that generates a saliency map that visualizes the predicted gaze area. A method for predicting a gaze area of an advertising video, the method comprising:
the step of predicting a gaze area of the advertising video by inputting the advertising video into a machine-learned model;
A method characterized in that the learned model is generated by machine learning using learning images including a moving image region and a still image region and information on a gaze region in each learning image as a learning data set. 4. The method according to claim 3, further comprising the step of generating a saliency map visualizing the predicted gaze region. A program for causing a computer to operate as each part of the apparatus according to claim 1 or 2.

Images