JP7632871B2 - Behavior determination device, behavior determination method, program, and recording medium - Google Patents

Description

The present invention relates to a behavior determination device, a behavior determination method, a program, and a recording medium.

In various fields such as revitalizing the local economy and health management, it is important to encourage specific user behavior. For example, Patent Document 1 reports a technology that encourages users to exercise.

JP 2013-090758 A

However, the technology disclosed in Patent Document 1 and other publications has the problem that it is not clear whether a user has actually taken a particular action.

The present invention aims to provide a behavior determination device, a behavior determination method, a program, and a recording medium that can determine whether a user has performed a specific behavior.

In order to achieve the above object, the behavior determination device of the present invention comprises:
A setting unit, an acquisition unit, a detection unit, and a determination unit are included,
The setting unit sets recommended action information including first-stage pass label information,
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquisition unit acquires an image captured by a user,
The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image,
The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Passed" means that the device performed the action indicated by the recommended action information and captured an image related to the action.

The behavior determination method of the present invention includes the steps of:
The method includes a setting step, an acquisition step, a detection step, and a determination step,
The setting step sets recommended action information including first pass label information;
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquiring step acquires an image captured by a user,
The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image,
The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the action indicated by the recommended action information is executed and an image related to the action is captured.

The present invention makes it possible to determine whether a user has performed a specific action.

FIG. 1 is a block diagram illustrating a configuration of an example of a behavior determining device according to the first embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the behavior determination device according to the first embodiment. FIG. 3 is a flowchart illustrating an example of processing in the behavior determining device of the first embodiment. FIG. 4A is a schematic diagram showing an example of adding secondary pass label information. FIG. 4B is a schematic diagram showing an example of adding secondary pass label information. FIG. 4C is a schematic diagram showing an example of adding secondary pass label information. FIG. 5A is a schematic diagram showing an example in which secondary pass label information is not added. FIG. 5B is a schematic diagram showing an example in which secondary pass label information is not added. FIG. 5C is a schematic diagram showing an example in which secondary pass label information is not added.

In the behavior determination device of the present invention, for example,
The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection unit further detects the confidence level for the label information,
The determination unit may further determine the image as "pass" when the certainty factor for the label information satisfies a threshold value of the primary pass label information.

The behavior determination device of the present invention includes, for example,
Further, the recording unit and the extracting unit are included,
the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other;
The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information;
the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information;
The judgment unit may be configured to judge the image as "pass" when at least one of the judgments based on the label information and the label information of the first passed label information, and the judgment based on the label information and the label information of the second passed label information, results in a "pass" judgment.

In the behavior determination device of the present invention, for example,
The extraction section may extract the label information when the number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number.

In the behavior determination device of the present invention, for example,
In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
The setting unit may calculate one of the average, median, or mode of the certainty levels from the certainty levels for each of the overlapping label information, and add the calculated numerical value to the imaging instruction information as the certainty level for the secondary pass label.

In the behavior determination device of the present invention, for example,
The condition preset in the setting unit may be at least one of the following (1) and (2).
(1) In the execution history data, label information other than the label information that matches the first pass label information is overlapped, and at least one of the average, median, and mode calculated from the respective certainties for the overlapping label information is equal to or greater than a predetermined value.
(2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.

The behavior determination device of the present invention includes, for example,
Further, the system includes a recommended behavior information presenting unit and a token granting unit,
the recommended action information presentation unit presents the recommended action information to the user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
The token granting unit may grant, to a user, token information linked to the recommended behavior information, when the determination unit determines that the user is “passed.”

In the behavior determination method of the present invention, for example,
The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection step further includes detecting the confidence level for the label information,
The judging step may further include judging the image as "pass" when the certainty factor for the label information satisfies a threshold value of the primary pass label information.

The behavior determination method of the present invention includes, for example,
Further, the method includes a recording step and an extracting step,
The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other;
The extraction step extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information;
the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information;
The judgment process may be in a form in which the image is judged to be "passed" when at least one of the judgments based on the label information and the label information of the first passed label information, and the judgment based on the label information and the label information of the second passed label information, results in a "pass" judgment.

In the behavior determination method of the present invention, for example,
The extraction step may be configured to extract the label information when the number of images determined to be "passed" in the execution history data is equal to or greater than a preset number.

In the behavior determination method of the present invention, for example,
In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
The setting process may include calculating one of the average, median, or mode of the certainty degrees from the respective certainty degrees for the overlapping label information, and adding the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.

In the behavior determination method of the present invention, for example,
The condition preset in the setting step may be at least one of the following (1) and (2).
(1) In the execution history data, label information other than the label information that matches the first pass label information is overlapped, and at least one of the average, median, and mode calculated from the respective certainties for the overlapping label information is equal to or greater than a predetermined value.
(2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.

The behavior determination method of the present invention includes, for example,
Further, the system includes a recommended action information presenting step and a token granting step,
The recommended action information presenting step includes presenting the recommended action information to a user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
The token granting step may be configured to grant, to the user, token information linked to the recommended behavior information, when the result of the determination step is “passed.”

The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

The application field of this invention is any field in which specific actions are recommended to users, in other words, the actions that users should take toward a goal are managed, or the actions that users have already taken are confirmed, such as tourism, revitalization of the local economy, health management, employee management, and education.

Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. Furthermore, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
Fig. 1 is a block diagram showing an example of the configuration of a behavior determination device 10 according to the present embodiment. As shown in Fig. 1, the device 10 includes a setting unit 11, an acquisition unit 12, a detection unit 13, and a determination unit 14. The device 10 may further include, as optional components, a recording unit 15, an extraction unit 16, a recommended behavior information presentation unit 17, a token granting unit 18, and the like.

The device 10 may be, for example, a single device including each of the above-mentioned parts, or a device to which each of the above-mentioned parts can be connected via a communication line network. The device 10 may also be connected to an external device described later via the communication line network. The communication line network is not particularly limited and may be a publicly known network, for example, wired or wireless. Examples of the communication line network include the Internet line, WWW (World Wide Web), telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (Ad Hoc communication), infrastructure communication, indirect communication via an access point, and the like. The device 10 may be incorporated into a server as a system. The device 10 may be, for example, a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the parts is on a server and the other parts are on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, etc. Each part of the device 10 is connected to each other via the bus 103 by its respective interface (I/F).

The central processing unit 101 is responsible for the overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various information. Specifically, for example, the central processing unit 101 functions as a setting unit 11, an acquisition unit 12, a detection unit 13, and a determination unit 14. The central processing unit 101 also functions as, for example, a recording unit 15, an extraction unit 16, a recommended behavior information presentation unit 17, and a token granting unit 18, etc.

The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 10 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory. As described above, the storage device 104 stores operating programs including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a hard disk drive (HDD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD).

In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used by the present device 10 when executing processing. At least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner on multiple terminals using blockchain technology or the like. The memory 102 and the storage device 104 may also store, for example, various information recorded by the recording unit 15.

The device 10 may further include, for example, an input device 105 and a display device 106. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc.

Next, an example of the behavior determination method of this embodiment will be described based on the flowchart in FIG. 3. The behavior determination method of this embodiment is implemented, for example, as follows using the behavior determination device 10 in FIG. 1. Note that the behavior determination method of this embodiment is not limited to use of the behavior determination device 10 in FIG. 1. Also, the steps in parentheses in FIG. 3 are optional steps.

First, the setting unit 11 sets recommended action information including the primary pass label information (S11a, setting step). The recommended action information is information indicating an action recommended to the user. The action is not particularly limited, and may be, for example, actions related to shopping, exercise, sightseeing, watching, learning, etc. The primary pass label information is label information indicating an imaging target related to the action recommended to the user, and may include, for example, a threshold value of confidence for the label information. The label information may be, for example, a character string such as a word, or identification information such as numbers, alphabets, and any combination of two or more of them, but is information specific to the imaging target. The imaging target is not particularly limited. The confidence is an index indicating the likelihood that the imaging target is the imaging target. There may be multiple label information of the primary pass label information, and in this case, there may be a threshold value of the confidence for each label information. The setting unit 11 performs the setting according to, for example, information inputted via the input device 105. The recommended action information may be transmitted to the outside, for example, via the communication line network.

Next, the acquisition unit 12 acquires an image captured by the user (S12, acquisition step). The user is, for example, a user to whom the recommended action information is presented. That is, it is presumed that the image is an image related to the action captured according to the recommended action information. The user acquires the recommended action information and captures the image, for example, using the user's terminal (smartphone, etc.). The acquisition unit 12 acquires the image from the user's terminal (smartphone, etc.) via, for example, the communication line network. The image may be, for example, a video composed of multiple frames (images). The acquisition unit 12 may acquire, for example, location information of the user (user's terminal), identification information of the terminal that captured the image, identification information of the terminal that transmitted the image to the device 10, etc., in association with the image. The acquisition unit 12 may also acquire, for example, the recommended action information acquired (executed) by the user in association with the image.

Next, the detection unit 13 inputs the acquired image into the trained model to detect label information of the imaging target included in the image (S13, detection step). The detection unit 13 may, for example, input the acquired image into the trained model to detect label information of the imaging target included in the image and the confidence level for the label information. The label information detected by the detection unit 13 is also referred to as a detected label, for example. The trained model may be a model trained to detect label information of the imaging target included in the image from the input image, or a model trained to detect label information of the imaging target included in the image and the confidence level for the label information from the input image. The trained model may be stored in the memory 102 and the storage device 104 in advance, or may be acquired from an external device via a communication line network. In addition, the detection unit 13 may use a trained model held by the external device, for example, without acquiring the trained model from the external device. The label information may be detected in multiple pieces, and the confidence level may be detected for each piece of label information. As the trained model, for example, a trained model provided by the Vision API of Google Cloud Platform can be used.

The trained model includes, for example, an input layer for inputting an image, an output layer for outputting label information of the imaged object (and the confidence level for the label information), and at least one intermediate layer provided between the input layer and the output layer. The trained model may be a program module that is a part of artificial intelligence software. An example of the multi-layered network is a neural network. An example of the neural network is a convolution neural network (CNN), but is not limited to CNN, and may be a trained model constructed with other learning algorithms such as a neural network other than CNN, an SVM (Support Vector Machine), a Bayesian network, or a regression tree.

Then, the judgment unit 14 judges whether the image is "passed" or not based on the label information and the label information of the primary pass label information (S14a, judgment step), and ends (END). The judgment unit 14 may further judge the image as "passed" if the confidence level for the detected label information meets the threshold value of the primary pass label information, for example. Note that "passed" means that the user has taken the action indicated by the previous recommended action information and captured an image related to the action, but the term is not limited to this and may be "OK", "mission successful", etc. Specifically, the judgment unit 14 may judge the image as "passed" if, for example, the detected label information matches the label information of the primary pass label information. Note that the match is not limited to a perfect match, and may be, for example, a partial match, or may be a "match" based on the rate of match (hereinafter, the same applies to "match"). The rate is not particularly limited and can be set arbitrarily. When there are multiple pieces of label information in the first pass label information, the judgment unit 14 may judge the image to be a "match" if at least one of the multiple pieces of label information is satisfied, or may judge the image to be a "match" if all of the multiple pieces of label information are satisfied. In other words, the first pass label information is information for judging the image to be a "pass."

On the other hand, the judgment unit 14 may judge the image to be "failed" when, for example, the detected label information does not match the label information of the first pass label information, or when the confidence level for the label information does not meet the threshold value of the first pass label information. Note that "failed" means, for example, that the user performed the action indicated by the recommended action information but was unable to capture an image related to the action, or was unable to perform the action indicated by the recommended action information, but the term is not limited thereto and may be "NG", "mission failed", etc.

The setting unit 11 may further set recommended action information including prohibited label information. The prohibited label information is, for example, label information indicating an image target that should not be included in an image, and may include, for example, a threshold value of confidence for the label information. The determination unit 14 may determine the image as "failed" based on, for example, the detected label information and the label information of the prohibited label information (for example, when the detected label information matches the label information of the prohibited label information, etc.). The determination unit 14 may further determine the image as "failed" when, for example, the confidence for the label information satisfies the threshold value of the prohibited label information. That is, the prohibited label information is, in other words, information for determining the image as "failed". Note that the determination unit 14 does not necessarily determine the image as "passed" when, for example, the detected label information does not match the label information of the prohibited label information or the confidence for the label information does not satisfy the threshold value of the prohibited label information.

Furthermore, the primary pass label information may be information including, for example, the position information of the imaging target. Then, the judgment unit 14 may, for example, compare the position information of the user linked to the image with the position information of the imaging target in the primary pass label information, and judge the image to be "failed" if there is no match. Note that, for example, if the position information of the user linked to the image matches the position information of the imaging target, which is part of the imaging instruction information, the judgment unit 14 does not necessarily judge the image to be "passed."

The determination unit 14 may also, for example, compare the identification information of the terminal that captured the image with the identification information of the terminal that transmitted the image to the device 10, and determine that the image is "failed" if they do not match. Note that the determination unit 14 does not necessarily determine that the image is "passed" if, for example, the identification information of the terminal that captured the image matches the identification information of the terminal that transmitted the image to the device 10.

As described above, the device 10 may include a recording unit 15 and an extraction unit 16. For example, after the step (S14a), the recording unit 15 links and accumulates and records the label information of the image capture target included in the image captured by the user and the result of the judgment by the judgment unit 14 as execution history data (S15, recording step). The recording unit 15 may also link and accumulate and record the confidence level for the label information as the execution history data. The execution history data may be stored in the memory 102 and the storage device 104, for example, or may be stored in an external database. The device 10 may also include a database that stores the execution history data, and store the execution history data in the database.

After executing the step (S15), for example, the extraction unit 16 extracts from the execution history data label information (hereinafter also referred to as label candidates) other than the label information that matches the primary pass label information and is for the images for which the judgment result is "pass" (S16, extraction step). In other words, the label candidates are label information that are not set as the primary pass label information. For example, the extraction unit 16 may extract the label candidates (step (S16)) when the number of images judged to be "pass" in the execution history data is equal to or greater than a preset number. The number is not particularly limited and can be set arbitrarily.

After executing the step (S16), for example, the setting unit 11 adds the label candidate that satisfies a preset condition from among the extracted label candidates to the recommended action information as label information of secondary pass label information and sets it (S11b, setting step). That is, it can be said that the secondary pass label information is, for example, label information indicating an imaging target different from the imaging target indicated by the label information of the primary pass label information. The secondary pass label information may include, for example, an arbitrarily set threshold value of the confidence level. For example, a preset numerical value (for example, 50, etc.) may be assigned as the confidence level of the secondary pass label information. For example, the label candidates may overlap in the execution history data. When the label candidates overlap, the confidence level for the secondary pass label may be, for example, any one of the average value, median value, or mode value calculated from the respective confidence levels for the overlapping label candidates.

The conditions are not particularly limited and can be set arbitrarily. For example, the conditions include at least one of the following (1) and (2).
(1) The label candidates overlap in the execution history data, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label candidates is equal to or greater than a predetermined value (e.g., 50).
(2) The label candidates are duplicated in the execution history data, and the number of occurrences of the duplicated label candidates in the execution history data is a preset number (e.g., 40 or more) or more.

The condition may also include, for example, an upper limit on the number of secondary passed label information to be added. For example, if the upper limit on the number of secondary passed label information to be added is "1", then, among the multiple label candidates that satisfy the condition, the label information with the highest certainty (which may be an average value calculated from the respective certainty values for the overlapping label candidates) is added as the label information of the secondary passed label information.

After executing the step (S11b), for example, if the judgment unit 14 judges the image to be "pass" in at least one of the judgments (1) to (4) below, the image is judged to be "pass" (S14b, judgment step), and the process ends (END).
(1) Determination based on the detected label information and the label information of the first pass label information (for example, when the detected label information matches the label information of the first pass label information)
(2) In addition to the above (1), the certainty factor for the detected label information satisfies a certainty factor threshold of the first pass label information. (3) A judgment is made based on the detected label information and the label information of the second pass label information (for example, when the detected label information and the label information of the second pass label information match).
(4) In addition to (3) above, the certainty of the detected label information satisfies the certainty threshold of the secondary passed label information. In other words, even if the detected label information is judged to be “failed” (e.g., does not match) in a judgment based on the label information of the primary passed label information, it is sufficient that the detected label information is judged to be “passed” (e.g., matches) in a judgment based on the label information of the secondary passed label information.

As described above, the device 10 may include a recommended behavior information presentation unit 17 and a token granting unit 18. For example, after the step (S11a) and before the step (S12), the recommended behavior information presentation unit 17 presents the recommended behavior information to the user by linking it with token information (S17, recommended behavior information presentation step). The token information is information about a token granted to the user upon completion of the behavior. The token information may be, for example, points, coins, tickets, etc. having monetary value. "Points, coins, tickets, etc. having monetary value" means, for example, that points, etc. can be converted into actual money, can be used as money (for example, 1 point = 1 yen, etc.), can be exchanged for products, etc. The token information having monetary value may be, for example, available only in a limited area. This can promote the revitalization of economic activity in the area. In addition, by making tickets, etc., for example, tickets to watch sports teams in the area, it can be used to expand the number of supporters of the sports team as well as the area. Furthermore, the recommended action information presented to the user may be information excluding the primary pass label information and the secondary pass label information, for example, as long as the recommended action information can be associated with the primary pass label information in a later process (e.g., step S14a, etc.).

For example, after the step (S14a), if the determination unit 14 determines that the user has passed the recommended behavior information, the token granting unit 18 grants the user token information linked to the recommended behavior information (S18, token granting step), and the process ends (END). Specifically, the token information may be electronically granted to the user's terminal as, for example, an electronic coupon or electronic money.

The device 10 may, for example, tally up the tokens granted to each user and display the users in a ranking format.

According to this embodiment, for example, by setting recommended behavior information including primary pass label information, it is possible to determine whether a user has taken a specific behavior using an image captured by the user. Also, according to this embodiment, for example, secondary pass label information can be added, and the image can be more accurately determined to be "pass" or "fail."

In recent years, the revitalization of regional economies has become an important issue due to the problem of concentration in one place in a large city. Specifically, it is important to lay the foundation for many people to participate in purchasing activities widely in various fields such as shopping, eating and drinking, and tourism. However, according to the present embodiment, for example, by linking the recommended behavior information with the token information and presenting it to the user, the user can be guided to various places, promoting behavior such as purchasing, and also returning services (tokens) to the user. This can lead to active behavior of the user and lead to regional revitalization. The scope of the region is not particularly limited, and may be, for example, a town unit, a ward unit, a city unit, or a prefecture unit. Therefore, when the device 10 includes the recommended behavior information presentation unit 17 and the token granting unit 18, the device 10 can also be called a behavior promotion device. Similarly, the behavior determination method can be called a behavior promotion method.

[Embodiment 2]
An example of a process for adding the secondary passed label information will be described. In this embodiment, the primary passed label information and the secondary passed label information each include a threshold value of the confidence level for the label information, but as described above, this is not limited to this.

Figures 4A to 4C are schematic diagrams showing an example of adding the secondary pass label information. As a prerequisite for Figures 4A to 4C, the action recommended to the user is "Play soccer and get moving!" and the image capture instruction is to capture an image of "a scene of soccer being played." Then, as shown in the figure, the setting unit 11 sets "soccer" and "ball" as label information for the primary pass label information, sets the confidence level for each of "soccer" and "ball" to "80%," and sets it so that at least one of these needs to be satisfied (match condition: OR). The primary pass label information at this time is in the "initial state" data state, as shown in Figure 4A.

4A, it is assumed that the execution start condition for the step (S16) shown in Fig. 3 is set to be that the number of images judged as "pass" in the execution history data is 100 or more. It is also assumed that the following two conditions are set for the label candidate to be added as the secondary pass label information.
(1) The average value (average confidence) calculated from the confidences of each of the label candidates is 50% or more.
(2) The number of occurrences of the label candidate is 50% or more for all label information other than the label information of the first pass label information among the images judged to be “pass”.

Next, assume that many users carry out the recommended action information (also called a mission) and send images to the device 10. Then, the process from step (S12) and before step (S16) shown in FIG. 3 is repeatedly executed, and the recording unit 15 accumulates and stores execution history data linking the label information of the object contained in the image captured by the user, the confidence level for the label information, and the result of the judgment by the judgment unit. An example of the execution history data is shown in the row of item "2" in FIG. 4B.

When the number of images judged as "passed" by the judgment unit 14 exceeds 100, the step (S16) shown in FIG. 3 is executed. Specifically, as shown in the row of item "3" in FIG. 4B, the extraction unit 16 extracts, from the execution history data, label information (label candidates of provisional secondary pass label information) other than the label information that matches the primary pass label information for the images whose judgment result is "passed". Then, as shown in the row of item "4" in FIG. 4C, the setting unit 11 adds and sets the label candidates that satisfy the above conditions (1) and (2) from the extracted label candidates as label information of secondary pass label information to the recommended action information. In this example, "football" and "sportswear" were added to the recommended action information as label information of the secondary pass label information. The confidence threshold of the secondary pass label information is also set in association with the label information. In this example, the setting unit 11 sets, for example, the added secondary pass label information as "matching condition: OR".

After that, assume that a new mission is performed by the user, and the label information and the confidence level detected by the detection unit 13 are "label information: soccer, confidence level: 75%" and "label information: football, confidence level: 80%". In this case, even if the detected "label information: soccer, confidence level: 75%" does not satisfy the confidence level threshold "label information: soccer, threshold level: 80%" of the first pass label information, the judgment unit 14 judges the mission as "passed" because the detected "label information: football, confidence level: 80%" satisfies the label information and the threshold "label information: football, confidence level: 75%" of the second pass label information. In this way, by adding the second pass label information, it is not necessary to periodically review the judgment conditions (matching conditions with the first pass label information), and the judgment conditions can be dynamically corrected.

5A to 5C are schematic diagrams showing an example in which the secondary pass label information is not added. The preconditions for Fig. 5A to 5C are the same as those for Fig. 4A to 4C, except that the above condition (2) is changed as follows.
(2) The number of occurrences of the label candidate is 80% or more for all label information other than the label information of the first pass label information among the images judged to be “pass”.

Figure 5A is similar to Figure 4A, except that condition (2) has been changed as described above, and Figure 5B is similar to Figure 4B.

As shown in the row of item "4" in FIG. 5C, since there is no label candidate that satisfies the above conditions (1) and (2) among the extracted label candidates, the setting unit 11 does not add the secondary pass label information.

After that, suppose that a new mission is performed by the user, and the label information and the certainty detected by the detection unit 13 are "label information: soccer, certainty: 75%" and "label information: football, certainty: 80%". In this case, the determination unit 14 determines that the detected "label information: soccer, certainty: 75%" does not meet the certainty threshold of the first pass label information, "label information: soccer, threshold: 80%", and the second pass label information corresponding to the detected "label information: football, certainty: 80%" has not been added, so that the image is determined to be "failed". Therefore, the image is not considered to be an image indicating that the action has been performed.

[Embodiment 3]
The program of the present embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, the "procedure" may be read as "processing." The program of the present embodiment may be recorded, for example, in a computer-readable recording medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments may be described as follows, but are not limited to the following supplementary notes.
(Appendix 1)
A setting unit, an acquisition unit, a detection unit, and a determination unit are included,
The setting unit sets recommended action information including first-stage pass label information,
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquisition unit acquires an image captured by a user,
The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image,
The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the behavior indicated by the recommended behavior information has been performed and an image related to the behavior has been captured by the behavior determination device.
(Appendix 2)
The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection unit further detects the confidence level for the label information,
2. The behavior determination device according to claim 1, wherein the determination unit further determines the image as "pass" when the certainty factor for the label information satisfies a threshold value of the primary pass label information.
(Appendix 3)
Further, the recording unit and the extracting unit are included,
the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other;
The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information;
the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information;
The behavior determination device according to claim 1 or 2, wherein the determination unit determines the image to be "passed" when at least one of the determination based on the label information and the label information of the first passed label information and the determination based on the label information and the label information of the second passed label information results in a "pass" determination.
(Appendix 4)
4. The behavior determination device according to claim 3, wherein the extraction unit extracts the label information when a number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number.
(Appendix 5)
In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
The behavior determination device according to claim 3 or 4, wherein the setting unit calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.
(Appendix 6)
6. The behavior determination device according to claim 3, wherein the condition preset in the setting unit is at least one of the following (1) and (2).
(1) In the execution history data, label information other than the label information that matches the first pass label information is overlapped, and at least one of the average, median, and mode calculated from the respective certainties for the overlapping label information is equal to or greater than a predetermined value.
(2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.
(Appendix 7)
Further, the system includes a recommended behavior information presenting unit and a token granting unit,
the recommended action information presentation unit presents the recommended action information to the user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
7. The behavior determination device according to claim 1, wherein, when the determination unit determines the user to be “passed,” the token granting unit grants token information linked to the recommended behavior information to the user.
(Appendix 8)
The method includes a setting step, an acquisition step, a detection step, and a determination step,
The setting step sets recommended action information including first pass label information;
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquiring step acquires an image captured by a user,
The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image,
The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information,
In this behavior determination method, "passed" means that the behavior indicated by the recommended behavior information is executed and an image related to the behavior is captured.
(Appendix 9)
The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection step further includes detecting the confidence level for the label information,
The behavior determination method according to claim 8, wherein the determination step further determines the image as "pass" if the certainty factor for the label information satisfies a threshold value of the primary pass label information.
(Appendix 10)
Further, the method includes a recording step and an extracting step,
The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other;
The extraction step extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information;
the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information;
The behavior judgment method described in Appendix 8 or 9, wherein the judgment process judges the image to be "passed" when at least one of the judgment based on the label information and the label information of the first passed label information and the judgment based on the label information and the label information of the second passed label information judges the image to be "passed".
(Appendix 11)
The behavior determination method according to claim 10, wherein the extraction step extracts the label information when the number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number.
(Appendix 12)
In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
The behavior determination method according to claim 10 or 11, wherein the setting step calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.
(Appendix 13)
13. The behavior determination method according to any one of appendices 10 to 12, wherein the condition preset in the setting step is at least one of the following (1) and (2).
(1) In the execution history data, label information other than the label information that matches the first pass label information is overlapped, and at least one of the average, median, and mode calculated from the respective certainties for the overlapping label information is equal to or greater than a predetermined value.
(2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.
(Appendix 14)
Further, the system includes a recommended action information presenting step and a token granting step,
The recommended action information presenting step includes presenting the recommended action information to a user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
The behavior determination method according to any one of appendices 8 to 13, wherein the token granting step grants token information linked to the recommended behavior information to the user when the determination step determines that the user is “passed.”
(Appendix 15)
A program for causing a computer to execute procedures including a setting procedure, an acquisition procedure, a detection procedure, and a determination procedure:
The setting step sets recommended action information including first pass label information;
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquiring step acquires an image captured by a user;
The detection step includes inputting the acquired image into a trained model and detecting label information of an object to be imaged that is included in the image;
The judgment step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the action indicated by the recommended action information was performed and an image related to the action was captured.
(Appendix 16)
The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection step further includes detecting the confidence level for the label information;
16. The program according to claim 15, wherein the judgment step further comprises judging the image as "pass" if the certainty factor for the label information satisfies a threshold value of the primary pass label information.
(Appendix 17)
Further, the method includes a recording step and an extracting step,
The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of a judgment made by the judgment step, in association with each other;
The extraction step includes extracting, from the execution history data, label information of the image for which the result of the judgment is “pass” and other than the label information that matches the first pass label information;
the setting step adding and setting, from the extracted label information, label information that satisfies a preset condition as secondary pass label information to the recommended action information;
The program described in Appendix 15 or 16, wherein the judgment procedure judges the image to be "passed" when at least one of the judgments based on the label information and the label information of the first passed label information, and the judgment based on the label information and the label information of the second passed label information, judges the image to be "passed."
(Appendix 18)
18. The program according to claim 17, wherein the extraction step extracts the label information when the number of the images judged to be "passed" in the execution history data is equal to or greater than a preset number.
(Appendix 19)
In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
The program described in Appendix 17 or 18, wherein the setting procedure calculates one of the average, median, or mode of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.
(Appendix 20)
20. The program according to any one of appendices 17 to 19, wherein the preset condition in the setting procedure is at least one of the following (1) and (2).
(1) In the execution history data, label information other than the label information that matches the first pass label information is overlapped, and at least one of the average, median, and mode calculated from the respective certainties for the overlapping label information is equal to or greater than a predetermined value.
(2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.
(Appendix 21)
Further, the method includes a procedure for presenting recommended action information and a procedure for granting tokens,
The step of presenting recommended action information includes presenting the recommended action information to a user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
The program according to any one of appendices 15 to 20, wherein the token granting step grants token information linked to the recommended behavior information to the user when the judgment step judges the user to be “passed.”
(Appendix 22)
A computer-readable recording medium having a program according to any one of appendices 15 to 21 recorded thereon.

According to the present invention, it is possible to determine whether a user has taken a specific action by image analysis. Therefore, the present invention is useful, for example, in fields where a specific action is recommended to a user.

REFERENCE SIGNS LIST 10 Behavior Determination Device 11 Setting Unit 12 Acquisition Unit 13 Detection Unit 14 Determination Unit 15 Recording Unit 16 Extraction Unit 17 Recommended Behavior Information Presentation Unit 18 Token Granting Unit 101 Central Processing Unit 102 Memory 103 Bus 104 Storage Device 105 Input Device 106 Display Device 107 Communication Device

Claims (8)

A setting unit, an acquisition unit, a detection unit, and a determination unit are included,
The setting unit sets recommended action information including first-stage pass label information,
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquisition unit acquires an image captured by a user,
The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image,
The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured;
Further, the recording unit and the extracting unit are included,
the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other;
The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information;
the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information;
The determination unit determines the label information based on the label information and the label information of the first pass label information, and
determining that the image is “passed” when at least one of the determinations based on the label information and the label information of the secondary pass label information is “passed”;
Behavior determination device. The behavior determining device according to claim 1 , wherein the extracting unit extracts the label information when the number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number. The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection unit further detects the confidence level for the label information,
The behavior determining device according to claim 1 , wherein the determining unit further determines the image as “pass” when the certainty factor for the label information satisfies a threshold value of the primary pass label information. In the execution history data, label information other than the label information that matches the first pass label information is duplicated,
4. The behavior determination device according to claim 3, wherein the setting unit calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label . Further, the system includes a recommended behavior information presenting unit and a token granting unit,
the recommended action information presentation unit presents the recommended action information to the user in association with token information;
The token information is information regarding a token granted to a user upon completion of the action,
The behavior determination device according to claim 1 , wherein the token granting unit grants, to the user, token information linked to the recommended behavior information, when the user is determined to be “passed” by the determining unit. The method includes a setting step, an acquisition step, a detection step, and a determination step,
The setting step sets recommended action information including first pass label information;
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquiring step acquires an image captured by a user,
The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image,
The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured;
Further, the method includes a recording step and an extracting step,
The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other;
The extraction step extracts, from the execution history data, label information of the image for which the judgment result is “pass” other than the label information that matches the first pass label information;
the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information;
The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and
A behavior judgment method in which each step is executed by a computer , and when at least one of the judgments based on the label information and the label information of the secondary pass label information is judged to be "pass", the image is judged to be "pass" . The first pass label information includes a confidence threshold for the label information,
The certainty is an index indicating a likelihood that the image is the subject,
The detection step further includes detecting the confidence level for the label information,
7. The behavior determination method according to claim 6 , wherein the determination step further comprises determining the image as "pass" when the confidence factor for the label information satisfies a threshold value of the primary pass label information. A program for causing a computer to execute procedures including a setting procedure, an acquisition procedure, a detection procedure, a determination procedure, a recording procedure, and an extraction procedure:
The setting step sets recommended action information including first pass label information;
the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action,
The first pass label information is label information indicating an imaging target related to an action recommended for a user,
The acquiring step acquires an image captured by a user;
The detection step includes inputting the acquired image into a trained model and detecting label information of an object to be imaged that is included in the image;
The judgment step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information,
"Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured;
The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of a judgment made by the judgment step, in association with each other;
The extraction step includes extracting, from the execution history data, label information of the image for which the result of the judgment is “pass” and other than the label information that matches the first pass label information;
the setting step adding and setting, from the extracted label information, label information that satisfies a preset condition as secondary pass label information to the recommended action information;
The judgment procedure judges the image to be "passed" when at least one of the judgments based on the label information and the label information of the first passed label information, and the judgment based on the label information and the label information of the second passed label information, judges the image to be "passed."

Images