JP7729466B2 - Image classification device, image classification method, and program - Google Patents

Description

This disclosure relates to technology for classifying captured images.

Photographs and videos of pets (hereafter referred to as "pets") and other subjects can amount to a huge volume, and among these vast amounts of photographs and videos, there may be some that do not suit the pet owner's preferences, such as when the pet is facing away from them. It is time-consuming for pet owners to sort through such a vast number of photographs and videos to find their favorites. For example, Patent Document 1 describes a device that identifies and classifies the movements of a subject from multiple image data of the subject.

Japanese Patent Application Laid-Open No. 2005-267604

However, even with Patent Document 1, it is difficult to classify photos and videos that pet owners prefer.

One objective of the present disclosure is to provide an image classification device that can classify images that a user prefers from multiple images.

In order to solve the above problem, in one aspect of the present disclosure, an image classification device includes:
image acquisition means for acquiring an image including a target subject;
a condition determination means for determining whether a predetermined condition for the target subject to be photographed is satisfied; and
an image classification means for classifying images of a target subject that show a predetermined state based on the images acquired by the image acquisition means and the determination result made by the condition determination means , using a machine-learned model of the relationship between an image of the target subject , a determination result as to whether a condition for occurrence of a predetermined state is satisfied, and the predetermined state of the target subject;
an output means for outputting the image and the classification result;
Equipped with
The condition determining means determines whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a photographer of the target subject .

In another aspect of the present invention, a computer implemented method for image classification comprises:
An image acquisition process is performed to acquire an image containing the target subject.
performing a condition determination process for determining whether a predetermined condition for the target subject to be photographed is satisfied;
performing an image classification process to classify images of the target subject that show the predetermined state based on the images acquired by the image acquisition process and the determination result determined by the condition determination process , using a model in which the relationship between the image in which the target subject is captured , the determination result of whether or not a condition for occurrence of a predetermined state is satisfied , and the predetermined state of the target subject has been machine-learned ;
performing an output process for outputting the image and the classification result;
The condition determination process determines whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a photographer of the target subject .

In yet another aspect of the invention, a program includes:
An image acquisition process is performed to acquire an image containing the target subject.
performing a condition determination process for determining whether a predetermined condition for the target subject to be photographed is satisfied;
performing an image classification process to classify images of the target subject that show the predetermined state based on the images acquired by the image acquisition process and the determination result determined by the condition determination process , using a model in which the relationship between the image in which the target subject is captured , the determination result of whether or not a condition for occurrence of a predetermined state is satisfied , and the predetermined state of the target subject has been machine-learned ;
performing an output process for outputting the image and the classification result;
The condition determination process causes a computer to execute a process of determining whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a person who is taking a photograph of the target subject .

This disclosure makes it possible to classify images that a user prefers from multiple images.

1 shows the overall configuration of an image classification system according to a first embodiment. FIG. 2 is a block diagram showing the configuration of a server and a user terminal. FIG. 2 is a block diagram showing the functional configuration of a server. FIG. 2 is a block diagram showing the functional configuration of the learning device. 1 is a flowchart of an image classification system. FIG. 10 is a block diagram showing a functional configuration of a first modified example of the first embodiment. FIG. 10 is a block diagram showing the functional configuration of an information processing apparatus according to a second embodiment. 10 is a flowchart of a process performed by an information processing apparatus according to a second embodiment.

First Embodiment
[Overall configuration]
1 shows the overall configuration of an image classification system to which an image classification device according to the present disclosure is applied. The image classification system 1 includes a server 200 and a user terminal 300 used by a pet owner. The server 200 is an example of an image classification device. The server 200 and the owner's user terminal 300 are capable of wireless communication.

As a basic operation, the server 200 acquires images showing a predetermined state of the pet based on video transmitted from the owner's user terminal 300. Specifically, the owner switches the user terminal 300 to continuous recording mode and captures video when playing with the pet P, for example. The user terminal 300 then transmits the captured video (hereinafter also referred to as "captured video") to the server 200. The server 200 extracts still images for each frame from the video captured by the user terminal 300 and classifies the images as showing the predetermined state of the pet through image analysis using AI (Artificial Intelligence). Here, an image showing the predetermined state of the pet (hereinafter also referred to as a "good shot") is an image of the pet that the pet owner finds pleasing, such as an image showing the pet's face, the pet jumping, or the pet playing. The server 200 then classifies still images (hereinafter also referred to as "extracted images") extracted from the video captured by the user terminal 300 as good shots or not, associates them with the owner, and stores them in a database. The owner then accesses the server 200 from the user terminal 300 or a terminal other than the user terminal 300 and checks only the good shots using a slideshow or the like. This allows the owner to capture images of their pet without missing a photo opportunity. Furthermore, by using smart glasses as the user terminal 300, the owner can capture good shots while interacting with their pet. Note that instead of smart glasses, other eyeglass-type wearable devices such as AR (Augmented Reality) glasses, MR (Mixed Reality) glasses, or VR (Virtual Reality) glasses may be used.

Note that images classified as GOOD shots are not limited to still images, but may also be videos. In this case, the server 200 extracts videos from the videos shot by the user terminal 300 at predetermined time intervals. The server 200 then classifies the videos as to whether they contain GOOD shots, and saves the extracted videos (also referred to as "extracted images") with the classification result of whether they are GOOD shots or not.

[server]
2A is a block diagram showing the configuration of the server 200. The server 200 mainly includes a communication unit 211, a processor 212, a memory 213, a recording medium 214, and a database (DB) 215.

The communication unit 211 sends and receives data to and from external devices. Specifically, the communication unit 211 sends and receives information to and from the owner's user terminal 300.

Processor 212 is a computer such as a CPU (Central Processing Unit) that controls the entire server 200 by executing pre-prepared programs. Processor 212 may also be a GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), DSP (Demand-Side Platform), ASIC (Application Specific Integrated Circuit), etc.

Memory 213 is composed of ROM (Read Only Memory), RAM (Random Access Memory), etc. Memory 213 is also used as working memory while various processes are being executed by processor 212. Memory 213 also temporarily stores a series of videos captured by user terminal 300 under the control of processor 212. These videos are stored in memory 213 in association with, for example, the owner's identification information and timestamp information.

The recording medium 214 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the server 200. The recording medium 214 records various programs executed by the processor 212.

Database (DB) 215 stores extracted images with classification results as to whether they are GOOD shots or not. DB 215 may include an external storage device such as a hard disk connected to or built into server 200, or may include a storage medium such as a removable flash memory. Note that instead of providing DB 215 in server 200, DB 215 may be provided on an external server, and extracted images with classification results as to whether they are GOOD shots or not may be stored in the server via communication.

In addition, the server 200 may be equipped with input units such as a keyboard and mouse for administrators to give instructions and input, and a display unit such as an LCD display.

[User terminal]
2B is a block diagram showing the internal configuration of a user terminal 300 used by a pet owner. The user terminal 300 is a terminal device such as smart glasses or a smartphone. The user terminal 300 includes a communication unit 311, a processor 312, a memory 313, a display unit 314, a camera 315, and a microphone 316.

The communication unit 311 sends and receives data to and from external devices. Specifically, the communication unit 311 sends and receives information to and from the server 200.

Processor 312 is a computer such as a CPU, and controls the entire user terminal 300 by executing a pre-prepared program. Processor 312 may also be a GPU, FPGA, DSP, ASIC, etc. Processor 312 transmits video captured by camera 315 to server 200 by executing a pre-prepared program.

Memory 313 is composed of ROM, RAM, etc. Memory 313 stores various programs executed by processor 312. Memory 313 is also used as working memory while processor 312 is executing various processes. Video captured by camera 315 is stored in memory 313 and then transmitted to server 200. Display unit 314 is, for example, an LCD display device, and displays videos captured by camera 315, extracted images of good shots stored on server 200, etc.

The camera 315 includes a camera that captures the user's field of vision (also called an "out-camera") and a camera that captures the user's eyeballs (also called an "eye camera"). The out-camera is mounted on the outside of the user terminal 300. The out-camera captures the user's field of vision, including subjects such as pets, and transmits the images to the server 200. This allows the server 200 to acquire images of subjects such as pets. The eye camera is mounted on the inside of the user terminal 300 to capture images of the user's eyeballs. The eye camera captures images of the user's eyeballs and transmits the images to the processor 312. The processor 312 detects the user's line of sight and other information based on the images of the user's eyeballs captured by the eye camera. This allows the user terminal 300 to acquire information such as the user's line of sight.

Microphone 316 collects the user's voice and surrounding sounds and transmits them to server 200. Server 200 can infer, for example, based on the user's voice or the pet's cries, whether the user has uttered a specific word or whether the user has given an instruction or command to the pet.

[Functional configuration]
3 is a block diagram showing the functional configuration of the server 200. Functionally, the server 200 includes an image acquisition unit 411 and an image classification unit 412.

Video captured by the user terminal 300 is input to the server 200. The video captured by the user terminal 300 is input to the image acquisition unit 411. The image acquisition unit 411 extracts still images or videos from the video captured by the user terminal 300 as extracted images. The image acquisition unit 411 outputs the extracted images to the image classification unit 412.

The image classification unit 412 classifies the extracted image acquired from the image acquisition unit 411 as a GOOD shot or not, using a pre-prepared image recognition model or the like. This image recognition model is a machine learning model that has been trained in advance to classify whether an image is a GOOD shot or not, and is hereinafter also referred to as the "image classification model." If the extracted image is classified as a GOOD shot by the image classification model, the image classification unit 412 attaches additional information to the extracted image indicating that it is a GOOD shot. On the other hand, if the extracted image is classified as a BAD shot by the image classification model, i.e., not a GOOD shot, the image classification unit 412 attaches additional information to the extracted image indicating that it is a BAD shot. A BAD shot is an image other than a GOOD shot, such as an image that does not show a pet's face. The image classification unit 412 outputs the extracted image with the attached additional information to DB215.

[Image classification model training]
Next, we will explain the training of the image classification model used by the image classification unit 412. The image classification model is generated by so-called supervised learning. Fig. 4 is a block diagram showing the training method of the image classification model, and includes training data 511 and a training device 512.

The training data 511 is image data (hereinafter also referred to as "teaching data") that has been pre-labeled as to whether it is a GOOD shot or not. Image data is labeled based on criteria such as whether a specific part of the pet is captured, or whether the pet is performing a specific action. A specific part of the pet refers to the pet's face, for example. For example, an image that shows the pet's face is labeled as a GOOD shot. On the other hand, an image that does not show the pet, an image in which the pet is facing away, or an image that shows only the pet's torso or legs is labeled as a BAD shot. Furthermore, a specific action of the pet refers to an eye-catching action of the pet, for example. For example, an image in which the pet is jumping or holding a tool in its mouth is labeled as a GOOD shot.

In addition, pet owners can sort multiple images of their pets into good shots and label them accordingly, and use the resulting labeled images as training data. This makes it possible to generate an image classification model that can classify images that better suit the pet owner's preferences.

Also, images of animals posted by pet owners or third parties on social networking services (SNS) can be collected and used as training data. In this case, images posted by pet owners or third parties on SNS are labeled as "good shots." This increases the amount of training data, making it possible to generate more accurate image classification models.

The learning device 512 learns the patterns of good shots based on the learning data 511 and outputs an image classification model as a trained model. This generates an image classification model that has learned the relationship between images containing pets and the state of the pet that corresponds to a good shot.

[Classification using image classification model]
The image classification unit 412 uses an image classification model to estimate whether an image is a good shot. Specifically, the image classification model estimates whether an input image is a good shot and calculates a score indicating the probability that the image is a good shot (referred to as a "good shot score") and a score indicating the probability that the image is a bad shot (referred to as a "bad shot score"). The image classification model calculates each score, for example, so that the sum of the good shot score and the bad shot score is "1." The image classification model then compares the good shot score and the bad shot score with a predetermined threshold TH and adopts the score greater than the threshold TH as the classification result. For example, for a certain image, the image classification model calculates a good shot score of "0.8" and a bad shot score of "0.2" and compares them with a predetermined threshold TH. If the threshold TH is "0.5," the image classification model estimates that the image is a good shot.

[Image classification processing]
Next, the image classification process for performing the above-described image classification will be described. Fig. 5 is a flowchart of the image classification process performed by the server 200. This process is realized by the processor 212 shown in Fig. 2 executing a program prepared in advance and operating as each element shown in Fig. 3.

First, the image acquisition unit 411 acquires a shot video from the user terminal 300. Then, the image acquisition unit 411 acquires an image (still image or video) from the shot video (step S11). Next, the image classification unit 412 classifies the image acquired by the image acquisition unit 411 as whether it is a GOOD shot or not (step S12). Specifically, the image classification unit 412 calculates a score indicating the probability that the image is a GOOD shot and a score indicating the probability that the image is a BAD shot. The image classification unit 412 compares each calculated score with a threshold TH and classifies the image as either a GOOD shot or a BAD shot.

Next, the image classification unit 412 assigns classification results to the images acquired by the image acquisition unit 411 and stores them in the database (DB) 215 (step S13). For example, the image classification unit 412 assigns a flag such as "1" to images classified as GOOD shots and "0" to images classified as BAD shots, and stores them in the DB 215. The image classification process then ends.

As a result, good shot images that match the user's preferences are extracted from the vast number of images taken by the user and stored in DB215 of server 200. The user can access server 200 and view the good shot images stored in DB215. The user can also download good shot images from server 200 and save them on a terminal device such as user terminal 300.

[Modification]
Next, a description will be given of modifications of the first embodiment. The following modifications can be applied to the first embodiment in appropriate combinations.
(Variation 1)
In the first embodiment, the server 200 classifies images based on extracted images extracted from a captured video. In addition to the above, the server 200 may determine whether a predetermined condition for occurrence is met and classify the images using the determination result. The predetermined condition for occurrence is a condition under which a good shot is estimated to have been captured, and is hereinafter also referred to as a "condition for occurrence of a good shot." The condition for occurrence of a good shot is determined based on, for example, biometric information or behavioral information of the photographer.

Specifically, Figure 6 shows the functional configuration of server 200a in variant example 1. As shown in the figure, in variant example 1, server 200a is provided with a condition determination unit 413. The condition determination unit 413 acquires the photographer's biometric information, etc., along with a timestamp, from the user terminal 300. Then, using a pre-trained model, the condition determination unit 413 determines whether the photographer's biometric information, etc., satisfies predetermined conditions, and outputs the determination result to the image classification unit 412.

Biometric information of the photographer includes gaze, voice, heart rate, and the like. The photographer's biometric information is acquired by the user terminal 300. The user terminal 300 may acquire the biometric information from a camera, microphone, sensor, or the like built into the user terminal 300, or may acquire the biometric information from an external device via wireless communication with the external device via Bluetooth (registered trademark) or Wi-Fi (registered trademark). Predetermined conditions include, for example, the photographer directing their gaze at a pet, the photographer speaking a voice louder than a predetermined threshold, the photographer uttering a predetermined phrase such as "like," or the photographer's heart rate rising above a predetermined threshold. If the photographer's biometric information satisfies the above conditions, it is estimated that a good shot was likely taken at that time and at times before and after that time. The condition determination unit 413 may determine that the photographer's biometric information satisfies the predetermined conditions not only at the time when it satisfied the predetermined conditions, but also at times before and after that time, and output the determination results to the image classification unit 412.

The condition determination unit 413 may also determine whether the conditions for generating a GOOD shot are met based on behavioral information about the photographer and the pet. For example, if the photographer gives a signal and the pet acts in accordance with the signal, or if the photographer gives an instruction or command and the pet acts in accordance with the instruction or command, the condition determination unit 413 determines that the conditions for generating a GOOD shot are met and outputs the determination result to the image classification unit 412. Note that behavioral information about the photographer and the pet may be obtained from a microphone, sensor, etc. installed in the user terminal 300, or from videos captured by the user terminal 300.

The image classification unit 412 classifies the extracted image as being a good shot or not based on the extracted image input from the image acquisition unit 411 and the judgment result input from the condition judgment unit 413. In this case, the image classification model used by the image classification unit 412 is a trained model that has been trained in advance to estimate whether or not an extracted image is a good shot based on the extracted image and the judgment result.

As described above, by taking into account the photographer's biometric information and behavioral information to classify whether a shot is good or not, it is possible to obtain with high accuracy an image of a pet captured at a moment that the photographer feels is good.

(Variation 2)
Based on the GOOD shots classified by the first embodiment, training data for retraining the image classification model may be created. Specifically, the pet owner determines whether or not the GOOD shots classified by the server 200 are necessary. The server 200 determines that images that the pet owner determines are necessary are GOOD shots. On the other hand, the server 200 determines that images that the pet owner determines are unnecessary are BAD shots and changes the labels. The server 200 then uses the image data of the GOOD shots and the image data of the BAD shots as training data and retrains the image classification model. This enables the server 200 to classify GOOD shots that better suit the pet owner's preferences.

(Variation 3)
In the first embodiment described above, the user terminal 300 sets the camera to continuous recording mode and transmits the captured video to the server 200. Alternatively, the user terminal 300 may start recording when a subject is captured on the camera, stop recording when the subject is no longer captured on the camera, and transmit the captured video from the start to the end of recording to the server 200. Specifically, the user terminal 300 captures images captured on the camera at predetermined intervals and transmits them to the server 200. The server 200 determines whether a pet is captured on the camera of the user terminal 300 based on a pre-created image recognition model or the like. If a pet is captured on the camera of the user terminal 300, the server 200 sets the user terminal 300 to recording mode and starts recording. Thereafter, if the pet is no longer captured on the camera of the user terminal 300, the server 200 ends the recording mode of the user terminal 300. This reduces the amount of data of the captured video transmitted from the user terminal 300 to the server 200.

The user terminal 300 may determine whether or not the pet is captured by the camera of the user terminal 300. In this case, the user terminal 300 may use a pre-created image recognition model or the like to determine whether or not the pet is captured by the camera of the user terminal 300. The user terminal 300 may then control the start and end of recording according to the determination result.

(Variation 4)
In the first embodiment described above, the server 200 classifies good shots based on videos shot with a pet as the subject, but the subject is not limited to a pet and may be another subject, such as a child, that often misses a photo opportunity.

(Variation 5)
In the first embodiment described above, information acquired by the user terminal 300 is basically transmitted as is to the server 200, and the server 200 classifies the good shots based on the received information. Instead, the user terminal 300 may perform a process for classifying the good shots and transmit the processing results to the server 200. Alternatively, the server 200 may not be used, and the process for classifying the good shots and the storage of the processing results may be performed by the user terminal 300. This reduces the communication load from the user terminal 300 to the server 200 and the processing load on the server 200. In these cases, the user terminal 300 is an example of an image classification device.

Second Embodiment
7 is a block diagram showing the functional configuration of an image classification device 50 according to the second embodiment. The image classification device 50 according to the second embodiment includes an image acquisition unit 51, an image classification unit 52, and an output unit 53.

Figure 8 is a flowchart of processing by the image classification device 50. The image acquisition means 51 acquires images containing target subjects (step S51). The image classification means 52 classifies the images into those containing images of the target subjects in a predetermined state, using a machine-learned model of the relationship between images containing the target subjects and a predetermined state of the target subjects (step S52). The output means 53 outputs the images and the classification results (step S53).

The image classification device 50 of the second embodiment makes it possible to easily classify images that users prefer.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
image acquisition means for acquiring an image including a target subject;
an image classification means for classifying, from the images, images showing a predetermined state of the target subject using a machine-learned model of the relationship between an image showing a target subject and a predetermined state of the target subject;
an output means for outputting the image and the classification result;
An image classification device comprising:

(Appendix 2)
2. The image classification device according to claim 1, wherein the image classification means classifies images that include a predetermined part of the target subject.

(Appendix 3)
3. The image classification device according to claim 1, wherein the image classification means classifies images in which the target subject is performing a predetermined action.

(Appendix 4)
a condition determination means for determining whether or not the occurrence condition of the predetermined state is satisfied;
The image classification device according to any one of claims 1 to 3, wherein the image classification means classifies the images based on the images and the determination results of the occurrence conditions.

(Appendix 5)
The image classification device according to claim 4, wherein the condition determination means determines whether the occurrence condition is satisfied based on a gaze direction of a photographer of the target subject.

(Appendix 6)
The image classification device according to claim 4 or 5, wherein the condition determination means determines whether the occurrence condition is satisfied based on a heart rate of a photographer of the target subject.

(Appendix 7)
The image classification device according to any one of claims 4 to 6, wherein the condition determination means determines whether the occurrence condition is met based on the voice of a photographer of the target subject.

(Appendix 8)
The image classification device according to any one of appendices 4 to 7, wherein the condition determination means detects the voice of the photographer and determines that the target subject has acted in response to the voice of the photographer as the occurrence condition.

(Appendix 9)
The image classification device according to any one of appendixes 1 to 8, wherein the image acquisition means starts acquiring images containing the target subject when the target subject is captured by the camera of the terminal device, and stops acquiring images containing the target subject when the target subject is no longer captured by the camera of the terminal device.

(Appendix 10)
10. The image classification device according to claim 1, further comprising a learning means for re-training the model using images, among the results output by the output means, for which a user has determined whether or not they are necessary, as training data.

(Appendix 11)
Acquire an image containing the target subject,
classifying, from the images, images showing a predetermined state of the target subject using a machine-learned model of the relationship between images showing the target subject and a predetermined state of the target subject;
The image classification method outputs the image and the result of the classification.

(Appendix 12)
Acquire an image containing the target subject,
classifying, from the images, images showing a predetermined state of the target subject using a machine-learned model of the relationship between images showing the target subject and a predetermined state of the target subject;
A recording medium on which a program for causing a computer to execute a process for outputting the image and the classification results is recorded.

The present disclosure has been described above with reference to embodiments and examples, but the present disclosure is not limited to the above embodiments and examples. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

200 Server 215 Database (DB)
300 User terminal 411 Image acquisition unit 412 Image classification unit 413 Condition determination unit 511 Learning data 512 Learning device

Claims (8)

image acquisition means for acquiring an image including a target subject;
a condition determination means for determining whether a predetermined condition for the target subject to be photographed is satisfied; and
an image classification means for classifying images of a target subject that show a predetermined state based on the images acquired by the image acquisition means and the determination result made by the condition determination means , using a machine-learned model of the relationship between an image of the target subject , a determination result as to whether a condition for occurrence of a predetermined state is satisfied, and the predetermined state of the target subject;
an output means for outputting the image and the classification result;
Equipped with
The condition determination means determines whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a photographer of the target subject . The image classification device described in claim 1, wherein the image classification means classifies images that include a specified part of the target subject. The image classification device described in claim 1 or 2, wherein the image classification means classifies images in which the target subject is performing a predetermined action. The image classification device according to claim 1 , wherein the condition determination means determines whether the occurrence condition is satisfied based on a gaze direction of a photographer of the target subject. The image classification device according to claim 1 , wherein the condition determination means determines whether the occurrence condition is met based on a voice of a photographer of the target subject. The image classification device according to claim 1 , wherein the condition determination means detects a voice of a photographer and determines that the target subject has taken an action in response to the voice of the photographer as the occurrence condition. 1. A computer-implemented method for image classification, comprising:
An image acquisition process is performed to acquire an image containing the target subject.
performing a condition determination process for determining whether a predetermined condition for the target subject to be photographed is satisfied;
performing an image classification process to classify images of the target subject that show the predetermined state based on the images acquired by the image acquisition process and the determination result determined by the condition determination process , using a model in which the relationship between the image in which the target subject is captured , the determination result of whether or not a condition for occurrence of a predetermined state is satisfied , and the predetermined state of the target subject has been machine-learned ;
performing an output process for outputting the image and the classification result;
The condition determination process determines whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a photographer of the target subject . An image acquisition process is performed to acquire an image containing the target subject.
performing a condition determination process for determining whether a predetermined condition for the target subject to be photographed is satisfied;
performing an image classification process to classify images of the target subject that show the predetermined state based on the images acquired by the image acquisition process and the determination result determined by the condition determination process , using a model in which the relationship between the image in which the target subject is captured , the determination result of whether or not a condition for occurrence of a predetermined state is satisfied , and the predetermined state of the target subject has been machine-learned ;
performing an output process for outputting the image and the classification result;
The condition determination process is a program that causes a computer to execute a process of determining whether or not the occurrence condition of the predetermined state is satisfied based on the heart rate of a person who is taking a photograph of the target subject .