JP7364348B2 - Service providing device, service providing system, and service providing method - Google Patents

Description

The present invention relates to a service providing device, a service providing system , and a service providing method.

2. Description of the Related Art Conventionally, there has been known a device that estimates a person's emotion and provides a service according to the estimated emotion (for example, see Patent Document 1). The device described in Patent Document 1 estimates and digitizes the individual emotions of a plurality of occupants in a vehicle, and uses the sum of the values of the individual emotions as the value of the overall emotion. This device provides services such as playing music from a speaker in accordance with the overall emotional value. For example, it has been disclosed that when there is no conversation in the car and the overall emotion value is low, a music service is provided that plays music that matches the quiet environment inside the car.

JP2018-63486A

The device described in Patent Document 1 determines a service in accordance with the estimated overall emotion value itself.
By the way, when trying to deal with changes in a person's emotions, it is effective to consider the factors that caused the changes in emotions. However, no method for taking such measures has been proposed so far.
The present invention has been made in view of the above background, and provides a service providing device, a payment information processing device, and a payment information processing device that are capable of responding to changes in a person's emotions by taking into consideration the factors that caused the change in emotions. The purpose is to provide a service provision method.

In order to achieve the above object, an acquisition unit acquires captured image data of the target range and audio data of sounds collected in the target range as data related to the target range; and the acquisition unit acquires a person detection unit that detects a plurality of people in the target range by extracting data related to humans from the data obtained by the acquisition unit; and extracting an image of the person from the data acquired by the acquisition unit; a process of obtaining an emotion estimate value that quantifies the personal emotion of each person by extracting the voice of the person from the data acquired by the department; and a process of calculating the target range from the emotion estimate values of the plurality of people. an emotion estimating unit that performs processing to obtain an overall emotional value indicating the overall emotional state of the person; a change in the estimated emotion value of each of the people calculated by the emotion estimating unit; and a change in the overall emotional value based on the change in the overall emotional value. , as the cause of the change in the emotional state of the entire target range or the emotional state of any one or more of the persons, a change in the environment that was the cause of the occurrence, or a factor person who was the person who was the cause of the occurrence. a factor estimating unit that estimates a factor estimating unit; and a service that is compatible with the change in the environment or the person who is the cause estimated by the factor estimating unit, and which works on a specific person or covers the entire target range. a service determining unit that determines a service to be provided to the entire target range or one or more of the people in the target range; Among the people, the service providing device estimates that the person whose emotion estimated value exceeds a predetermined threshold earlier than the other people is the factor person.

In the service providing device, the factor estimating unit is configured to detect the emotion when the estimated emotion value of any person exceeds a first threshold, when the sum of the estimated emotion values exceeds the first threshold, or when the estimated emotion value of any person exceeds a first threshold. The configuration may be such that when the amount of change in the sum of estimated values exceeds a second threshold, the cause of the change is estimated.

In the service providing device, the factor estimating unit uses a third threshold as the predetermined threshold, and the estimated emotion value reaches the third threshold earlier than other people among the plurality of people in the target range. The configuration may be such that the person who exceeds the change is estimated to be the factor person who caused the change.

In the service providing device, the factor estimating unit estimates that the person whose estimated emotion value first exceeds the third threshold is a provisional factor person, sets a detection period, and in the detection period, If the estimated emotion value of the person different from the temporary factor person exceeds a fourth threshold, the temporary factor person may be estimated as the factor person who is the cause of the change. .

In the service providing device, the factor estimating unit may detect the emotion when the estimated emotion value of the person different from the tentative factor person does not exceed a fourth threshold after the tentative factor person in the detection period. The configuration may be such that it is estimated that the emotion of the hypothetical factor person is in a suppressed state.

In the service providing device, the service determining unit may be configured to determine a service to be provided to the temporary factor person whose emotions are estimated to be in the suppressed state.

The service providing device includes an environment detection unit that acquires a detected value related to the environment in the target range, and the factor estimation unit detects a change in the estimated emotion value of one or more of the people and the environment detection unit. If it is estimated that the change in the environment is the cause of the change in the estimated emotion value, based on the acquired change in the detected value, the service determination unit determines the change in the environment in order to respond to the change in the environment. The configuration may be such that a service that covers the entire target range is determined.

The service providing device includes a behavior detection unit that detects conversations or actions of the plurality of people in the target range, and the factor estimation unit detects a change in the estimated emotion value of any one or more of the people, and The configuration may be such that it is estimated whether the conversation or behavior of any one or more of the persons is the cause of the change in the estimated emotion value, based on the conversation or behavior detected by the speech and behavior detection unit.

In the service providing device, the speech and behavior detection section is configured to detect conversations or actions of the plurality of people in the target range, and obtain a speech and behavior detection value that digitizes the detected conversations or actions, and the emotion estimation section a correction unit that corrects the estimated emotion value estimated by the behavior detection value based on the detected speech and behavior value, and the factor estimation unit estimates the cause of the change based on the estimated emotion value corrected by the correction unit. It may be a configuration.

The service providing device includes a service providing unit that provides the service determined by the service determining unit, and the service providing unit is configured to provide a service when one of the persons is estimated to be the factor person by the factor estimating unit. Furthermore, the configuration may be such that a service is provided to notify the person different from the cause person of the cause person.

In the service providing device, the service determining unit is configured to, when the factor estimating unit estimates that any of the persons is the factor person, respond to the factor person's hobbies or preferences. The configuration may be such that it decides to provide a service.

The above-mentioned service providing device includes a notification unit that reports information to one or more of the persons in the target range, and the notification unit is configured to determine whether one of the persons is the factor person by the factor estimation unit. The structure may also be such that, when it is estimated, guidance to urge the person different from the factor person to move away from the factor person may be provided.

Another aspect of achieving the above object includes: an audio acquisition unit that acquires speech voices of a plurality of people in a target range; a captured image acquisition unit that acquires captured images of one or more of the people; a digitization processing unit that digitizes the information acquired by the audio acquisition unit and the captured image acquisition unit to generate numerical information; a change in the numerical information generated by the digitization processing unit; and a change in the numerical information. an event information storage unit that stores correspondence with event information regarding an event that is a cause of the occurrence of the event, and the numerical information based on information acquired by the audio acquisition unit and the captured image acquisition unit during the detection target period, an event information acquisition unit that acquires the event information corresponding to the change from the event information storage unit when a change that satisfies the change occurs; and an event information acquisition unit that operates based on the event information acquired by the event information acquisition unit. An example of a service providing apparatus includes an action determining unit that determines the action, and an execution unit that executes the action determined by the action determining unit.

Another aspect of achieving the above object includes the service providing device according to any one of the above and a payment information processing device, wherein the payment information processing device is configured to provide services by the service providing device or to a registration unit that registers a person whose emotion is to be estimated; an evaluation acquisition unit that acquires the evaluation of the person regarding the provision of the service; A service providing system includes a consideration determination unit that determines consideration.

In the service providing system , the consideration determination unit determines the consideration for providing the service based on the evaluation acquired by the evaluation acquisition unit and the change situation in which the personal feelings of the person have changed in response to the provision of the service. It may also be configured to determine.

Another aspect of achieving the above object is a service providing method executed by a processor included in a service providing device, wherein the data regarding the target range includes captured image data obtained by capturing the target range and data collected in the target range. A plurality of people in the target range are detected by acquiring audio data of the sound by an acquisition unit and extracting data related to humans from the data acquired by the acquisition unit, and detecting a plurality of people in the target range, a process of obtaining an emotion estimate value that quantifies the personal emotion of each person by extracting an image of the person from the data and extracting the voice of the person from the data acquired by the acquisition unit; Then, a process is performed to obtain an overall emotional value indicating the entire emotional state of the target range from the estimated emotional values of the plurality of people, and a change in the estimated emotional value of each person and a change in the overall emotional value are calculated. Based on the change , as the cause of the change in the emotional state of the entire target range or the emotional state of any one or more of the persons, a change in the environment that was the cause of the occurrence, or a person who was the cause of the occurrence. A service that estimates a factor person, and is adapted to the change in the environment or the factor person, which is the estimated occurrence factor, and includes an action on a specific person or a process that changes the entire target range. , determines a service to be provided to the entire target range or one or more of the people in the target range, and determines a service to be provided to the entire target range or one or more of the people in the target range, and the estimated emotion value is earlier than other people among the plurality of people in the target range. There is a service providing method in which the person who exceeds a predetermined threshold is estimated to be the factor person.

According to the above-mentioned service providing device, a service can be provided to a person within the target range in response to a factor that causes a change in the overall emotion in the target range or a change in the emotion of an individual. Therefore, it is possible to deal with changes in emotions more effectively by considering the factors that caused the changes in emotions. For example, it can be expected to effectively promote emotional improvement and alleviation of specific emotional states. Similar effects can be obtained by using the above service providing method.
In addition, according to the payment information processing device, users and service providers can An appropriate price can be set for both parties.

FIG. 2 is an explanatory diagram showing an example of the usage state of the service providing device according to the first embodiment. FIG. 2 is a block diagram showing the functional configuration of a service providing device. 5 is a flowchart showing the operation of the service providing device. FIG. 3 is a schematic diagram showing an example of emotional values estimated by the service providing device. FIG. 3 is a schematic diagram showing an example of emotional values estimated by the service providing device. FIG. 2 is a block diagram showing the functional configuration of a service providing device. FIG. 2 is an explanatory diagram showing a schematic configuration of a service providing system according to a second embodiment. FIG. 2 is a block diagram showing the functional configuration of a service providing server and a payment information processing device.

Embodiments of the present invention will be described below with reference to the drawings.
[1. Overview of first embodiment]
FIG. 1 is a diagram showing an example of a usage state of a service providing apparatus 100 according to a first embodiment to which the service providing apparatus of the present invention is applied. FIG. 2 is a block diagram showing the functional configuration of the service providing device 100.

FIG. 1 shows a living room RM that can accommodate a plurality of people as an example of an environment in which the service providing device 100 is used. The service providing device 100 uses the internal space of the living room RM as a target range A, and performs emotion estimation and provides a service to a person within the target range A. In FIG. 1, the plurality of people in the living room RM are hereinafter referred to as users U1, U2, and U3.

The target range A does not need to be clearly separated from spaces other than the target range A, and for example, the target range A may be the entire living room RM or a part thereof. Further, it is also possible to consider the range in which the service providing device 100 functions as the target range A. For example, the range that can be imaged by a camera 111 (described later) may be set as target range A, the range that can be collected by microphone 112 may be set as target range A, and the range where a person can hear the sound output from speaker 113 is set as target range A. It may be range A.

As shown in FIG. 2, the service providing apparatus 100 includes a CPU (Central Processing Unit) 130 and a memory 160, and uses the functions of the CPU 130 to control each part of the service providing apparatus 100. The service providing device 100 includes a camera 111, a microphone 112, a speaker 113, a motor 114, an infrared communication section 115, and a wireless communication section 116, which operate under the control of a CPU 130.

The camera 111 is a digital camera that captures an image of a range including at least a part of the target range A and outputs captured image data.
The microphone 112 collects sounds in the target range A and outputs audio data.
The speaker 113 outputs audio under the control of the CPU 130.
The motor 114 operates a drive mechanism provided at the bottom of the main body of the service providing device 100. The drive mechanism of the service providing device 100 is a mechanism for moving the service providing device 100 in and out of the living room RM, and includes, for example, a plurality of wheels and gears that transmit the power of the motor 114 to the wheels. The motor 114 moves the service providing apparatus 100 by operating a drive mechanism under the control of the CPU 130.

The infrared communication unit 115 transmits or receives infrared signals. The infrared communication unit 115 controls the operation of the air conditioner 250, for example, by transmitting an infrared signal to the air conditioner 250 that performs air conditioning of the room RM. In this case, the service providing device 100 can start and stop the operation of the air conditioner 250, set the target temperature and air volume of the air conditioner, and the like. Further, the service providing apparatus 100 may obtain information regarding the room temperature of the living room RM detected by the air conditioner 250 by having the infrared communication unit 115 receive an infrared signal from the air conditioner 250.

Moreover, an opening/closing device (not shown) for opening/closing the window W may be provided in the living room RM. This opening/closing device may be configured to operate by receiving an infrared signal from the service providing device 100. In this case, the service providing device 100 can open and close the window W by transmitting an infrared signal to the opening/closing device.

The wireless communication unit 116 performs short-range wireless communication such as Wi-Fi (registered trademark) and Bluetooth (registered trademark). For example, the wireless communication unit 116 communicates with the biological information detection device 220 that the user U1 wears on his body or the user terminal device 230 that the user U1 possesses. The biometric information detection device 220 detects biometric information such as the pulse, blood pressure, blood oxygen concentration, body temperature, and perspiration of the user U1, and transmits the detection results to the service providing device 100.The user terminal device 230 is, for example, a personal It is a computer, smartphone, etc., and has functions such as voice calls and screen display.
In the example shown in FIG. 1, the user U1 uses the biological information detection device 220 and the user terminal device 230, but it is of course possible for the users U2 and U3 to use similar devices.
The wireless communication unit 116 may be configured to perform communication via a mobile wireless communication line or a public line network.

By reading and executing the control program 161, the CPU 130 controls the data acquisition unit 131, the person detection unit 132, the emotion estimation unit 133, the factor estimation unit 134, the service determination unit 135, the service provision unit 136, the environment detection unit 137, It functions as a behavior detection section 138, a correction section 139, a notification section 140, and a payment information processing section 150. The payment information processing unit 150 includes a registration unit 151, an evaluation acquisition unit 152, a price determination unit 153, and a payment processing unit 154, and each of these functional units is also realized by the CPU 130 executing the control program 161. .

The CPU 130 may be configured with another processor such as a microcomputer, or may be a device in which a processor, ROM (Read Only Memory), RAM (Random Access Memory), etc. are integrated.

In addition to the control program 161, the memory 160 stores captured image data 162, audio data 163, target information 164, user information 165, estimated emotion value 166, overall emotion value 167, detected behavior value 168, and payment information 170. . Further, a threshold storage section 169 is provided in the storage area of the memory 160.

The data acquisition unit 131 acquires data regarding the users U1, U2, and U3 who are in the target range A or the target range A. The data acquisition unit 131 acquires captured image data of the camera 111 and stores captured image data 162 in the memory 160. The data acquisition unit 131 acquires the audio data collected by the microphone 112 and stores the audio data 163 in the memory 160. Furthermore, the data acquisition unit 131 may acquire data regarding the room temperature and the like of the room RM from the air conditioner 250 using the infrared communication unit 115. Further, the data acquisition unit 131 may acquire data of the detection result of detecting the biometric information of the user U1 from the biometric information detection device 220. These data are also stored in memory 160.

The person detection unit 132 detects a person within the target range A based on the data acquired by the data acquisition unit 131. For example, the person detection unit 132 detects users U1, U2, and U3 by extracting human images from the captured image data 162.
The person detection unit 132 may perform individual identification of the detected person. For example, the user information 165 stored in the memory 160 stores users who use the service providing device 100 and identification information for identifying each user. The data acquisition unit 131 may refer to the user information 165 and perform processing to identify the user identification information of the person detected in the target range A.

The emotion estimation unit 133 estimates the emotions of each of the users U1, U2, and U3 based on the data acquired by the data acquisition unit 131. The emotions of each of the users U1, U2, and U3 are referred to as personal emotions. The emotion estimation unit 133 divides the emotions of each of the users U1, U2, and U3 into eight basic emotions (expectation, joy, acceptance, anxiety, surprise, sadness, disgust, and anger) using, for example, Plutchik's emotion sum. , and applied emotions that are a combination of two adjacent emotions. The emotion estimation unit 133 digitizes personal emotions according to the application to the emotion wheel.

For example, the emotion estimation unit 133 extracts an image of the face of the user U1 from the captured image data 162, determines the facial expression of the user U1 from the shape of the corners of the mouth, the shape of the eyebrows, the shape of the wrinkles on the face, etc., and determines the facial expression of the user U1 based on the determination result. Estimate user U1's personal feelings. The emotion estimating unit 133 estimates the personal emotion of the user U1 based on the voice data 163 in addition to the captured image data 162. For example, the voice data of the user U1 is extracted from the voice data 163, and the personal feelings of the user U1 are estimated based on the user U1's utterance content, voice accent, pronunciation, voice volume, and the like. Furthermore, the emotion estimation unit 133 estimates the personal emotion of the user U1 based on the biometric information of the user U1 that the data acquisition unit 131 acquired from the biometric information detection device 220. The emotion estimating unit 133 synthesizes the personal emotions of the user U1 estimated based on various data, and estimates the overall personal emotions.

In the process of quantifying personal emotions, the emotion estimation unit 133, for example, expresses a favorable emotion such as joy as a positive emotion with a positive value, and expresses an unfavorable emotion such as sadness as a negative emotion with a negative value. Expressed as Furthermore, the stronger the emotion, that is, the closer to the inside of the emotional circle, the larger the absolute value of the value representing the emotion.

For example, the emotion estimation unit 133 determines whether or not the user U1 is moving based on the captured image data 162 and the detection results of the biological information detection device 220, and when the user U1 is not moving, it is determined that the user U1 is sleeping. It may be determined that In this case, the emotion estimation unit 133 sets the personal emotion of the sleeping user U1 to 0, for example. The emotion estimating unit 133 estimates the personal emotions of the users U2 and U3 using the same process as that of the user U1, and converts them into numerical values. The emotion estimating unit 133 stores the numerical value of the personal emotion in the memory 160 as an estimated emotion value 166.

Furthermore, the emotion estimation unit 133 estimates the emotions of all the people in the target range A based on the personal emotions of each of the users U1, U2, and U3. The emotions of all the people in target range A are called overall emotions. The overall emotion represents the atmosphere of the target range A that the users U1, U2, and U3 feel. For example, the emotion estimation unit 133 takes the sum of the estimated emotion values 166 of the people in the target range A as the overall emotion of the target range A. That is, the total emotion of the target range A is the sum of the estimated emotion value 166 of the user U1, the estimated emotion value 166 of the user U2, and the estimated emotion value 166 of the user U3. The emotion estimation unit 133 determines the overall emotion of the target range A as a numerical value, and stores the determined numerical value in the memory 160 as the overall emotion value 167.

The estimated emotion value 166 is expressed as a positive value or a negative value, with the reference value being 0. When the estimated emotion value 166 is 0, this corresponds to a state where the personal emotion is calm or a case where the corresponding person is asleep as described above.

The factor estimation unit 134 monitors the estimated emotion value 166 and the overall emotional value 167, and detects changes in the estimated emotional value 166 or the overall emotional value 167 of any of users U1, U2, and U3 in the target range A. Estimate the cause of the occurrence.
For example, when the estimated emotion value 166 of any one of users U1, U2, or U3 changes, and when the overall emotional value 167 changes, the factor estimation unit 134 determines whether the change requires action or not. judge. If the change requires action, estimate the cause of the change. The factor estimating unit 134 refers to the threshold value stored in the threshold value storage unit 169 in the process of determining whether a change requires action or not. The threshold value storage unit 169 stores one or more preset threshold values.

The service determining unit 135 determines a service to be executed by the service providing device 100 in accordance with the cause of occurrence estimated by the factor estimating unit 134. For example, the service determining unit 135 determines to provide a service that alleviates changes in the estimated emotion value 166 and/or the overall emotion value 167. Alternatively, the service determining unit 135 determines to provide a service that improves the estimated emotion value 166 and/or the overall emotion value 167 by changing it in a positive direction.

The service providing unit 136 provides the service determined by the service determining unit 135. Details of the services provided by the service providing unit 136 will be described later.

The environment detection unit 137 detects the environment of the target range A based on the captured image data 162 and/or the audio data 163.
The speech and behavior detection unit 138 detects the speech and behavior of each of the users U1, U2, and U3 based on the captured image data 162 and/or the audio data 163. The speech and behavior detection unit 138 digitizes the speech and behavior of each of the users U1, U2, and U3 to generate speech and behavior detection values, and stores the speech and behavior detection values in the memory 160 as the speech and behavior detection values 168.

The correction unit 139 corrects the estimated emotion values 166 of the users U1, U2, and U3 estimated by the emotion estimation unit 133 based on the detected behavior values 168.

The notification unit 140 selects one or more of the users U1, U2, and U3 and notifies them of information. For example, the notification unit 140 notifies information by outputting directional audio from the camera 111 so that only a specific person among the users U1, U2, and U3 can hear the audio. Furthermore, the notification unit 140 notifies the user U1 of information by making a voice call or sending a text message to the user terminal device 230. When users U2 and U3 use the user terminal device 230, similar notifications can be made to users U2 and U3 as well. When the notification unit 140 is controlled by the service providing unit 136, the notification by the notification unit 140 is a type of service provision by the service providing unit 136.

The payment information processing unit 150 determines the compensation for the service provided by the service providing device 100 and executes payment.
The registration unit 151 registers a person to whom consideration is to be paid. Information regarding the person registered by the registration unit 151 is included in the user information 165, for example. In this embodiment, as an example, users U1, U2, and U3 are registered by the registration unit 151.

The evaluation acquisition unit 152 acquires the evaluation of the service provided by the service provision unit 136. For example, the evaluation acquisition unit 152 generates an evaluation based on changes in the estimated emotion value 166 and the overall emotion value 167 after the service providing unit 136 provides the service.

Further, for example, the evaluation acquisition unit 152 acquires evaluations by the users U1, U2, and U3 registered by the registration unit 151 after the service is provided by the service provision unit 136 or at each preset period. . In this case, the evaluation acquisition unit 152 may acquire evaluation information transmitted from a device other than the service providing device 100, such as the user terminal device 230. Furthermore, the evaluation acquisition unit 152 may perform dialogue processing with the users U1, U2, and U3 using the microphone 112 and the speaker 113, and acquire evaluations from the utterances of the users U1, U2, and U3.

The consideration determining unit 153 determines the consideration for providing the service by the service providing unit 136. Examples of the form of consideration include (1) a fixed amount system, (2) a fixed amount system, and (3) a success fee system. The payment information 170 includes information indicating the consideration charged by the service providing unit 136 and the manner in which the consideration is charged. The consideration determining unit 153 may be configured to be able to execute all of the following (1) fixed rate system, (2) fixed rate system, and (3) success fee system, or only a part of it. Good too.

(1) In the case of a flat-rate system, a fixed price is set for each month, year, or other time unit for the service provided by the service providing device 100. In this case, a fixed amount of compensation may be charged to each of the users U1, U2, and U3 registered by the registration unit 151, or a fixed amount may be charged to each group including multiple people, each location where the service providing device 100 is used, or , a fee may be charged for each service providing device 100.

(2) In the case of a quantitative system, a preset fee is charged each time the service providing unit 136 provides a service. In this case, before the service providing unit 136 provides the service, the registration unit 151 may confirm whether or not all or some of the users U1, U2, and U3 registered with the service can use the service. Quantitative compensation may be provided only to the contributing factors, which will be described later.

(3) In the case of a success fee type, the consideration determination unit 153 determines the consideration for the service provided by the service providing unit 136 based on the evaluation acquired by the evaluation acquisition unit 152. The consideration determined by the consideration determination unit 153 is included in the payment information 170. The consideration determining unit 153 updates the payment information 170 every time the consideration is determined. The consideration determined by the consideration determination unit 153 is imposed on all of the users U1, U2, and U3, or on the factor person described later.

Payment processing unit 154 executes payment based on payment information 170. For example, there is a configuration in which the service providing device 100 gives valuable points to the users U1, U2, and U3 registered by the registration unit 151. In this configuration, the user information 165 stores information indicating the points held by each of the users U1, U2, and U3. In this case, the payment processing unit 154 can receive payment by subtracting the points held by the users U1, U2, and U3.

The payment processing unit 154 may connect to a payment server operated by a financial institution or a payment service provider through the wireless communication unit 116 to perform payment processing for consideration. For example, the evaluation acquisition unit 152 may cause the payment server to perform a withdrawal from a bank account, a credit card payment, a payment using electronic money or virtual currency, or the like.

[2. Operation of service providing device of first embodiment]
FIG. 3 is a flowchart showing the operation of the service providing device 100.
The data acquisition unit 131 starts acquiring data (step S101). Thereafter, captured image data 162 and audio data 163 are stored in memory 160 at preset intervals. Since the captured image data 162 and audio data 163 are accumulated in the memory 160 within the set data capacity, the CPU 130 can refer to the captured image data 162 and audio data 163 that were acquired in the past.

The person detection unit 132 detects a person in the target range A (step S102), and specifies the detected person (step S103). The process in step S103 can be omitted.

The emotion estimating unit 133 estimates personal emotions based on the captured image data 162 and the audio data 163, and calculates an estimated emotion value 166 (step S104). In step S104, the estimated emotion value 166 is calculated using the data acquired by the data acquisition unit 131 for a predetermined period. Subsequently, the emotion estimation unit 133 calculates the overall emotion value 167 indicating the overall emotion of the target range A by calculating the sum of the estimated emotion values 166 (step S105).

The factor estimation unit 134 performs a process of determining an emotional change to be dealt with and estimating the cause of the change in the estimated emotion value 166 and/or the overall emotion value 167 (steps S106-S114). The factor estimation unit 134 refers to the first threshold TH1, second threshold TH2, third threshold TH3, and fourth threshold TH4 stored in the threshold storage unit 169. The first threshold TH1, the third threshold TH3, and the fourth threshold TH4 are values that are compared with the absolute values of the estimated emotion value 166 and the overall emotion value 167. Further, the second threshold TH2 is a threshold that is compared with the amount of change in the overall emotion value 167. The amount of change in the overall emotion value 167 will be described as, for example, an integral value of change in value over a set period, but may also be the difference between the maximum value and minimum value during the set period.

In a broad sense, the occurrence factor refers to a factor that causes a change in the estimated emotion value 166 or the overall emotion value 167. In a narrow sense, it refers to the factors that changed the overall emotional value 167. For example, if a change in the personal emotions of any of users U1, U2, or U3 affects the personal emotions of another person and changes the overall emotion of target range A, the change in the overall emotion of target range A The reason for this is one person. In this case, the occurrence factor is the person who caused the change, and can also be called an influencer. This person is called a factor person.

In the following description of the operation of the factor estimation unit 134, reference will be made to specific examples of emotional changes in FIGS. 4 and 5.
4 and 5 are schematic diagrams showing examples of the estimated emotion value 166 and the overall emotion value 167 estimated by the service providing device 100.
4(A), (B), (C) and FIG. 5(A), (B) are charts in which the horizontal axis represents the passage of time and the vertical axis represents the estimated emotion value 166. And the curve F1 in FIG. 5(A) shows the estimated emotion value 166 of the user U1. A curve F2 in FIG. 4(B) indicates the estimated emotion value 166 of the user U2, and a curve F3 in FIG. 4(C) indicates the estimated emotional value 166 of the user U3. Curve F5 in FIG. 5(B) shows another example regarding the estimated emotion value 166 of user U2.
FIG. 4(D) is a chart in which the horizontal axis represents the passage of time and the vertical axis represents the overall emotional value 167, and the curve FU indicates the overall emotional value 167. The curve FU is the sum of the estimated emotion values 166 shown by the curves F1, F2, and F3.
4(A)-(D) and FIGS. 5(A) and (B), positive values and negative values corresponding to the first threshold TH1, the third threshold TH3, and the fourth threshold TH4 are illustrated.

The outline of the emotion change illustrated in FIG. 4 is as follows.
The estimated emotion value 166 of the user U1 indicated by the curve F1 decreases due to some factor that occurred at time T1, and becomes a negative value. The estimated emotion value 166 of the user U1 fluctuates wildly within a negative value range, indicating that the user U1 is becoming emotional.
Changes in the personal feelings of the user U1 also affect the users U2 and U3 who are in the target range A together with the user U1. For example, the user U2 senses a change in the user's U1's personal feelings from the user's U1's facial expression, etc., and the user's U2's personal feelings change under the influence of this. Therefore, the estimated emotion value 166 of the user U2 indicated by the curve F2 decreases from time T2 when the state of the user U1 is detected, and becomes a negative value.
Furthermore, the overall emotion value 167 has changed due to changes in the personal emotions of the users U1 and U2. The overall emotion value 167 shown by the curve FU decreases following the change in the estimated emotion value 166 of the user U1 and the user U2, and becomes a negative value.
User U3 senses that the overall emotional value 167 of target range A has decreased at time T3. Therefore, the estimated emotion value 166 of user U3 has decreased to a negative value.

In steps S106 and S107, the factor estimation unit 134 determines whether an emotional change that should be dealt with has occurred. The criteria for determination are, for example, (1) that the absolute value of any one or more of the estimated emotion values 166 exceeds the first threshold TH1, (2) that the absolute value of the overall emotional value 167 exceeds the first threshold TH1, (3) The amount of change in the overall emotion value 167 exceeds the second threshold TH2.

Specifically, the factor estimation unit 134 first determines whether either the estimated emotion value 166 or the overall emotion value 167 exceeds the first threshold TH1 (step S106). If it is determined that the first threshold TH1 has not been exceeded (step S106; NO), the factor estimation unit 134 determines whether the amount of change in the overall emotion value 167 has exceeded the second threshold TH2 (step S107). In step S107, the factor estimation unit 134 determines a retrospective detection period with the latest overall emotion value 167 as the end point, and compares the amount of change in the overall emotion value 167 within the detection period with the second threshold TH2. If the amount of change in the overall emotion value 167 does not exceed the second threshold TH2 (step S107; NO), the process of the CPU 130 returns to step S104.

If either the estimated emotion value 166 or the overall emotion value 167 exceeds the first threshold TH1 (step S106; YES), and the amount of change in the overall emotion value 167 exceeds the second threshold TH2 (step S107; YES) ), the factor estimation unit 134 moves to step S108.

In step S108, the factor estimating unit 134 identifies the emotion estimate 166 that first exceeds the third threshold TH3 among the emotion estimate 166 of the person in the target range A, and selects the emotion estimate 166 that corresponds to the identified emotion estimate 166. The person is set as a temporary factor person (step S108). The temporary factor person may be a person corresponding to the estimated emotion value 166 that exceeds the first threshold TH1, or may be another person. Further, the third threshold TH3 is the same value as the first threshold TH1, or a value smaller than the first threshold TH1.

In the example of FIG. 4, since the absolute value of the estimated emotion value 166 of the user U1 exceeds the first threshold TH1 at time T11, the factor estimation unit 134 makes an affirmative determination in step S106. Further, since the absolute value of the estimated emotion value 166 of the user U1 exceeds the third threshold TH3 at time T12, the factor estimation unit 134 sets the user U1 as a temporary factor person in step S108. The tentative factor person is a person who is highly likely to be a factor person, and is a candidate for a factor person.

The factor estimation unit 134 sets a first detection period TE1 (step S109). The length of the first detection period TE1 is set in advance. The starting point of the first detection period TE1 is the time when an affirmative determination is made in step S106 or step S107, which is time T11 in the example of FIG. 4.

The factor estimation unit 134 determines whether the estimated emotion value 166 of the temporary factor person is a negative value in the first detection period TE1 (step S110). If the estimated emotion value 166 of the temporary factor person becomes a positive value in the first detection period TE1 (step S110; NO), the process of the CPU 130 returns to step S104.

If the estimated emotion value 166 of the temporary factor person maintains a negative value during the first detection period TE1 (step S110; YES), the factor estimation unit 134 sets a second detection period TE2 (step S111). The length of the second detection period TE2 is set in advance, and may be the same as or different from the first detection period TE1. The second detection period TE2 corresponds to an example of the detection period of the present invention. The starting point of the second detection period is the time when an affirmative determination is made in step S106 or step S107, which is time T11 in FIG. 4.

The factor estimation unit 134 determines whether the estimated emotion value 166 of the person excluding the temporary factor person exceeds the fourth threshold TH4 in the second detection period TE2 (step S112). If the estimated emotion value 166 of the person other than the temporary factor person does not exceed the fourth threshold TH4 (step S112; NO), the factor estimation unit 134 determines that the temporary factor person is suppressing emotions (step S112; NO). S113). That is, it is determined that the user U1 is in an emotion suppressed state.

The estimated emotion value 166 shown in FIG. 5(B) maintains a state in which it does not exceed the fourth threshold TH4 during the second detection period TE2. This example shows that the change in user U1's emotions has not propagated to user U2. In other words, the user U1 is in a state of suppressing emotional expression so that his emotions do not affect other people. In this case, although the target range A is not affected by the change in the user U1's emotions, it is desirable to alleviate the change in the user U1's emotions or to improve the emotions.

On the other hand, if the estimated emotion value 166 of a person other than the temporary factor person exceeds the fourth threshold TH4 in the second detection period TE2 (step S112; YES), the factor estimation unit 134 determines the temporary factor person as the factor person. (Step S114). In the example of FIG. 4, since the absolute value of the estimated emotion value 166 of the user U2 exceeds the fourth threshold TH4 at time T13, the user U1 is determined to be the factor person.
When the user U1 is a factor person, by alleviating the change in the user U1's emotion or improving the user U1's emotion, it is possible to improve the overall emotion of the target range A or to alleviate the change.

After the suppressed state is detected in step S113 and after the factor person is estimated in step S114, the service determining unit 135 determines a service (step S115). The service providing unit 136 executes the service determined by the service determining unit 135 (step S116).

The service determined by the service determining unit 135 and executed by the service providing unit 136 is, for example, a service that applies to the entire target range A or a service that applies to a specific person.

Examples of services that affect the entire target range A include actions that calm the atmosphere in the target range A. Specifically, the actions include playing music from the speaker 113 and telling a joke. If the service providing unit 136 is equipped with a mechanism for supplying odor to the target range A, it may discharge aromatic odor to the target range A to relieve feelings. These music, jokes, odors, etc. may be selected to suit the tastes of the person responsible for the occurrence. Information regarding the preferences of users U1, U2, and U3 is included in the user information 165, for example.

Further, if the factor estimating unit 134 can estimate that the environment or the change in the environment detected by the environment detecting unit 137 is the cause of the change in personal feelings, the service providing unit 136 eliminates or suppresses the factor. You may perform an action. For example, if it is determined that a large volume of noise from the window W is a cause of a change in personal feelings, the service providing unit 136 transmits a signal from the infrared communication unit 115 to the opening/closing device of the window W, and You may also perform control to close the . In this case, the speaker 113 may issue a voice prompting the user to close the window, and the overall emotion of the target area A may be improved by the voice.

An example of a service that works on a specific person is to output a voice saying, "Are you in trouble?" to the person who caused the outbreak or the person who is in a state of suppressed emotion.

Further, the service providing unit 136 may cause the speaker 113 to pronounce the voice message and cause the motor 114 to operate, and guide the cause person out of the room RM through the exit D. For example, the service providing device 100 moves toward the exit D by pronouncing a voice message to the person concerned, ``Let's go outside.'' Thereby, by physically separating the factor person from other people, it is possible to calm down the individual emotions of the factor person and the overall emotions of the target range A.

Further, the service providing unit 136 may cause the notification unit 140 to perform notification. For example, the notifying unit 140 notifies a person other than the person who caused the occurrence of the problem and outputs a message urging them to solve the problem by voice. In this case, if the factor estimating unit 134 can estimate the cause of the change in the personal emotion of the person, the cause of the change in the personal emotion may also be reported.

The service determining unit 135 prioritizes and determines services that improve the personal feelings of all persons in the target range A or alleviate changes in personal feelings. If it is difficult to implement such a service, or if it is determined that the service is ineffective, it is decided to implement the service for a limited number of people.

After providing the service, the evaluation acquisition unit 152 acquires the evaluation of the service, and the consideration determination unit 153 determines the consideration (step S117). Thereafter, the payment processing unit 154 executes the payment of the consideration (step S118), and this process ends. Note that the operations in steps S117 and S118 may not be performed immediately after the service providing unit 136 provides the service, but may be performed at preset intervals as described above.

As described above, the service providing device 100 to which the present invention is applied includes the person detection unit 132 that detects a plurality of people in the target range A, the emotion estimation unit 133 that estimates the personal emotions of each person, and the emotion estimation unit 133, a factor estimating unit 134 that estimates the cause of a change in the personal emotion of one or more persons or the overall emotion of the target range, based on the estimation result of the factor estimating unit 134; , a service determining unit 135 that determines a service to be provided to one or more persons in the target range A.

According to this configuration, a service can be provided to a person in the target range A in response to a factor that causes a change in the overall emotion or a change in the individual emotion in the target range A. Therefore, it is possible to deal with changes in emotions more effectively by considering the factors that caused the changes in emotions. For example, it can be expected that the estimated emotion value 166 and the detected behavior value 168 will be improved, and that a specific emotional state will be alleviated effectively. Similar effects can be obtained by using the service providing method executed by the service providing device 100.

Further, the payment information processing unit 150 corresponds to an example of the payment information processing device of the present invention.
The payment information processing unit 150 acquires the registration unit 151 that registers the person to whom the service providing device 100 provides the service or estimates the personal feelings, and the evaluations of the users U1, U2, and U3 regarding the provision of the service. It includes an evaluation acquisition section 152 and a consideration determination section 153 that determines the consideration for providing the service based on the evaluation acquired by the evaluation acquisition section 152.
With this configuration, it is possible to set an appropriate compensation for the use of the service providing apparatus 100 for both the users U1, U2, and U3 and the service provider.

Further, the consideration determination unit 153 may determine the consideration for providing the service based on the evaluation acquired by the evaluation acquisition unit 152 and the change situation in which the personal feelings of the person have changed in response to the provision of the service. For example, in the example of FIG. 4, as a result of the service being provided at time T12, curves F1, F2, F3, and FU all rise and change from negative values to positive values. The consideration determination unit 153 may determine the effectiveness of the service and determine the consideration based on the estimated emotion value 166 and the overall emotion value 167 of the users U1, U2, and U3. In this case, the price of the service can be determined appropriately by reflecting not only the subjective evaluations of the users U1, U2, and U3 but also the objective effects.

In the service providing device 100, the emotion estimation unit 133 calculates the emotion estimation value 166 representing the personal emotion of each person, and the factor estimation unit 134 calculates the emotion estimation value 166 representing the personal emotion of each person, and the factor estimation unit 134 calculates the emotion estimation value 166 when the emotion estimation value 166 of any person exceeds the first threshold TH1. , when the total sum of estimated emotion values 166 exceeds the first threshold TH1, or when the amount of change in the total sum of estimated emotion values 166 exceeds the second threshold TH2, the cause of the change is estimated. According to this configuration, a change in emotion can be appropriately detected based on the digitized overall emotion of the target range A and the individual emotions of the users U1, U2, and U3.

The factor estimation unit 134 determines that among the plurality of persons in the target range A, the person whose estimated emotion value 166 exceeds the third threshold TH3 earlier than the other persons is the factor person who is the cause of the change. presume. According to this configuration, it is possible to quickly estimate a factor person who has influenced another person's personal feelings.

The factor estimation unit 134 sets a second detection period TE2, estimates that the person whose emotion estimated value 166 exceeds the third threshold TH3 for the first time is a temporary factor person, and sets the temporary factor person in the second detection period TE2. When the estimated emotion value 166 of a person different from the person exceeds the fourth threshold TH4, the temporary factor person is estimated as the factor person who is the cause of the change.
According to this configuration, by estimating a temporary factor person and then determining whether or not the temporary factor person influenced the personal feelings of another person, the factor person can be estimated accurately and quickly. can.

The factor estimation unit 134 determines that the emotion of the temporary factor person is different from the temporary factor person when the estimated emotion value 166 of a person different from the temporary factor person does not exceed the fourth threshold TH4 after the temporary factor person in the second detection period TE2. It is presumed to be in a suppressed state. According to this configuration, it is possible to estimate a person who is suppressing emotional expression even though his or her emotional state is changing.

The service determining unit 135 determines a service to be provided to the hypothetical factor person whose emotions are estimated to be in a suppressed state. According to this configuration, by providing a service to a person whose emotional state is changing but suppressing emotional expression, it is possible to improve the emotional state.

The service providing device 100 includes a service providing unit 136 that provides the service determined by the service determining unit 135. Provides a service that notifies a person different from the cause person of the cause person when the person is estimated. According to this configuration, the overall emotion value 167 and the estimated emotion value 166 can be improved by conveying the cause of the change in the overall emotion value 167 to a person who is unaware that the overall emotion value 167 has changed due to the influence of the factor person. can be achieved.

The service determining unit 135 is configured to provide, to the factor person, a service corresponding to the factor person's hobbies or preferences when the factor estimation unit 134 estimates that any person is the factor person who is the cause of the change. Determine. According to this configuration, it is possible to effectively deal with the personal emotions of the factor person, and it is possible to improve the overall emotion value 167 and the estimated emotion value 166 more effectively.

The service providing device 100 includes a notification unit 140 that reports information to one or more people within the target range A.
When the factor estimating unit 134 estimates that a person is a factor person who is the cause of a change, the notification unit 140 notifies a person different from the factor person of guidance urging them to move away from the factor person. It's okay. For example, when the user U1 is the factor, the notification unit 140 may notify the users U2 and U3 that they will be leaving the user U1. In this case, it is possible to suppress deterioration of personal feelings of persons other than the factor person.

The service providing device 100 includes a behavior detection unit 138 that detects conversations or actions of a plurality of people in the target range A. The factor estimating unit 134 determines whether the conversation or behavior of any one or more people is the estimated emotion value 166 based on the change in the estimated emotion value 166 of any one or more people and the conversation or behavior detected by the behavior detection unit 138. It is also possible to estimate whether or not this is the cause of the change.

In this configuration, the behavior detection unit 138 may be configured to detect conversations or actions of a plurality of people in the target range A, and obtain a behavior detection value 168 that is a numerical value of the detected conversations or actions. The service providing device 100 includes a correction unit 139 that corrects the estimated emotion value 166 estimated by the emotion estimation unit 133 based on the detected behavior value 168. The factor estimation unit 134 estimates the cause of the change based on the estimated emotion value 166 corrected by the correction unit 139.

For example, in the operation shown in FIG. 3, the factor estimation unit 134 may analyze the words and actions of the users U1, U2, and U3 to determine the estimated emotion value 166 and the overall emotion value 167. Specifically, before the process of estimating the temporary factor person in step S108, the behavior detection unit 138 generates the behavior detection values 168 of the users U1, U2, and U3, and the correction unit 139 corrects the estimated emotion value 166. You may. The speech and behavior detection value 168 is an index that indicates the magnitude of the speech and behavior of the users U1, U2, and U3, and includes the volume of the voice of the user, the length of the statement, the magnitude of the motion, the volume of the sound generated by the motion, etc. The larger the speech and behavior detection value 168 is. The speech and behavior detection value 168 is calculated for each of the users U1, U2, and U3. Therefore, it can be said that the speech and behavior detection values 168 of each of the users U1, U2, and U3 indicate the magnitude of the influence that the speech and behavior of the users U1, U2, and U3 have had on the target range A. When the detected behavior value 168 of any one of users U1, U2, and U3 exceeds the threshold, the threshold storage unit 169 corrects the estimated emotion value 166 after the detected behavior value 168 exceeds the threshold. For example, correction is made to increase the change in the estimated emotion value 166 for a person different from the person corresponding to the detected speech and behavior value 168 that exceeds the threshold.

In this case, the emotion estimation unit 133 recalculates the overall emotion value 167 based on the emotion estimation value 166 corrected by the correction unit 139. The factor estimation unit 134 re-executes the process from step S106 on the corrected emotion estimated value 166 and overall emotion value 167.
According to this configuration, it is possible to greatly evaluate the influence of the words and actions of any of the users U1, U2, and U3 on the estimated emotion value 166 of another person, and estimate the cause of the occurrence and determine the service. .

The service providing device 100 includes an environment detection unit 137 that acquires detected values related to the environment of the target range A, and the factor estimation unit 134 detects changes in the estimated emotion value 166 of one or more people and the environment detection unit 137. Based on the change in the acquired detected value, it may be estimated whether a change in the environment is a cause of the change in the estimated emotion value 166. In this case, it is possible to improve the estimated emotion value 166 and the overall emotion value 167 in response to changes in the estimated emotion value 166 due to changes in the environment.

From another perspective, the service providing device 100 can have the configuration shown in FIG. 6.
FIG. 6 is a block diagram showing the functional configuration of the service providing apparatus 100 when viewed from another perspective. The CPU 130 includes an audio acquisition section 191 , a captured image acquisition section 192 , a digitization processing section 193 , an event information acquisition section 194 , an operation determination section 195 , and an execution section 196 . The memory 160 also includes an event information storage section 197.

The service providing device 100 in FIG. 6 includes a voice acquisition unit 191 that acquires speech voices of a plurality of people in a target range A, a captured image acquisition unit 192 that acquires captured images of one or more people, and a voice acquisition unit 192 that acquires captured images of one or more people. A digitization processing section 193 that digitizes the information acquired by the acquisition section 191 and the captured image acquisition section 192 to generate numerical information, a change in the numerical information generated by the digitization processing section 193, and an occurrence of a change in the numerical information. The event information storage unit 197 that stores the correspondence with event information regarding the event that is the cause, and the numerical information based on the information acquired by the audio acquisition unit and the captured image acquisition unit during the detection target period, have changed to meet the detection conditions. an event information acquisition unit 194 that acquires event information corresponding to the change from the event information storage unit 197; an action determination unit 195 that determines an action based on the event information acquired by the event information acquisition unit 194; and an execution unit 196 that executes the operation determined by the determination unit.

The audio acquisition unit 191 acquires the spoken voice collected by the microphone 112 , and the captured image acquisition unit 192 acquires image data captured by the camera 111 . These correspond to the data acquisition section 131 described above. The digitization processing unit 193 digitizes the information acquired by the data acquisition unit 131 to generate numerical information such as an estimated emotion value 166 and an overall emotion value 167. The digitization processing section 193 corresponds to the emotion estimation section 133.

The event information indicates the cause of a change in the estimated emotion value 166 and/or the overall emotion value 167. The event information storage unit 197 stores information that associates the manner of change in the estimated emotion value 166 and/or the overall emotion value 167 with the cause of occurrence, and the event information acquisition unit 194 estimates the cause of occurrence. That is, the event information acquisition unit 194 and the event information storage unit 197 correspond to an example in which the operation of the factor estimation unit 134 is executed based on data stored in the memory 160.
The operation determining unit 195 corresponds to the service determining unit 135, and the executing unit 196 corresponds to the service providing unit 136. These operations are as described above.

Furthermore, the configuration of each unit included in the CPU 130 may be configured using artificial intelligence (AI). For example, the functions of the person detection unit 132, the emotion estimation unit 133, the factor estimation unit 134, the service determination unit 135, the environment detection unit 137, and the correction unit 139 may be performed by AI installed in the service providing device 100. The AI implemented in the service providing device 100 may be a program and data that constitute a trained model. Further, the AI installed in the service providing device 100 may be configured to continue learning along with the operation of the service providing device 100.

Further, the service providing device 100 described above is not limited to a device that independently executes each function. An example in which a plurality of devices cooperate to function in the same manner as the service providing device 100 will be described as a second embodiment.

[3. Second embodiment]
FIG. 7 is an explanatory diagram showing a schematic configuration of the service providing system 1 of the second embodiment, and also shows the functional configuration of the service providing device 100a installed in the target range A. FIG. 8 is a block diagram showing the functional configurations of the service providing server 300 and the payment information processing device 400. The service providing system 1 as a whole corresponds to the service providing device of the present invention.

Service providing device 100 is connected to communication network 500 via communication device 200 . The communication network 500 may be a wide area network constituted by the Internet or a public line network, or may be a specific area network such as a LAN (Local Area Network).

A service providing server 300 and a payment information processing device 400 are connected to the communication network 500 . The service providing device 100a, the service providing server 300, and the payment information processing device 400 that constitute the service providing system 1 are connected to each other so as to enable data communication.

The service providing device 100a, like the service providing device 100, is installed in the living room RM corresponding to the target range A. The service providing device 100a includes a CPU 130a as a processor similar to the CPU 130. The CPU 130a has the functions of a data acquisition section 131, a service provision section 136, and a notification section 140, and further includes a communication control section 142.

The service providing server 300 also includes a CPU 310 and a memory 330. The CPU 310 is a computer that executes a control program 331, and by the functions of the control program 331, a communication control unit 311, a person detection unit 312, an emotion estimation unit 313, a factor estimation unit 314, a service determination unit 315, an environment detection unit 317, a behavior detection section 318, and a correction section 319.

Similarly to the data acquisition unit 131, the communication control unit 142 acquires data from the camera 111, the microphone 112, the infrared communication unit 115, and the like. The communication control unit 142 transmits the acquired data to the service providing server 300.
The communication control unit 311 of the service providing server 300 receives the data transmitted by the service providing device 100a, and stores captured image data 332 and audio data 333.

The person detection unit 312 functions similarly to the person detection unit 132 of the service providing device 100 shown in FIG. 2. Emotion estimation section 313 functions similarly to emotion estimation section 133, and factor estimation section 314 functions similarly to factor estimation section 134. Further, the service determining unit 315 functions similarly to the service determining unit 135. The environment detection unit 317 functions similarly to the environment detection unit 137, and the behavior detection unit 318 functions similarly to the behavior detection unit 138. Further, the correction section 319 functions similarly to the correction section 139.

The captured image data 332 and audio data 333 stored in the memory 330 correspond to the captured image data 162 and audio data 163 of the service providing apparatus 100 shown in FIG. 2, respectively. Target information 334 corresponds to target information 164. The user information 335 corresponds to the user information 165, the estimated emotion value 336 corresponds to the estimated emotion value 166, the overall emotional value 337 corresponds to the overall emotional value 167, and the detected behavior value 338 corresponds to the detected behavior value 168. The threshold storage section 339 is similar to the threshold storage section 169.

Payment information processing device 400 includes a CPU 410 and a memory 430. The CPU 410 is a computer that executes a control program 431, and operates as a communication control unit 411, a registration unit 412, an evaluation acquisition unit 413, a consideration determination unit 414, and a payment processing unit 415 by the functions of the control program 431. . The memory 430 also stores payment information 432.

The registration unit 412 functions similarly to the registration unit 151 of the service providing device 100 shown in FIG. 2. The evaluation acquisition unit 413 functions in the same manner as the evaluation acquisition unit 152, the consideration determination unit 414 functions in the same manner as the consideration determination unit 153, and the payment processing unit 415 functions in the same manner as the payment processing unit 154. It is something. Payment information 432 corresponds to payment information 170 stored by service providing device 100.

The communication control unit 311 transmits data regarding the service determined by the service determination unit 315 to the service providing device 100a.
The communication control unit 142 receives data transmitted from the service providing server 300 and causes the service providing unit 136 to execute the service determined by the service determining unit 315. Furthermore, if the data received from the service providing server 300 is data that instructs the notification unit 140 to make a notification, the communication control unit 142 causes the notification unit 140 to perform the notification.

The communication control unit 311 transmits data regarding the service determined by the service determination unit 315 to the payment information processing device 400. Further, the communication control unit 142 transmits data regarding the execution status of the service executed by the service providing unit 136 to the payment information processing device 400.

Based on the data transmitted from the service providing device 100a and the service providing server 300, the communication control unit 411 causes the evaluation acquisition unit 413 to acquire the evaluation, causes the consideration determination unit 414 to determine the consideration, and causes the payment processing unit 415 to make the payment. Let it run.

According to the service providing system 1 of the second embodiment, similarly to the service providing device 100, a person in the target range A is can provide services to Therefore, it is possible to deal with changes in emotions more effectively by considering the factors that caused the changes in emotions. For example, it can be expected that the personal feelings of the users U1, U2, and U3, the overall feelings of the target range A can be improved, or the relaxation of a specific emotional state can be effectively encouraged. Similar effects can be obtained by using the service providing method executed by the service providing system 1.

[4. Other embodiments]
In the embodiment described above, if the factor estimation unit 134 cannot estimate the factor person in the process of steps S108 to S114 in FIG. . For example, this applies when the estimated emotion values 166 of a plurality of people simultaneously exceed the first threshold TH1 or when they simultaneously exceed the third threshold TH3. The factor estimating unit 134 makes a voice call or sends a text message to the user terminal device 230, or causes the speaker 113 to produce a voice. Using these methods, the factor estimating unit 134 outputs a message asking the users U1, U2, and U3 if something happened, and the data acquiring unit 131 acquires and analyzes data regarding their responses. , estimate the cause of the occurrence. In this case, the data acquisition unit 131 can collect more information and estimate the cause of the occurrence.

Further, as shown in FIG. 5, when the estimated emotion value 166 of the user U2 maintains a positive value while the estimated emotional value 166 of the user U1 is a negative value, it is determined that the user U1 is in a suppressed state. Alternatively, it is possible that user U1 is changing his personal feelings due to circumstances unrelated to users U2 and U3. In such a case as well, the factor estimating unit 134 may influence the users U1, U2, and U3 to determine the contributing person or the occurrence factor.

Furthermore, in the above embodiment, the process that corresponds to a change in which the estimated emotion value 166 and the overall emotion value 167 become negative values has been described, but the process that corresponds to a change in which the overall emotion value 167 becomes a positive value is explained. It's okay. For example, the service providing unit 136 may perform an operation such as playing music or having the service providing device 100 join the conversation of the users U1, U2, and U3 so as to further increase the overall emotional value 167 of the target range A.

Further, although the service providing device 100 and the service providing server 300 of the above embodiment are configured to include the environment detection units 137 and 317, they may be configured without these. The same applies to the speech and behavior detection units 138 and 318 and the correction units 139 and 319.

Furthermore, although the service providing apparatuses 100 and 100a of the above embodiments are configured to include the notification section 140, they may be configured not to include the notification section 140. The same applies to the motor 114 and the infrared communication section 115. For example, the service providing devices 100 and 100a may be devices fixedly installed in the target range A, or may be so-called smart speakers.

Further, although the service providing device 100 and the payment information processing device 400 of the above embodiment are configured to include the evaluation acquisition units 152 and 413, they may be configured without these.

Note that, in order to facilitate understanding of the present invention, FIGS. 2, 6, 7, and 8 show the functional configurations of the service providing device 100 and the service providing system 1 divided by main processing contents. It is a schematic diagram, and these configurations may be configured by other divisions. Furthermore, the processing of each component may be executed by one hardware unit or by multiple hardware units. Furthermore, the processing of each component may be executed by one program or by multiple programs.

1... Service providing system (service providing device), 100, 100a... Service providing device, 130, 130a... CPU, 131... Data acquisition section, 132, 312... Person detection section, 133, 313... Emotion estimation section, 134, 314 ...Factor estimation section, 135, 315...Service determination section, 136...Service provision section, 137, 317...Environment detection section, 138, 318...Speech and behavior detection section, 139, 319...Correction section, 140...Notification section, 142, 311 , 411... Communication control section, 150... Payment information processing section, 151, 412... Registration section, 152, 413... Evaluation acquisition section, 153, 414... Consideration determining section, 154, 415... Payment processing section, 160... Memory, 161 ... Control program, 162, 332... Captured image data, 163, 333... Audio data, 164, 334... Target information, 165, 335... User information, 166, 336... Emotion estimated value, 167, 337... Overall emotion value, 168, 338... speech and behavior detection value, 169, 339... threshold storage unit, 170, 432... payment information, 191... audio acquisition unit, 192... captured image acquisition unit, 193... digitization processing unit, 194... event information acquisition unit, 195...Operation determining unit, 196...Executing unit, 197...Event information storage unit, 300...Service providing server, 310...CPU, 311...Communication control unit, 330...Memory, 331...Control program, 400...Payment information processing device , 410...CPU, 411...Communication control unit, 430...Memory, 431...Control program, 500...Communication network, A...Target range.

Claims (13)

an acquisition unit that acquires captured image data of the target range and audio data of sounds collected in the target range as data related to the target range;
a person detection unit that detects a plurality of people in the target range by extracting data related to humans from the data acquired by the acquisition unit;
Emotions that quantify the personal feelings of each person by extracting an image of the person from the data acquired by the acquisition unit and extracting the voice of the person from the data acquired by the acquisition unit. an emotion estimation unit that performs a process of calculating an estimated value, and a process of calculating an overall emotional value indicating the emotional state of the entire target range from the emotion estimated values of the plurality of people;
Based on the change in the estimated emotion value of each person calculated by the emotion estimation unit and the change in the overall emotion value , determine the emotional state of the entire target range or the emotional state of any one or more of the people. a factor estimating unit that estimates, as the cause of the change, a change in the environment that is the cause of the change, or a person who is the cause of the change;
A service that is compatible with the change in the environment or the person who is the cause of the occurrence estimated by the factor estimator , and includes an action on the specific person or a process that changes the entire target range. a service determining unit that determines a service to be provided to the entire range or one or more of the persons in the target range,
The factor estimating unit estimates, among the plurality of persons in the target range, the person whose estimated emotion value exceeds a predetermined threshold earlier than the other persons, to be the factor person. Device. The factor estimation unit is configured to determine whether the estimated emotion value of any person exceeds a first threshold, the sum of the estimated emotion values exceeds the first threshold, or the amount of change in the sum of the estimated emotion values. The service providing device according to claim 1, wherein the cause of the change is estimated when the change exceeds a second threshold. The factor estimation unit uses a third threshold as the predetermined threshold, and selects a person whose estimated emotion value exceeds the third threshold earlier than other people among the plurality of people in the target range. The service providing device according to claim 2, wherein is estimated to be the factor person who caused the change. The factor estimation unit estimates that the person whose emotion estimated value first exceeds the third threshold is a provisional factor person,
Set the detection period,
If the estimated emotion value of the person different from the tentative factor person exceeds a fourth threshold during the detection period, the tentative factor person is estimated as the factor person who is the cause of the change. The service providing device according to item 3. The factor estimating unit determines the emotion of the tentative factor person when the estimated emotion value of the person different from the tentative factor person after the tentative factor person during the detection period does not exceed a fourth threshold. The service providing device according to claim 4, wherein the service providing device estimates that the is in a suppressed state. The service providing device according to claim 5, wherein the service determining unit determines a service to be provided to the temporary factor person whose emotions are estimated to be in the suppressed state. comprising an environment detection unit that acquires a detected value regarding the environment of the target range,
The factor estimating unit determines whether the change in the environment is the cause of the change in the estimated emotion value based on the change in the estimated emotion value of any one or more of the people and the change in the detected value acquired by the environment detection unit. If it is estimated that this is the cause of the change, the service determining unit determines a service that covers the entire target range in order to respond to the change in the environment. The service provision equipment described in Section 1. comprising a service providing unit that provides the service determined by the service determining unit,
The service providing unit performs a service of notifying the person different from the factor person of the factor person when any of the persons is estimated to be the factor person by the factor estimation unit. 8. The service providing device according to claim 7. The service determining unit determines to provide the factor person with a service corresponding to the hobby or preference of the factor person, when any of the persons is estimated to be the factor person by the factor estimating unit. The service providing device according to any one of claims 1 to 8. comprising a notification unit that notifies information to one or more of the persons in the target range,
The notification unit notifies the person, who is different from the cause person, of guidance urging the person to move away from the cause person when any of the people is estimated to be the cause person by the cause estimation unit. The service providing device according to any one of claims 1 to 9. comprising the service providing device according to any one of claims 1 to 10 and a payment information processing device,
The payment information processing device includes:
a registration unit that registers a person to whom the service provision device provides services or estimates personal feelings;
an evaluation acquisition unit that acquires the evaluation of the person regarding the provision of the service;
A service providing system, comprising: a consideration determination unit that determines consideration for providing the service based on the evaluation acquired by the evaluation acquisition unit. The consideration determination unit determines the consideration for providing the service based on the evaluation acquired by the evaluation acquisition unit and a change situation in which the personal feelings of the person have changed in response to the provision of the service. The service provision system described in . A service providing method executed by a processor included in a service providing device, the method comprising:
As data related to the target range, an acquisition unit acquires captured image data of the target range and audio data of sounds collected in the target range,
detecting a plurality of people in the target range by extracting data related to humans from the data acquired by the acquisition unit;
Emotions that quantify the personal feelings of each person by extracting an image of the person from the data acquired by the acquisition unit and extracting the voice of the person from the data acquired by the acquisition unit. performing a process of calculating an estimated value, and a process of calculating an overall emotional value indicating the emotional state of the entire target range from the emotion estimated values of the plurality of people,
Based on the change in the estimated emotion value of each person and the change in the overall emotion value , as a factor for the change in the overall emotional state of the target range or the emotional state of any one or more of the people, Estimating the change in the environment that was the cause of the occurrence or the person who was the cause of the occurrence,
A service that is compatible with a change in the environment or a person who is an estimated cause of occurrence, and includes an action on a specific person or a process that changes the entire target range, and includes a process that changes the entire target range or the determining services to be provided to one or more of the persons in the target range;
Among the plurality of persons in the target range, the person whose estimated emotion value exceeds a predetermined threshold earlier than the other persons is estimated to be the factor person.

Images