JP7498688B2 - Model, device and method for estimating acceptability using relationship between change in target state and target state - Google Patents

Description

The present invention relates to technology for evaluating acceptability in autonomous bodies, including robots, self-driving cars, and humans.

In recent years, the provision of various services using robots has been considered in various fields related to daily life, such as nursing care, welfare, medical care, transportation, and housework. Here, robots that will share the activity space with humans must be reliably controlled so as to have as little adverse effect on humans as possible.

For example, when robots and humans move in the same space, the robot is required to observe the movements of the humans around it and appropriately plan and adjust its own movement path so as not to interfere with the movements of the humans. For this reason, currently, in the field of robotics, active research is being conducted on robot control technologies to realize robots that can coexist with humans, and technologies to provide suitable services using robots.

To realize robots that can coexist with humans, it is important to evaluate whether the activities of robots achieved through newly developed control technologies and service provision technologies are actually acceptable to humans, and furthermore, whether they are adapted to human society.

Conventionally, the evaluation of the acceptability of such robots has been carried out by questionnaire surveys or interview surveys based on human subjectivity. For example, Non-Patent Document 1 discloses an example of evaluating the acceptability of a guide robot installed in a public place using an acceptability questionnaire designed based on a technology acceptance model called UTAUT. In addition, Non-Patent Publication 2 discloses a method of evaluating the acceptability of a customer-attracting robot by analyzing the contents of an interview conducted with a store manager who manages the robot introduced in a real store.

Furthermore, for example, Patent Document 1 discloses a technology that uses an acceleration sensor in the robot's leg to detect collisions between a walking robot and an obstacle. The occurrence of such collisions between a robot and an object can be considered a factor that reduces the acceptability of the robot, so by investigating the occurrence of such collision events, it is possible to evaluate the acceptability of the robot.

JP 2009-095906 A

A. Weiss et al., "A Methodological Variation for Acceptance Evaluation of Human-Robot Interaction in Public Places", RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication, 2018, ＜https://doi.org/10.1109/ROMAN.2008.4600751＞ Ishikawa et al., "Social Experiments for the Introduction of Customer-Picking Robots," Journal of the Robotics Society of Japan, Vol. 35, No. 4, pp. 334-345, 2017.

The above describes conventional technologies and methods for evaluating robot acceptability. However, evaluating robot acceptability based on a questionnaire survey requires a great deal of manual work, such as designing the survey, selecting and collecting subjects, distributing the survey (questionnaire), collecting completed surveys, and tallying and analyzing the responses. (Incidentally, the problems with questionnaire surveys discussed here also apply to interview surveys.)

In fact, to obtain statistically significant evaluation results, it is necessary to conduct questionnaire surveys targeting hundreds to thousands of users, and the amount of manual work required and the costs of conducting such surveys are enormous. Furthermore, when developing new technologies or services, it is generally important to repeatedly test, analyze, and evaluate the technology and to continually improve it. However, the enormous amount of work and costs required to repeatedly or continually conduct evaluations using questionnaire surveys makes it impractical in reality.

Furthermore, a questionnaire survey can only obtain an evaluation of the participant's overall impression of a robot, which is based on their past experiences; in other words, it is extremely difficult to evaluate individual events that have occurred in the past involving a robot. For example, if an event occurs in which the participant and a robot come into close contact, and the participant feels danger, and answers "I felt danger" to a questionnaire item asking whether or not the robot felt dangerous, it is not possible to identify from this answer which specific behavior of the robot was problematic.

In this regard, the technology described in Patent Document 1 can detect a collision event between a robot and an obstacle, and therefore can evaluate whether or not a decrease in acceptability has occurred due to this collision event based on, for example, a predetermined rule. However, this evaluation method cannot determine whether or not the robot actually performed anti-acceptance behavior in the collision event that occurred.

For example, if a collision event occurs because an obstacle approaches at such a high speed that even the robot cannot avoid it, it would be inappropriate to judge that there was an acceptability problem with the robot's behavior, but the above evaluation method may result in an evaluation that "the robot's acceptability is low." As such, up until now, there has been no attempt to appropriately evaluate the actual acceptability of a robot based on the actual behavior of the robot as it moves in the same space as humans, for example.

By the way, the problems with acceptability assessment described above can of course also arise when assessing the acceptability of autonomous entities other than robots, such as self-driving cars, autonomous drones, and even humans.

The present invention aims to provide an acceptability estimation model, an acceptability information estimation device, and an acceptability information estimation method that can estimate information related to the acceptability of a subject belonging to a subject group relative to other subjects based on changes in the subject's state and its relationships with others.

According to the present invention, there is provided an acceptability estimation model for estimating information regarding acceptability of a subject belonging to a subject group with respect to other subjects, comprising:
A state change regression neural network (RNN) cell receives state change information, which is information related to a change in state of the target, and generates new state change hidden information by reflecting the state change information on state change hidden information, which is hidden state information generated by itself at a previous point in time;
A relative relationship RNN cell provided for each of the other objects receives relative relationship information, which is information related to a relative relationship between the state of the object and the state of the other object, and generates new relative relationship hidden information by reflecting the relative relationship information on the relative relationship hidden information, which is hidden state information generated by the object itself at a previous point in time;
an influence feature generating unit that generates an influence feature which is a feature expressing an influence between a target and the other target by combining the relative relationship hidden information for each of the other targets with respect to the other targets using a weight that reflects the state change hidden information;
An acceptability estimation model is provided that causes a computer to function as an acceptability information determination unit that receives the influence feature amount as an input and determines information related to the acceptability of the target.

As an embodiment of the acceptability information determination unit according to the present invention, the acceptability information determination unit
A receptive RNN cell that generates a receptive feature expressing the receptivity of the target based on receptive hidden information, which is hidden state information generated by itself at a previous time point, and the received influence feature;
It is also preferable to have an acceptability classification unit that receives the acceptability feature amount as an input and determines the likelihood of each class relating to the presence or absence or degree of pre-set acceptability as information relating to the acceptability.

In addition, as an embodiment of the acceptability estimation model according to the present invention,
The present acceptability estimation model further functions as a self-other relation expression extraction unit that receives self-other relation identification information including information related to a direction from the subject to the other subject, and generates self-other relation information as the relative relation information;
The relative relationship RNN cell includes a self-other relationship RNN that receives the self-other relationship information, reflects the self-other relationship information on the self-other relationship hidden information as the relative relationship hidden information generated by itself at a previous point in time, and generates new self-other relationship hidden information;
It is also preferable that the influence feature generation unit generates a self-influence feature as the influence feature expressing the influence of the other object on the other object by combining the self-other relationship hidden information for each other object across the other objects using a weight that reflects the state change hidden information.

Furthermore, in the above embodiment, it is also preferable that the influence feature generation unit calculates an attention score that is a function of the state change hidden information and the self-other relationship hidden information, and determines the weight based on the attention score.

Furthermore, at least the target has a recognition range for recognizing the other target,
It is also preferable that the influence feature generation unit calculates an attention score that is smaller when the other object is not included in the recognition range of the target than when it is included, or does not use the hidden self-other relationship information relating to the other object that is not included in the recognition range in calculating the attention score.

In addition, as another embodiment of the acceptability estimation model according to the present invention,
The acceptability estimation model further functions as an other-self relation expression extraction unit that receives other-self relation identification information including information related to a direction from the other object to the subject object and generates other-self relation information as the relative relation information;
The relative relationship RNN cell also includes a relative-self relationship RNN that receives the other-self relationship information, reflects the other-self relationship information on the other-self relationship hidden information as relative relationship hidden information generated by itself at a previous point in time, and generates new other-self relationship hidden information;
The influence feature generation unit also generates an other influence feature as the influence feature expressing the influence of the subject object on the subject object by combining the other-self relation hidden information for each subject object across the subject object using a weight reflecting the state change hidden information generated in the state change RNN cell for the subject object;
It is also preferable that the acceptability information determination unit generates information regarding the acceptability of the target by inputting the self-influence feature and the other-influence feature, or the combined amount of the self-influence feature and the other-influence feature, as the influence feature.

Furthermore, in the above embodiment, it is also preferable that the influence feature generation unit calculates a reverse attention score, which is a function of the state change hidden information and the other-self relationship hidden information generated in the state change RNN cell for the other target, and determines the weight based on the reverse attention score.

Furthermore, at least the other object has a recognition range for recognizing the object,
It is also preferable that the influence feature generation unit calculates the reverse attention score to be smaller when the object is not within the recognition range of the other object than when it is within the recognition range of the other object, or that the hidden self-other relationship information relating to the object that is not within the recognition range is not used in calculating the reverse attention score.

In yet another embodiment of the acceptability estimation model according to the present invention, the target performs a process related to a predetermined control based on control information received from an external source or generated internally,
It is also preferable that the acceptability information determination unit generates information regarding the acceptability of the subject by inputting the influence feature generated within a specified time interval that is set based on the time at which the subject received or generated the control information.

Furthermore, as yet another embodiment of the acceptability estimation model according to the present invention, the target performs processing related to a predetermined control based on control information received from an external source or generated internally,
It is also preferable that the status change RNN cell generates new status change hidden information by reflecting information related to the control information and the status change information, or information combining the information related to the control information and the status change information, and/or the relative relationship RNN cell generates new relative relationship hidden information by reflecting information related to the control information and the relative relationship information, or information combining the information related to the control information and the relative relationship information.

Furthermore, as an application example of the acceptability estimation model according to the present invention, the subject is an autonomous body that autonomously determines an action or state to be taken, and the other subject is a person,
It is also preferable that the acceptability information determination unit determines information related to sociability as the acceptability of the autonomous entity.

The present invention also provides an acceptability information estimation device that uses the acceptability estimation model described above to estimate information related to the acceptability of the subject.

According to the present invention, further comprising:
Using information on the state of each subject in the subject group and correct answer information on acceptability, construct the above-mentioned acceptability estimation model through learning,
There is provided a method for estimating acceptability information by a computer, which is characterized by estimating information related to the acceptability of the subject using the constructed acceptability estimation model.

The acceptability estimation model, acceptability information estimation device, and acceptability information estimation method of the present invention can estimate information about the acceptability of a subject belonging to a subject group relative to other subjects based on changes in the subject's state and relationships with others.

1 is a schematic diagram showing an embodiment of an acceptability estimation model according to the present invention and an embodiment of an acceptability information estimation device according to the present invention. FIG. 2 is a schematic diagram showing a specific example of a space-time graph representing input information to the acceptability estimation model according to the present invention. FIG. 13 is a schematic diagram for explaining another embodiment of the acceptability estimation model according to the present invention. FIG. 13 is a schematic diagram for explaining yet another embodiment of the acceptability estimation model according to the present invention. FIG. 13 is a schematic diagram for explaining yet another embodiment of the acceptability estimation model according to the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[Acceptance estimation model]
FIG. 1 is a schematic diagram showing an embodiment of an acceptability estimation model according to the present invention and an embodiment of an acceptability information estimation device according to the present invention.

The acceptability estimation model 1 of this embodiment shown in FIG.
(a) For a group of objects including a robot (object) and multiple humans (other objects) moving close to each other in a certain area,
(b) acquiring time-series data of the location of each of the robots (subjects) and multiple humans (other subjects) (time-series information of data (i.e., location data) indicating the state (called location) taken at each point in time);
(c) Using the time series data of these locations, a machine learning model is used to estimate information regarding the acceptability of this robot (target) between multiple humans (other targets), which in this embodiment is social information about this robot (target).

Here, the "acceptability" of a robot (subject) is defined as the degree to which this robot (subject) is accepted by society (the target group in this embodiment) and its members (other subjects) with their understanding and approval. The "sociability" of a robot (subject) is defined as the degree to which it is suitable to be a member of society (the target group in this embodiment), and is one index of the above-mentioned "acceptability." For example, humans, who are generally "social," are expected to take adjustment actions when moving within the same space, such as changing the direction or speed of movement to avoid collisions and keeping a certain distance between each other. If a robot moving through a crowd takes similar adjustment actions, the robot can be said to be "social."

Specifically, the acceptability estimation model 1 (for example, at time t) is
(A) a state change regression neural network (RNN) cell 11 that receives “state change information” e _v ^t , which is information related to a change in state (change in location in this embodiment) for a robot (target v), and generates new “state change hidden information” h _v ^t by reflecting the “state change information” e _v ^t on “state change hidden information” h _v ^t-1 , which is hidden state information generated by the robot itself at the previous time point (time point t-1);
(B) a relative relationship RNN cell 13 provided for each human (another target i) that receives "relative relationship information" e _vi ^t , which is information related to the relative relationship (relative positional relationship) between the state of the robot (target v) and the state of the human (another target i), and generates new "relative relationship hidden information" h vi t by reflecting the "relative relationship information" e _vi ^t on "relative relationship hidden information" h _{vi t} ^-1 , which is hidden state information generated by the robot itself at the previous point in time (point in time t- ¹ );
(C) an influence feature ^generator 15 that generates an "influence feature" _Hv , which is a feature expressing the influence between the robot (target v) and humans (other targets i), by combining the "relative relationship hidden information" _hvit for each human (other target i) across multiple humans (other targets i) using a weight ^that reflects the "state change hidden information"_hvt;
(D) This is a machine learning model that causes a computer to function as an acceptability information determiner 16 that determines “acceptability information” (sociality information in this embodiment) of a robot (target v) using the influence feature H _v as input.

Here, in this embodiment, a “state” refers to a state in which the robot is located. Therefore, the “state change information” e _v ^t in (A) above can be an amount obtained by converting a time displacement vector x v ^t , which is defined by the magnitude and direction from the location of the robot (target _v ) at the previous time point (time point t−1) to its location at the time point of information calculation (time point t), into a latent expression by the state change expression extraction unit 12.

In addition, in this embodiment, the “relative relationship information” e _vi ^t in (B) above can be the amount obtained by converting the spatial displacement vector x _vi ^t , defined by the size and direction from the location of the robot (target v) to the location of the human (other target i) at the time of information calculation (time t), into a latent expression by the self-other relationship expression extraction unit 14.

The acceptability estimation model 1 embeds such "state change information" e _v and "relative relationship information" e _vi at each time point into "state change hidden information" h _v and "relative relationship hidden information" h _vi , respectively. As a result, the "influence feature" H _v in (C) above takes into account information (state change hidden information) on how the robot's state has changed (under the influence of humans), and becomes an amount that can be regarded as relative relationship (hidden) information between the state of the robot and the state of each human, in which the relationship part with a strong degree of influence is more emphasized, or in other words, is interpreted as "a feature expressing the influence between the robot and humans." This also makes it possible to derive "information related to acceptability" of the robot from such "influence feature."

In this way, the acceptability estimation model 1 can estimate "information related to acceptability" between a robot (subject) and humans (other subjects) based on changes in the state of the robot (subject) and its relationships with others. This also makes it possible to evaluate the acceptability (sociability) of a robot (subject) without relying on questionnaire surveys, which require a great deal of effort and cost and are difficult to conduct on an ongoing basis.

Furthermore, in the acceptability estimation model 1, "information related to acceptability" of a robot is derived, for example, using measurement data of the state of the robot (target) or a human (another target). Therefore, for example, by using measurement data of a time interval in which a specific event occurred, it is possible to perform an acceptability evaluation of the robot (target) based on the fact that the specific event occurred, separate from an acceptability evaluation based on the overall impression of the change in the state of the robot (target).

In addition, in the acceptability estimation model 1 of this embodiment, the relative relationship RNN cell 13 is provided for each human (other object). That is, with the subscript i (=1, 2, ..., m) representing each human (other object), each relative relationship RNN cell 13 generates and outputs "relative relationship hidden information" h _vi (i = 1, 2, ..., m) at each time point, and the influence feature generating unit 15 receives all of these h _vi (i = 1, 2, ..., m) at each time point to generate "influence feature" H _v . As a result, the "influence feature" H _v is a feature that reflects all the influences of each human (other object i) (depending on the degree and content), and the "information related to acceptability" of the robot (object v) derived therefrom is also information related to all humans (other objects i).

In this embodiment, the information estimation target for estimating "information related to acceptability" (information related to sociality) is a robot, and the other target is a human, but the present invention is not limited to this. For example, the information estimation target can be an autonomous body other than a robot, such as an autonomous driving car, an autonomous flying drone, or a human. Also, for example, all of the targets belonging to the target group can be robots, autonomous flying drones, autonomous driving cars, humans, etc. Furthermore, it is also possible to adopt a target group that is a mixture of robots, autonomous flying drones, autonomous driving cars, humans, etc. In other words, a wide variety of targets that can affect each other in terms of the state they should be in at each point in time fall under the target and other targets of the present invention.

In addition, in this embodiment, the "state" of the target (other target) is the state of being located, information relating to changes in state (state change information) is information relating to changes in location, and information relating to the relative relationship of the states (relative relationship information) is information relating to relative positional relationships, but the present invention is of course not limited to this. For example, it is also possible for the target and other targets to be humans, the "state" to be an emotional state, and the information relating to the relative relationship of the states to be a pre-set "distance between emotional states," etc.

In fact, the inventors of the present application created Acceptability Estimation Model 1 by applying the "Social Attention Model," which represents crowds as spatio-temporal graphs and makes it possible to predict crowd behavior from this graph information. Therefore, any information that can be node information in a spatio-temporal graph can be considered to be the "state" of an object (another object), and any information that can be time edge information in a spatio-temporal graph can be considered to be "information relating to changes in state," and any information that can be spatial edge information in a spatio-temporal graph can be considered to be "information relating to the relative relationship of states."

Here, the "Social Attention model" is a crowd behavior prediction model disclosed in the non-patent document: Anirudh Vemula, Katharina Muelling, Jean Oh, "Social Attention: Modeling Attention in Human Crowds", 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, <https://doi.org/10.1109/ICRA.2018.8460504>. Note that the "influence feature" in (C) above in the acceptability estimation model 1 corresponds to the output vector H _v ^t in this non-patent document (equation (4) in this document).

Figure 2 is a schematic diagram showing a specific example of a space-time graph representing input information to the acceptability estimation model 1.

The space-time graph shown in Figure 2 is a graph in which the robot (target) and human (other target) are "nodes" at time t-1, time t, and time t+1, nodes that can affect each other (for example, within a certain distance) are connected by "spatial edges," and corresponding nodes at adjacent times on the time axis are connected by "time edges."

In this embodiment, the above-mentioned time displacement vector _xv is adopted as information related to this "time edge", and the amount obtained by converting this to a latent expression by the state change expression extraction unit 12 is set as "state change information" e _v . Also, the above-mentioned space displacement vector _xvi is adopted as information related to the above-mentioned "spatial edge", and the amount obtained by converting this to a latent expression by the self-other relation expression extraction unit 14 is set as "relative relationship information" e _vi .

In this embodiment, the information related to the "nodes," i.e., the position information of each of the robot (target) and human (other target), can be obtained by analyzing measurement data obtained by various known position measurement means, such as an RGB camera, a near-infrared camera, or LiDAR (Light Detection And Ranging), using a known image recognition technology, such as the technology described in non-patent document: Joseph Redmon, Ali Farhadi, "YOLOv3: An Incremental Improvement", Tech Report, Computer Vision and Pattern Recognition (cs.CV), arXiv:1804.02767 [cs.CV], 2018.

In addition, the information related to the "node" may include skeletal information of the robot (target) and the human (other target). In this case, for example, a heat map showing the position of each joint point can be generated using the technology described in non-patent literature: Zhe Cao et al., "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Journal version of arXiv:1611.08050, Computer Vision and Pattern Recognition (cs.CV), arXiv:1812.08008 [cs.CV], 2019, and this can be used as skeletal information.

In addition, the information related to the "node" can also include gaze direction information of the robot (target) and humans (other targets). In this case, for example, the technique described in Non-Patent Document: Petr Kellnhofer, "Gaze360: Physically Unconstrained Gaze Estimation in the Wild", Computer Vision and Pattern Recognition (cs.CV), arXiv:arXiv:1910.10088 [cs.CV], 2019 can be used to estimate the gaze direction in three-dimensional space, and the estimated result can be used as gaze direction information.

Note that in this specification, the notation of time points is, for example, as shown on the time axis in Figure 2, when viewed from a certain time point t, a time point one unit time before is expressed as t-1. Therefore, for example, a time point two unit times before would be expressed as t-2. Furthermore, a time point one unit time after the passage of time is expressed as t+1. Here, the unit time can be set arbitrarily, but it may also be the time interval (sampling time) between adjacent time points at which node information (location information) can be obtained.

[Model configuration, acceptability estimation method]
The configuration of the acceptance estimation model 1 of this embodiment will be described in more detail below. Returning to FIG. 1, the acceptance estimation model 1 includes:
(A) a state change expression extraction unit 12 and a state change RNN cell 11 including a state change RNN 111;
(a) a self-other relation expression extraction unit 14 and a relative relation RNN cell 13 including a self-other relation RNN 131;
(c) an influence feature generation unit 15 including a first linear transformation unit 151, a second linear transformation unit 152, a mixer 153, a scaler 154, a softmax unit 155, and a weighted mixer 156; and
(e) It has a receptive RNN cell 16a including a receptive RNN 16aa, and an receptive information determination unit 16 equipped with a receptive classification unit 16b, as functional components embodied by a computer (a program installed in the computer).

Each of the above-mentioned functional components will be specifically described below. In Fig. 1, the state change expression extraction unit 12 receives, when the information generation time is time t, a time displacement vector ^xvt as "time edge information" defined by the magnitude and direction from the location of the robot (target _v ) at the previous time t-1 to the location at time ^t (as "state change identification information" as an identifier input to itself), and applies (accumulates) _a matrix ^WTM as a time displacement latent expression extraction operator to this time displacement vector _xvt to generate state change information _evt .

Also in FIG. 1, the state change RNN 111 of the state change RNN cell 11 receives the generated state change information e _v ^t , for example, at time t, and generates new state change hidden information h v t by reflecting this state change information e _v ^t in the state change hidden information h _v ^t-1 that it itself generated at the previous time point, that is, time _{point t-} ¹ .

Here, the state-changing RNN 111 may have a structure that employs a known mechanism capable of forming an RNN, such as a Long-Short Term Memory (LSTM) with a forget gate, or a GRU (Gated Recurrent Unit) with a reset gate and an update gate.

Also in Figure 1, the self-other relationship expression extraction unit 14, assuming that the time when the information is generated is time t, receives as input a spatial displacement vector x _vi ^t as "spatial edge information" defined by the size and direction from the location of the robot (target v) to the location of the human (other target i) at time t (as "self-other relationship identification information" as an identifier input to itself), and applies (accumulates) the matrix W _SP as a spatial displacement latent expression extraction operator to this spatial displacement vector x _vi ^t to generate self-other relationship information (relative relationship information) e _vi ^t .

Also in Figure 1, the self-other relationship RNN 131 of the relative relationship RNN cell 13 receives the generated self-other relationship information (relative relationship information) e _vi ^t , for example, at time point t, and generates new self-other relationship hidden information (relative relationship hidden information) h vi t by reflecting this self-other relationship information (relative relationship information) e _vi ^t on the self-other relationship hidden information (relative relationship hidden information) h _{vi t} ^-1 that it generated at the previous time point, time point t- ¹ .

Here, the self-other relationship RNN 131, like the state change RNN 111 described above, can have a structure that employs known mechanisms capable of constructing an RNN, such as an LSTM with a forget gate, or a GRU with a reset gate and an update gate.

In this embodiment, the self-other relation RNN 131 is provided for each person (other object i (= 1, 2, ..., m)). That is, each self-other relation RNN 131 generates and outputs self-other relation hidden information (relative relation hidden information) h _vi (i = 1, 2, ..., m) at each point in time.

1, the influence feature generation unit 15 generates a self-influence feature Hvt as an influence feature expressing the influence between the ^robot (target v) and the human (other target i) by combining the self-other relationship hidden information (relative relationship hidden information) _hvit (i=1, 2, ..., m) generated for each human (other target i) at the same ^{time t} using a weight reflecting the state change hidden information _hvt generated by the state change RNN 111 for the human (other target i (=1, 2, _... , m)). In this embodiment, this is exactly the process ^of generating the self-influence feature _Hvt using a known attention mechanism.

More specifically, in this embodiment,
(a) The state change hidden information hvt adjusted to a vector of dimension _de by multiplying it by a weighting matrix _W1 by the first linear transformation unit 151 of the influence feature generation unit 15 and the self-other relationship hidden information _hvit adjusted to ^a vector of dimension _de by multiplying it by a weighting matrix _W2 by the second linear transformation unit ₁₅₂ are ^input , ^and the mixer 153 mixes the two to generate a mixture amount (for example ^{, takes the inner product of the two to generate the mixture amount <W1·hvt} _{, W2 · hvit} >);
(b) The scaler 154 performs a normalization process (multiplying m·(d _e ) ^−0.5 ) on the mixed vector having m mixed amounts output from the mixer 13 as elements.
(c) The softmax unit 155 applies a well-known softmax function to this normalized mixed vector, and outputs an attention score vector that maximizes the maximum value of the vector elements in this normalized mixed vector and whose elements sum to 1.

Here, the attention score vector is interpreted as _a representation of the importance of the state change ^{hidden information hvt for} each element of the self-other relationship hidden information _hvt . The attention score, which is an element of this attention score vector, is a function of the state change hidden information _hvt ^and the self-other relationship hidden ^{information hvt} _, and is specifically expressed by the following formula (1): score( _hvt ^, _hvit )＝m・( _{d e} ^{) −0.5} < _W1・_hvt ^, _W2・_hvit ^>
It is a quantity expressed as:
Incidentally, the attention score score( _hvt ^, _hvit ⁾ in the above formula (1) can be understood to represent the degree of attention paid by the robot (target v) to a human (another target i).

Next, the weighted mixer 156
(d) Based on the determined attention score, the weight a _vi reflecting the weight state change hidden information h _v ^t is calculated, for example, by the following formula (2): a _vi = exp(score(h _v ^t , h _vi ^t ))/Σ _j=1 ^m exp(score(h _v ^t , h _vj ^t ))
Calculated by
(e) The self-other relationship hidden information h _vi ^t (i = 1, 2, ..., m) generated for each human (other object i) is combined across humans (other objects i (= 1, 2, ..., m) using the weights a _vi described above to generate the self-influence feature H _v ^t .

Specifically, the weighting mixer 156 uses the following equation (3): H _v ^t =Σ _i=1 ^m a _vi ·h _vi ^t
Here, such a self-influence feature H _v ^t is a feature _that expresses the influence of humans (other targets i) on the robot (target v) ^that occurs as a result of the robot paying attention to humans around it.

1, in this embodiment, the acceptability information determination unit 16 receives the self-influence feature _Hv (=[ _Hvt ^, _Hvt ⁺¹ , ..., _Hvt ⁺ⁿ ]) generated in a predetermined time interval as an input and determines the sociality score _SCv as "information related to acceptability" of the robot (target v). Specifically, the acceptability information determination unit 16:
(a) a receptive RNN cell 16a including a receptive RNN 16aa;
(b) an acceptability classifier 16b.

Here, the receptive RNN cell 16a in (a) above is equipped with a receptive RNN 16aa that generates a receptive feature S _v that expresses the receptiveness of the robot (target v) based on receptive hidden information h _s , which is hidden state information that it generated itself at a previous point in time, and the received influence feature H _v .

Specifically, the receptive RNN 16aa may sequentially import the self-influence feature _Hv = [ _Hvt ^, _Hvt ⁺¹ , ..., _Hvt ⁺ⁿ ] as time-series data, and output the prediction result of the last generated self-influence feature (or the last generated receptive hidden information _hst ⁺ⁿ ) as the receptive feature _Sv . In this case, the output receptive feature _Sv is a feature that represents receptiveness taking into account the time fluctuation of the influence of a human (another target i) on a robot (target v) (in the time interval t to t+n).

In addition, the receptivity classification unit 16b in (b) above is composed of a multilayer perceptron algorithm in this embodiment, and receives the receptivity feature Sv received from the receptivity RNN cell 16a (receptivity RNN 16aa) as an input, and for each class relating to the presence or absence or degree of receptivity that has been previously set, determines a sociality _{score SCv} , which is the likelihood of that class, as information relating to receptivity.

Here, in this embodiment, sociability (of the robot) is adopted as the acceptability, and the preset classes may be, for example, "high sociability (social)", "medium sociability (neutral)", and "low sociability (unsocial)". In this case, the determined sociability information (information related to acceptability) may be, for example, information that "is 'high sociability (social)' (the sociability score SC _v (likelihood) of 'high sociability (social)' is 0.7, which is the highest)" or information that "the sociability score SC _v (likelihood) of 'high sociability (social)' is 0.7, the sociability score SC _v (likelihood) of 'medium sociability (neutral)' is 0.2, and the sociability score SC _v (likelihood) of 'low sociability (unsocial)' is 0.1".

The "information related to acceptability" determined by the acceptability information determination unit 16 is not limited to the sociality score SC _v , of course. For example, it may be a favorability rating (score) (of the robot according to the degree to which the robot is acceptable). The acceptability information determination unit 16 may also directly input at least one of the self-influence feature H _v or its elements (H _v ^t , H _v ^t+1 , ..., H _v ^t+n ) to, for example, a DNN (Deep Neural Networks) and output the desired "information related to acceptability."

[Training acceptability estimation model 1]
The following describes the training (learning) of the acceptability estimation model 1. First, a set of time edge information (e.g., time series data of time displacement vector _xv ) and space edge information (e.g., time series data of space displacement vector _xvi (i = 1, 2, ..., m)) is prepared as learning data (correct answer data) labeled (e.g., with any one of "high sociality", "medium sociality", and "low sociality") (e.g., using the results of a questionnaire survey).

Next, using this learning data, the difference between the output from the acceptability classification determination unit 16b and the correct answer is treated as a loss, and various learnable parameters, matrices, etc. in model 1 are trained by applying the well-known backpropagation method using this loss or the well-known RNN training method, BPTT (Back Propagation Through Time).

As a modification of the training, the state change RNN cell 11, the state change expression extraction unit 12, the relative relationship RNN cell 13, the self-other relationship expression extraction unit 14, and the influence feature generation unit 15 may be pre-trained using learning data other than the sociality score. For example, it is also possible to adopt a pre-trained one using learning data (correct answer data) of the actual crowd behavior (including robots) as in the Social Attention model disclosed in the above-mentioned non-patent document: Anirudh Vemula, Katharina Muelling, Jean Oh, "Social Attention: Modeling Attention in Human Crowds", 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, <https://doi.org/10.1109/ICRA.2018.8460504>.

[Another embodiment of the acceptability estimation model]
FIG. 3 is a schematic diagram for explaining another embodiment of the acceptability estimation model according to the present invention.

In the embodiment shown in FIG. 3, in the acceptability estimation model 1 described with reference to FIG. 1, the following is used instead of the relative relation RNN cell 13 (FIG. 1):
(a) A relative relationship RNN cell 13' including an other-self relationship RNN 132 that receives other-self relationship information e _iv ^t from an other-self relationship expression extraction unit 142 in addition to a self-other relationship RNN 131 (which receives self-other relationship information e _iv ^t from a self-other relationship expression extraction unit 141).
In addition,
(b) A state change RNN cell 11′ including a state change expression extraction unit 12′ related to a human (other object i) and a state change RNN 111′, which is set for each human (other object i).
Further, instead of the influence feature generating unit 15 (FIG. 1),
(c) an influential feature generating unit 15′ including a first attention unit 15A and a second attention unit 15B
is provided.

Here, the self-other relationship expression extraction unit 142 (a) above takes as input a spatial displacement vector x _iv ^t as "spatial edge information" defined by the size and direction from the location of the human (other target i) to the location of the robot (target v) (for example, at time t) (as "self-other relationship identification information" as an identifier input to itself), and applies (accumulates) the matrix W _SP as a spatial displacement latent expression extraction operator to this spatial displacement vector x _iv ^t to generate self-other relationship information (relative relationship information) e _iv ^t .

Also, the self-other relationship RNN 132 (relative relationship RNN cell 13') shown in (a) above receives the self-other relationship information (relative relationship information) e _iv ^t generated (for example, at time t), and generates new self-other relationship hidden information (relative relationship hidden information) h _iv t by reflecting this self-other relationship information (relative relationship information) e _iv ^t on the self-other relationship hidden information (relative relationship hidden information) h _iv ^{t -1 that it generated at the previous time (time t-1} ).

Furthermore, the state change RNN cell 11' (state change RNN 111') in (b) receives the state change information e _i ^t generated by the state change expression extraction unit 12' (for example, at the same time point t), and generates new state change hidden information h _i ^t by reflecting this state change information _{e i t} ⁱⁿ the state change hidden information h i t ^-1 that it itself generated at the previous time point (time point t-1).

In addition, the first attention unit 15A (influence feature generation unit 15') in (c) above has a structure similar to that of the influence feature generation unit 15 (Figure 1), and like the influence feature generation unit 15 (Figure 1), generates a self-influence feature _Hv (= [ _Hvt ^{, Hvt+1} , ..., _Hvt ⁺ⁿ ]) using the state _change hidden information _hv received from the state change RNN 111 and the self-other relationship hidden information hvi received from the self-other relationship _RNN ¹³¹ .

On the other hand, the second attention unit 15B (influence feature generation unit 15′) in (c) above has a structure similar to that of the influence feature generation unit 15 (FIG. 1), but generates an other influence feature H _other (=[H other t, H other t+1, ..., H _other t+n]) as an influence feature by matching the other-self relationship hidden information h _iv received from ^the other-self relationship ^{RNN 132} with humans (other targets i) using a weight a _iv calculated from the “reverse attention score r-score” that reflects the state _{change hidden} information h i generated by the state _change ^{RNN 111′} .

Here, the reverse attention score r-score (for example, at time t) is specifically expressed by the following formula (4): r-score(h _i ^t , h _iv ^t )＝m・(d _e ) ^−0.5 <W ₁・h _i ^t , W ₂・h _iv ^t >
This quantity can be expressed as the degree of attention paid to the robot (target v) by a human (other target i).

In addition, the weight a _iv (for example, at the same time t) is expressed by the following formula (5): a _iv = exp(r-score(h _i ^t , h _iv ^t ))/Σ _j=1 ^m exp(r-score(h _i ^t , h _ij ^t ))
Thus, the other influence feature H _other ^t is calculated by the following formula (6): H _other ^t = Σ _i=1 ^m a _iv h _iv ^t
Here, such other influence feature H _other ^t is an influence feature that expresses the influence of the robot (target v) on humans (other targets i) that occurs as a result of surrounding humans paying attention to the robot.

As described above, the influence feature generating unit 15' of this embodiment generates not only the self-influence feature _Hv ( ⁼ [ _Hvt , _Hvt ⁺¹ , ..., _Hvt ⁺ⁿ ]) but also the other-influence feature _Hother (=[ ^Hothert _{, Hothert} ⁺¹ , ..., Hothert ⁺ⁿ ]). Here, the receptive RNN cell 16a' (receptive information determining unit ₁₆ ') can generate the receptive feature Sv of the robot (target v _{) using the self-influence feature Hv and the other-influence feature Hother , or the} combined amount (for example, the combined amount or the added amount) of the self-influence feature Hv and the other-influence feature Hother as input.

This makes it possible to determine more accurate "information related to acceptability" that reflects the actual situation in which robots and humans may influence _each other. Here, the sum of the self-influence feature _Hv and the other-influence feature Hother can be regarded as an "influence feature" that expresses the (mutual) influence between a robot (target) and a human (other target).

[Yet another embodiment of the acceptability estimation model]
FIG. 4 is a schematic diagram for explaining still another embodiment of the acceptability estimation model according to the present invention.

In the embodiment shown in FIG. 4, in the acceptability estimation model 1 described with reference to FIG. 1, instead of the influence feature generating unit 15 (FIG. 1),
(a) An influence feature generating unit 15″ including a recognition range weighting function unit 159 in addition to a first linear transform unit 151, a second linear transform unit 152, a mixer 153, a scale unit 154, a softmax unit 155, and a weighted mixer 156″
is provided.

Here, in this embodiment, at least the robot (target) has a recognition range for recognizing humans (other targets), and the recognition range weighting function unit 159 of the influence feature generator 15'' can receive "recognition range information" that is information related to this recognition range. This "recognition range information" may be, for example, information that represents a field of view range of a predetermined angle range ±θ centered on the gaze direction estimated for the robot (target). Alternatively, it may be information that represents a robot orientation range of a predetermined angle range ±θ centered on the direction in which the robot chest or robot head is facing, determined from skeletal information estimated for the robot (target) (position information of each joint point).

The above gaze direction estimation can be performed using the technology described in, for example, the non-patent literature: Zhe Cao et al., "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Journal version of arXiv:1611.08050, Computer Vision and Pattern Recognition (cs.CV), arXiv:1812.08008 [cs.CV], 2019. The above skeletal information estimation can also be performed using the technology described in, for example, the non-patent literature: Petr Kellnhofer, "Gaze360: Physically Unconstrained Gaze Estimation in the Wild", Computer Vision and Pattern Recognition (cs.CV), arXiv:arXiv:1910.10088 [cs.CV], 2019.

Also in FIG. 4, the recognition range weight function unit 159 calculates
(a) Receive the "recognition range information" as described above and location information of the robot (target v) and humans (other targets i), and based on this information, determine a function value fov(v, i) (i = 1, 2, ..., m) for each human (other target i) using a recognition range weighting function fov that takes a smaller value when a human (other target i) is not within the recognition range of the robot (target v) compared to when the human is within the recognition range (for example, set fov(v, i) to a positive value less than 1 or zero when the human is not within the recognition range, and set fov(v, i) to 1 when the human is within the recognition range);
(b) Multiply the attention score (vector) output from the softmax unit 155 by the determined function value f(v, i) (i = 1, 2, ..., m) to generate a recognition range adjusted attention score score_adj, and output it to the weighted mixing unit 156'' ^. Specifically, for example, the following formula (7) score_adj( _hvt , ^hvit ) = score( _hvt ^, _hvit ) fov( _v , ⁱ )
According to the above formula (7), when a human (another object i) is not within the recognition range of the robot (object v), the attention score score_adj will also be smaller than when a human (another object i) is within the recognition range of the robot (object v).

This allows us to incorporate into the attention score (adjusted for recognition range) the knowledge that the degree to which a robot (subject v) pays attention to a human (other target) is higher when the human (other target) is located within the robot's (subject's) recognition range. As a result, it is expected that the estimation accuracy of the self-influence feature that is subsequently determined and the "information related to acceptability" that is finally determined will be further improved.

As a modification, ^when the influence feature generating unit 15'' calculates the attention score score for each human (other target i (=1, ² , ..., m)) using the above formula (1): score( _hvt ^, _hvit ) = m (d _e ) ^-0.5 < _{W1 hvt} ^, _{W2 hvit} > (i = 1, 2, ..., m), it is also possible ^not to use the self-other relation hidden information hvpt related to a human (other target p) that is not within the recognition range of the robot (target _v ) in the calculation of this attention score score. In this case, so to speak, the spatial edge related to such a human (other target p) is excluded in the calculation of the attention score (influence feature). This method is also expected to further improve the estimation accuracy of the self-influence feature determined later and the "information related to acceptability" that is finally determined.

As another embodiment, it is also preferable to apply the above-described recognition range weight function unit 159 to the second attention unit 15B shown in FIG. 3. In this case, at least the human (other object) has a recognition range for recognizing the robot (object), and the recognition range weight function unit 159 of the second attention unit 15B (FIG. 3) receives "recognition range information" which is information related to this recognition range, and calculates the recognition range-adjusted reverse attention score r-score_adj using the recognition range weight function fov which takes a smaller value when the robot (object) is not in the recognition range of the human (other object) compared to when the robot (object) is in the recognition range, and outputs it to the weighting mixer unit (156) in the subsequent stage. Here, the reverse attention score r-score_adj is expressed, for example, by the following formula (8): r-score_adj(h _i ^t , h _iv ^t )=r-score(h _i ^t , h _iv ^t )·fov(i, v)
It may be calculated using the following formula.

This allows the knowledge that the degree of attention paid by humans (other objects) to a robot (object v) is higher when the robot (object) is located within the human's (other object's) range of recognition to be incorporated into the (recognition-range-adjusted) reverse attention score. As a result, it is expected that the estimation accuracy of the other-influence features determined subsequently and the "acceptance-related information" that is finally determined will be further improved.

As a modification, it is also possible that the second attention unit 15B (FIG. 3) does not use the other-self relation hidden information h _pv ^t related to the robot (target) that is not within the recognition range of the human (other target p) in the calculation of the reverse attention score r-score in the above formula (4). Even with such a method, it is expected that the estimation accuracy of the other influence feature amount determined later and the "information related to acceptability" that is finally determined will be further improved.

When introducing recognition range information into the second attention section 15B (Figure 3) as described above, it is also preferable to introduce the recognition range information into the first attention section 15A (Figure 3) in the manner described above with reference to Figure 4.

[Yet another embodiment of the acceptability estimation model]
FIG. 5 is a schematic diagram for explaining still another embodiment of the acceptability estimation model according to the present invention.

In the embodiment shown in Fig. 5, an autonomous vehicle (subject v) that is the subject of sociability (acceptance) estimation is traveling among humans (other subjects i). In order to estimate the sociability (acceptance) of such an autonomous vehicle (subject v), in the acceptability estimation model 1 described with reference to Fig. 1, instead of the state change RNN cell 11 (Fig. 1) and the relative relationship RNN cell 13 (Fig. 1),
(a) a control information expression extraction unit 18 that receives “control information” c _v ^t from an autonomous vehicle (target v) at a time t, for example, and applies (accumulates) a matrix W _CL as a control information latent expression extraction operator to the “control information” c v t to generate control expression information ec _v ^t ;
(b) a state change RNN cell 11″ including a state change RNN 111 ^″ that generates new state change _hidden information ^hvt by reflecting, for example, the control expression information _ecvt generated by the control information expression extraction unit 18 and the state change information _evt generated by the state change expression extraction unit ^{12, or information combining the control expression information ecvt} _and the state ^{change information evt} ₍ for example, information obtained by concatenating or adding them), and outputs the generated new state change hidden ^information hvt to the influence feature generation unit 15, at the same time point t;
(c) For example, at the same time t, a relative relationship RNN cell 13'^' is provided that includes a self-other relationship RNN 113 ^'' that generates new self-other relationship hidden information h _{vi t} by reflecting the control expression information ec _v ^t generated by the control information expression extraction unit 18 and the self-other relationship information e _vi ^t generated by the self-other relationship expression extraction unit 14, or information that combines the control expression information ec _v t and the self-other relationship information e vi ^t (for example, information obtained by concatenating or adding them), and outputs the generated information to the influence feature generation unit 15.

Here, the "control information" c _v in (a) above is information that the target v (the self-driving car in FIG. 5) receives from the outside or generates within the target v, and based on this, the target v performs processing related to a predetermined control related to a state change (e.g., movement) of the target v. For example, the self-driving car (target v) shown in FIG. 5 generates (or receives from the outside) a "signal instructing to blink a right-turn indicator" as "control information" c _v , blinks its own right-turn indicator, and transmits this "control information (right-turn indicator blinking instruction signal)" c _v to the acceptability estimation model 1 (the control information expression extraction unit 18).

As a modification, it is also possible to measure the actual control operation (e.g., the blinking operation of the right turn signal) of the object v (an autonomous vehicle in FIG. 5) by an external measuring means, estimate the control information that the object v would have accepted from the measurement results, and set this as the "control information" c _v . It should be noted that, as a matter of course, the object v related to the "control information" c _v is not limited to an autonomous vehicle as shown in FIG. 5.

Furthermore, with regard to (b) and (c) above, only one of them may be adopted. That is, in the acceptability estimation model 1 described using FIG. 1, the state change RNN cell 11 (FIG. 1) may be replaced with the state change RNN cell 11'' of (b) above, and the relative relationship RNN cell 13 (FIG. 1) may be adopted as is, or the relative relationship RNN cell 13 (FIG. 1) may be replaced with the relative relationship RNN cell 13'' of (c) above, and the state change RNN cell 11 (FIG. 1) may be adopted as is.

In any case, according to this embodiment, it is possible to generate an influence feature with high accuracy that is more suited to the actual situation, taking into account “control information” c _v related to a state change (e.g., movement) of an object v such as an autonomous vehicle or a robot.

Incidentally, as another embodiment of the "control information" _cv , it is also possible to input the control expression information _ecv to the state change RNN 111, the self-other relationship RNN 131, and/or the other-self relationship RNN 132 shown in Fig. 3. Even in this case, by taking into account the "control information" _cv , it is possible to generate an influence feature quantity with higher accuracy that is more suited to the actual situation.

It is also possible to adopt an embodiment in which the other target i is a robot, an autonomous vehicle, or the like, and the received or generated "control information" c _i is taken into the state change RNN (111, 111'', the self-other relation RNN (131), or the other-self relation RNN (132). Even in this case, by taking into account the "control information", it is possible to generate an influence feature with high accuracy that is more suited to the actual situation.

Furthermore, as yet another embodiment related to the "control information" c _v , the acceptability information determiner 16 (FIG. 1) may generate "information related to acceptability" of the target _v by inputting an influence feature generated within a "predetermined time interval" set based on the time when the target v (an autonomous vehicle in FIG. 5) received or generated the "control information" c v, or the time related to the control specified by the control information. Here, this is an embodiment that can be implemented whether or not the influence feature is generated taking into account the "control information" c _v as described above.

Specifically, as the "predetermined time interval", for example, a time interval (t 0 -T/2 to t 0 +T/2) including the reference time t ₀ at the center can be used, based on the time when the target v (autonomous vehicle ₎ acquires the "control information" _{c v} and performs _{a predetermined control. A time interval (t 0 to t 0 + T} ) from the reference time t ₀ to the time when a predetermined time T has elapsed can also be used. Furthermore, a predetermined delay time usually elapses between when the target v (autonomous vehicle) performs control and when another target i (human) recognizes it. Therefore, it is also possible to set the "predetermined time interval" to a time interval starting from the reference time t ₀ at which a predetermined delay time T_delay has elapsed (t ₀ +T_delay).

In this way, by performing acceptability (sociality) estimation using the influence feature generated within a "predetermined time interval" set based on the "control information" c _v , it becomes possible to determine whether or not the behavior/movement (state change) of the target v (autonomous vehicle) at a specific time point (within this predetermined time interval) is acceptable (social). Alternatively, it is also possible to determine whether or not a specific control behavior (related to this predetermined time interval) of the target v (autonomous vehicle) is acceptable (social).

[Receptivity information estimation device, receptivity information estimation program and method]
Returning to Fig. 1, the receptivity information estimation device 9 that is equipped with the receptivity estimation model 1 described above and estimates information related to the receptivity of a target will be described below. Incidentally, the device 9 will be described below as being equipped with the receptivity estimation model 1 shown in Fig. 1, but it may of course be equipped with the model 1 that employs the form shown in Fig. 3, Fig. 4, or Fig. 5.

The acceptability information estimation device 9 of this embodiment shown in the lower part of Figure 1 uses the installed trained acceptability estimation model 1 to generate and output information related to the acceptability of the robot (target), which in this embodiment is information determining whether or not the target is sociable.

Specifically, in FIG. 1, the input unit 91 of the receptivity information estimation device 9 includes:
(a) acquiring a group of time-series data on the locations of a robot (target) and a human (other target) from a target management server equipped with a communication function and installed, for example, externally (generating, storing, and managing time-series data on the location of a target recognized in an image using a known image recognition technique from an image generated by, for example, a surveillance camera);
(b) generating a time series data group of time edge information (e.g., time displacement vector x _v ) and space edge information (e.g., space displacement vector x _vi ) described with reference to FIG. 2 from the acquired time series data group of the location, and assigning a correct answer label to this time series data group (e.g., by a user of the device 9) to create a learning data set, and outputting the same to the training unit 92;
(c) On the other hand, time series data of time edge information (e.g., time displacement vector x _v ) and space edge information (e.g., space displacement vector x _vi ) relating to the robot (target) that is the target of sociality estimation is output to the acceptability information determination unit 93.

The training unit 92 trains the acceptability estimation model 1 using the received learning dataset.

In this embodiment, the acceptability information determination unit 93
(d) inputting the received time series data of the time-spatial edge information of (c) to the state change expression extraction unit 12 and the self-other relation expression extraction unit 14 of the trained acceptability estimation model 1, and then
(e) From the sociality score SC _v (likelihood) of each class output from the acceptability classification unit 16 b of the trained acceptability estimation model 1, one piece of sociality information, for example, “the robot (target) is social,” “the robot (target) is neutral with respect to sociality,” or “the robot (target) is non-social,” is determined as “information related to acceptability,” and output to the output unit 94.

The output unit 94 transmits the received social information to an external information processing device (if it has a communication function) as social information (information related to acceptability) of the robot (subject) that is the subject of sociality estimation, or displays it (if it has a display function).

The training unit 92 and the receptivity information determination unit 93 are main functional components that implement one embodiment of the receptivity information estimation method according to the present invention, and can also be considered as functions of a processor/memory that stores one embodiment of the receptivity information estimation program according to the present invention. For this reason, the receptivity information estimation device 9 may be a dedicated device for receptivity information estimation, but it can also be, for example, a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or a smartphone, etc., equipped with the receptivity information estimation program according to the present invention.

As described above in detail, according to the present invention, information on the acceptability of a subject (e.g., a robot) belonging to a subject group between other subjects (e.g., humans) can be estimated based on changes in the subject's (robot's) state and its relationships with others. In other words, for example, it is possible to evaluate the acceptability (sociability) of a subject (robot) without relying on a questionnaire survey, which requires a great deal of effort and cost and is difficult to conduct on an ongoing basis.

This will also make it possible to easily and reliably evaluate whether the activities of robots used in various service provision businesses in fields related to daily life, such as nursing care, welfare, medical care, transportation, and housework, are actually acceptable to the people receiving the services, and whether the robots are adapted to human society.

In addition, for example, the sociality information of children estimated by the present invention (from time-series information on their behavior) can be used to foster rich sociality in children and provide them with education suited to their current personality and behavioral tendencies. In other words, the present invention can contribute to Goal 4 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all."

Furthermore, for example, the social information of adults estimated by the present invention (from time-series information on the work content) can be used to provide them with decent work (rewarding work that is humane) that is socially beneficial and matches their current individual characteristics and behavioral tendencies. In other words, the present invention can contribute to Goal 8 of the United Nations-led SDGs, "Promote inclusive and sustainable economic growth, employment and decent work for all."

Furthermore, for example, to provide consumers with education about socially acceptable (e.g., sustainable) consumption and lifestyles, the social information of the consumer estimated by the present invention (from time-series information on the consumer's consumption behavior and life history) can be utilized. In other words, the present invention can contribute to Goal 12 of the United Nations-led SDGs, "Ensure sustainable consumption and production patterns."

With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical ideas and viewpoints of the present invention can be easily made by a person skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents.

1 Acceptability estimation model 11, 11', 11'' State change RNN cell 111, 111', 111'' State change RNN
12, 12' State change expression extraction unit 13, 13', 13'' Relative relationship RNN cell 131, 131'' Self-other relationship RNN
132 Self-Other Relations RNN
14, 141 Self-other relation expression extraction unit 142 Other-other relation expression extraction unit 15, 15', 15'' Influence feature quantity generation unit 15A First attention unit 15B Second attention unit 151 First linear transformation unit 152 Second linear transformation unit 153 Mixing unit 154 Scaling unit 155 Softmax unit 156, 156'' Weighted mixing unit 159 Recognition range weight function unit 16 Receptivity information determination unit 16a, 16a' Receptive RNN cell 16aa Receptive RNN
16b Acceptability classification unit 18 Control information expression extraction unit 9 Acceptability information estimation device 91 Input unit 92 Training unit 93 Acceptability information determination unit 94 Output unit

Claims (13)

An acceptability estimation model for estimating information regarding acceptability between a subject belonging to a subject group and other subjects, comprising:
A state change regression neural network (RNN) cell receives state change information, which is information related to a change in state of the target, and generates new state change hidden information by reflecting the state change information on state change hidden information, which is hidden state information generated by itself at a previous point in time;
A relative relationship RNN cell provided for each of the other objects receives relative relationship information, which is information related to a relative relationship between the state of the object and the state of the other object, and generates new relative relationship hidden information by reflecting the relative relationship information on the relative relationship hidden information, which is hidden state information generated by the object itself at a previous point in time;
an influence feature generating unit that generates an influence feature which is a feature expressing an influence between a target and the other target by combining the relative relationship hidden information for each of the other targets with respect to the other targets using a weight that reflects the state change hidden information;
The acceptability estimation model is characterized in that it causes a computer to function as an acceptability information determination unit that receives the influence feature amount as an input and determines information related to the acceptability of the target. The acceptability information determination unit,
A receptive RNN cell that generates a receptive feature expressing the receptivity of the target based on receptive hidden information, which is hidden state information generated by itself at a previous time point, and the received influence feature;
The acceptability estimation model according to claim 1, further comprising an acceptability classification unit that receives the acceptability feature as an input and determines the likelihood of each class relating to the presence or absence or degree of acceptability that has been previously set as information relating to the acceptability. and further causing the computer to function as a self-other relation expression extraction unit that receives self-other relation identification information including information related to a direction from the target to the other target and generates self-other relation information as the relative relation information;
The relative relationship RNN cell includes a self-other relationship RNN that receives the self-other relationship information, reflects the self-other relationship information on the self-other relationship hidden information as the relative relationship hidden information generated by itself at a previous point in time, and generates new self-other relationship hidden information;
The acceptability estimation model according to claim 1 or 2, characterized in that the influence feature generation unit generates a self-influence feature as the influence feature expressing the influence of the other object on the other object by combining the self-other relationship hidden information for each other object across the other objects using a weight that reflects the state change hidden information. The acceptability estimation model according to claim 3, characterized in that the influence feature generation unit calculates an attention score that is a function of the state change hidden information and the self-other relationship hidden information, and determines the weight based on the attention score. At least the target has a recognition range for recognizing the other target,
The acceptability estimation model described in claim 4, characterized in that the influence feature generation unit calculates an attention score that is smaller when the other object is not included in the recognition range of the target compared to when it is included, or does not use the self-other relationship hidden information related to the other object that is not included in the recognition range when calculating the attention score. and further causing the computer to function as an other-self relation expression extraction unit that receives other-self relation identification information including information related to a direction from the other object to the object and generates other-self relation information as the relative relation information;
The relative relationship RNN cell also includes a relative-self relationship RNN that receives the other-self relationship information, reflects the other-self relationship information on the other-self relationship hidden information as relative relationship hidden information generated by itself at a previous point in time, and generates new other-self relationship hidden information;
The influence feature generating unit also generates an other influence feature as the influence feature expressing the influence of the object on the other object by combining the other-self relation hidden information for each of the other objects across the other objects using a weight reflecting the state change hidden information generated in the state change RNN cell for the other object,
The acceptability estimation model according to any one of claims 3 to 5, characterized in that the acceptability information determination unit generates information related to the acceptability of the target by inputting the self-influence feature and the other-influence feature, or a combined amount of the self-influence feature and the other-influence feature, as the influence feature. The acceptability estimation model according to claim 6, characterized in that the influence feature generation unit calculates a reverse attention score which is a function of the state change hidden information and the other-self relation hidden information generated in the state change RNN cell for the other target, and determines the weight based on the reverse attention score. At least the other object has a recognition range for recognizing the object,
The acceptability estimation model described in claim 7, characterized in that the influence feature generation unit calculates the reverse attention score to be smaller when the object is not within the recognition range of the other object than when it is within the recognition range, or does not use the other-self relationship hidden information related to the object that is not within the recognition range when calculating the reverse attention score. The target performs a process related to a predetermined control based on control information received from an external source or generated internally,
The acceptability estimation model according to any one of claims 1 to 8, characterized in that the acceptability information determination unit generates information related to the acceptability of the subject by inputting the influence feature generated within a specified time interval set based on the time point at which the subject received or generated the control information, or the time point related to the control specified in the control information. The target performs a process related to a predetermined control based on control information received from an external source or generated internally,
The acceptability estimation model described in any one of claims 1 to 9, characterized in that the status change RNN cell generates new status change hidden information by reflecting information related to the control information and the status change information, or information combining the information related to the control information and the status change information, and/or the relative relationship RNN cell generates new relative relationship hidden information by reflecting information related to the control information and the relative relationship information, or information combining the information related to the control information and the relative relationship information. The subject is an autonomous entity that autonomously determines the action or state to be taken, and the other subject is a person;
The acceptability estimation model according to claim 1 , wherein the acceptability information determination unit determines information related to sociality as the acceptability of the autonomous body. An acceptability information estimation device that estimates information related to the acceptability of a subject using an acceptability estimation model according to any one of claims 1 to 11. constructing an acceptability estimation model according to any one of claims 1 to 11 by learning using information on the state of each subject in the subject group and correct answer information on acceptability;
A method for determining and estimating acceptability information by a computer, comprising estimating information relating to the acceptability of the subject using the constructed acceptability estimation model.

Images