JP7354344B2 - Image analysis device, image analysis method, and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Publication date July 16, 2018 Presentation name 27th International Conference on Artificial Intelligence, 23rd European Conference on Artificial Intelligence (IJCAI-ECAI2018) Main track 2nd day Presentation venue Stockholm Conference The date of the website of the website (HTTPS: / WWWW. /) (Http: // static.ijcai.org/2018-Program.html) (https://www.ijcai.org/proceedings/2018/0214.pdf)

The present invention relates to an image analysis device, an image analysis method, and a program.

Conventionally, based on the learning results of learning the characteristics of an image of a person (for example, the mannerisms and facial expressions seen before speaking) and the probability of speaking when those characteristics are confirmed, 2. Description of the Related Art There is a known technology that achieves smooth switching of microphones, cameras, etc. in TV conference systems and the like by predicting the speaker before speaking (see Patent Document 1).

Furthermore, a technique for detecting the line of sight of a person appearing in an image is known (see Non-Patent Document 1).

Japanese Patent Application Publication No. 2007-147762

Recasens, 3 others, Massachusetts Institute of Technology, “Where are they looking?”, [online], [searched on July 2, 2018], Internet <URL: http://people.csail.mit.edu/ khosla/papers/nips2015_recasens.pdf＞

However, with conventional techniques, it has not been possible to infer the speaker and the other party from an image of a person.

The present invention has been made in consideration of such circumstances, and provides an image analysis device, an image analysis method, and an image analysis device that can appropriately infer the person to whom the utterance is based based on the utterer and the utterance content included in the image. One of the purposes is to provide programs and programs.

An image analysis device according to one aspect of the present invention includes a line-of-sight position estimating unit that estimates a line-of-sight position of a speaker included in an image; a speech partner estimating section that estimates the speech partner of the speech target, and the line-of-sight position estimating section detects a target from the entire image excluding the part where the speaker is reflected in the image, At least one of the speaker's head position and pupil position in the image is detected, and based on the detected target and at least one of the head position and pupil position, the speaker's This is an image analysis device that estimates the position ahead of the line of sight.

According to one aspect of the present invention, it is possible to provide an image analysis device, an image analysis method, and a program that can appropriately infer the speaking partner based on the speaker who is the subject of the image and the content of the utterance. .

1 is a diagram showing a usage environment of an image analysis device 100 according to an embodiment. FIG. 4 is a diagram illustrating a line-of-sight detection method of the line-of-sight position estimating unit 132. 7 is a diagram illustrating an example of a result of visual line detection by the visual line position estimation unit 132. FIG. 3 is a diagram illustrating a procedure for estimating the content of a speaker's utterance by the utterance content estimation unit 140. FIG. FIG. 3 is a diagram schematically showing the flow of processing by the line-of-sight position estimating section 132 and the utterance content estimating section 140. 3 is a flowchart illustrating an example of the flow of estimation processing by the image analysis device 100. It is a schematic diagram of 100 A of image analysis apparatuses of 2nd Embodiment. It is a flowchart which shows an example of the flow of the estimation process by 100 A of image analysis apparatuses.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of an image analysis device, an image analysis method, and a program according to the present invention will be described below with reference to the drawings.

[overview]
The image analysis device is realized by one or more processors. The image analysis device acquires one or more images in which one or more people are captured, the speaker and the content of the utterance in the image, and based on the acquired information, determines the utterance partner (the person the speaker is talking to). guess. Note that the image may be a video. Furthermore, the image may be accompanied by imaging date and time information, geographic information, and the like.

The image analysis device estimates the speaker based on the characteristics of the image, the characteristics of the speaker, and the content of the utterance. The speaking partner may be the person captured in the image, or may be a person not included in the image (for example, the person who captured the image or the person existing in the real space assumed from the image). . This allows the image analysis device to infer the speaker and the other party from the image of the person.

<First embodiment>
[overall structure]
FIG. 1 is a diagram illustrating an example of the usage environment and configuration of an image analysis apparatus 100 according to the first embodiment. Image analysis device 100 communicates with request source terminal 200 via network NW. The network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, and the like. The request source terminal 200 is, for example, a computer device having a communication function. The communication function is realized by a network card such as a NIC (Network Interface Card), a wireless communication module, and the like.

The image analysis device 100 includes, for example, an acquisition unit 110, a feature derivation unit 120, a speaker estimation unit 130, an utterance content estimation unit 140, a speech partner estimation unit 150, an output unit 160, and a storage unit 170. Equipped with These components are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). (including circuitry), or may be realized by collaboration between software and hardware.

The acquisition unit 110 acquires one or more images including one or more people from the request source terminal 200. The acquisition unit 110 outputs the acquired image to the feature amount derivation unit 120. In the following, it is assumed that the images are of people.

The feature amount deriving unit 120 derives the feature amount of the image. Feature amounts are numerical values (including vectors and matrices) and other digital data calculated by analyzing an image. The image features include, for example, brightness, color, brightness distribution, saturation, color ratio, and the shape and size of each part of a person's face (eyes, nose, mouth, eyebrows, chin, hair, etc.). Note that the feature amount deriving unit 120 may identify the attributes of the person captured in the image and derive the feature amount of the person. The person's attributes include, for example, the person's age, gender, occupation, and patterned facial expressions (angry face, smiling face, etc.). In addition, when the images acquired by the acquisition unit 110 are a plurality of consecutive images or a video, the person's feature amount includes a movement vector of the person's reference position (such as the head or center of gravity of the body), Changes in the shape of a person (e.g. changes in facial expressions and gestures) and, if audio is associated with the image, features based on the audio (e.g., volume, pitch, intonation, etc.) are included. Good too.

Note that the feature amount deriving unit 120 may derive a scene title associated with the image as a feature amount based on the derived feature amount. A scene title is a simple explanation of an image. For example, it indicates a location such as "school" or "office," or indicates the location captured in an image such as "supervisor and subordinate" or "teacher and student." It also shows the relationship between people. The feature amount deriving unit 120 outputs the derived feature amounts to the speaker estimation unit 130, the line-of-sight position estimation unit 132, and the utterance content estimation unit 140.

The speaker estimation unit 130 estimates the speaker in the image based on the feature amount output by the feature amount derivation unit 120. For example, when the number of people captured in the image is one person, the speaker estimation unit 130 estimates the content of the utterance using that person as the speaker. In addition, for example, when there are two or more people captured in the image, the speaker estimation unit 130 determines the speaker based on the position ahead of the line of sight (the person at that position) among the people. presume. Furthermore, for example, if only one person in the image has their mouth open while other people in the image have their mouths closed, the speaker estimation unit 130 estimates that the person whose mouth is open is the speaker. do.

The speaker estimation unit 130 includes, for example, a line-of-sight position estimation unit 132. The line-of-sight position estimating unit 132 determines the position ahead of the speaker's line of sight (line-of-sight position) based on the feature quantity output by the feature quantity deriving unit 120 and the speaker estimated by the speaker estimation unit 130. Estimate. The line-of-sight position estimated by the line-of-sight position estimation unit 132 may be a position on the image plane or a position in real space assumed from the image.

Note that even if the line-of-sight position estimation unit 132 detects a person from the image, if part or all of the person's eyes cannot be confirmed (for example, if the person's back is visible in the image), (e.g., when the person in the image has their head hanging down and their face is not reflected), the line of sight may not be estimated. The method of estimating the line-of-sight position by the line-of-sight position estimation unit 132 will be described later. The line-of-sight position estimation unit 132 outputs the estimated line-of-sight position information to the utterance content estimation unit 140.

The utterance content estimation unit 140 is based on the feature quantity output by the feature quantity deriving unit 120, the speaker estimated by the speaker estimation unit 130, and the person's line-of-sight position estimated by the line-of-sight position estimation unit 132. , to infer the content of the speaker's utterance. The utterance content includes information about the object of the topic and the direction of the topic (for example, whether it is positive or negative).

Utterance content estimation section 140 outputs the estimated speaker and utterance content to speech partner estimation section 150 and output section 160.

The utterance partner estimation section 150 estimates to whom the utterance content outputted by the utterance content estimation section 140 is addressed. If there are two people captured in the image, the speaking partner estimation unit 150 infers that the person who is not the speaker is the speaking partner. If there are three or more people captured in the image, the speaking partner estimation unit 150 estimates the speaking partner based on the line of sight of the speaker, for example. If there is only one person captured in the image, the speaking partner estimation unit 150 estimates the speaking partner based on the scene title derived by the feature amount deriving unit 120, for example. The speaking partner estimation unit 150 outputs the estimated speaking partner to the output unit 160.

The output unit 160 outputs the utterance content outputted by the utterance content estimation unit 140 and the utterance partner outputted by the utterance partner estimation unit 150 to the request source terminal 200 shown in FIG.

The storage unit 170 stores utterance estimation information 171. The utterance estimation information 171 includes text information that is referenced by the utterance content estimation unit 140 and is used as reference information for the utterance content, and information regarding the context of the utterance content.

The utterance content estimation unit 140 may learn the estimation result by the utterance content estimation unit 140 by storing the estimation result as the utterance estimation information 171. Further, the utterance estimation information 171 may include a scene title derived by the feature amount deriving unit 120. In addition, the utterance content estimation unit 140 learns text information used as reference information of the utterance content and information regarding the context of the utterance content from other information sources as the utterance estimation information 171 via the network NW shown in FIG. You may let them.

[Gaze estimation]
FIG. 2 is a diagram illustrating a method of estimating the line-of-sight position by the line-of-sight position estimation unit 132. The line-of-sight position estimation method shown in FIG. 3 is realized, for example, by the following four-step procedure. Note that the following steps (a) and (b) may be performed in parallel, or the other process may be performed after one process is completed.
(a) Target object detection (b) Subject head and pupil position analysis (c) Line of sight calculation (d) Line of sight position (grid) estimation

Each of the steps (a) to (d) will be explained below.

(a) Target Detection The line-of-sight position estimating unit 132 detects the person as the subject and the candidate target that is ahead of the line of sight of the person as the subject of line-of-sight position estimation in the image (hereinafter referred to as image x _i ). Detection is performed from the entire image, excluding the part where the person is reflected. A target is an object, person, living thing, or other object that may attract your gaze.

Detection of the target object is realized by a deriving device 1 that uses a convolutional neural network (CNN) such as VGGNet, for example. Deriver 1 is generated in advance by machine learning. The line-of-sight position estimating unit 132 derives a saliency map S(x _i ) of size D×D from the image x _i .

(b) Analysis of the head and pupil positions of the subject The line-of-sight position estimating unit 132 estimates the direction of the line of sight based on the face orientation and pupil position of the subject. In order to estimate the direction of the line of sight, the line-of-sight position estimating unit 132 uses a derivation unit 2 that uses a convolutional neural network or the like to generate a spatial map different from the saliency map S(x _i ) from the image x _i , a line-of-sight mask map G(x _h , x _p ) of size D×D is derived. The derivator 2, like the derivator 1, is generated in advance by machine learning. x _p is a parameter (for example, coordinates) indicating the position of the head or eyes of the subject in the image x _i . Furthermore, x _h is an image parameter representing the degree of enlargement when the head position or pupil position of the subject in the image x _i obtained by deriving x _p is enlarged.

(c) Line-of-sight calculation The line-of-sight position estimating unit 132 uses the saliency map S(x _i ), which is the detection result of the target in step (a), and the line-of-sight mask map, which is the analysis result of the direction of the line of sight in (b). Based on G(x _h , x _p ), the direction of the subject is calculated, and the line-of-sight position of the subject is estimated. Here, if there are multiple people as subjects, the line-of-sight positions of all the people whose pupil positions can be estimated may be estimated, or if the speaker estimation unit 130 has already estimated the speaker. The position of the person's line of sight may be estimated.

For example, the line-of-sight position estimating unit 132 uses VGGNet to calculate each element of the saliency map S (x _i ) and the line-of-sight mask map G (x _h , x _p ) as shown in equation (1) below. By calculating the element-wise product, the line-of-sight position y (hat) of the subject (the left side of equation (1)) is calculated.

Note that F() on the right side of equation (1) is calculated from the element-by-element product of the saliency map S(x _i ) and the gaze mask map G(x _h , x _p ), and is calculated from the FC layer (Fully This is a function for deriving the parameters of the Connected Layer (layer that makes final decisions) in advance.

(d) Estimation of line-of-sight position (grid) The line-of-sight position estimating unit 132 estimates the line-of-sight position y (hat) of the subject based on the calculation result of step (c). The line-of-sight position estimating unit 132 may use only the line-of-sight mask map G (x _h , x _p ) or may use the line-of-sight mask map G (x _{h .}

Further, the line-of-sight position estimating unit 132 captures an image of the face of the subject from the front, and when the person is facing the camera (so-called looking at the camera), the line-of-sight position estimating unit 132 determines that the line-of-sight position is It may be assumed that the person is the person taking the image. In addition, when the line-of-sight position estimating unit 132 recognizes the face of the subject, if the person is looking toward the sky, it may be assumed that the person is not looking at anything, or it may be assumed that the person is not looking at anything outside the image. It may be a line of sight toward an unspecified person.

FIG. 3 is a diagram illustrating an example of the result of estimating the line-of-sight position by the line-of-sight position estimation unit 132. For example, the line-of-sight position estimating unit 132 estimates the line-of-sight position of the person on the right side of FIG. 3, and indicates the estimation result for which person the line-of-sight position has been estimated with a circle above the face. The image analysis device 100 assumes that the person on the right whose gaze position has been estimated by the gaze position estimation unit 132 is the speaker, and estimates the content of the utterance.

The line-of-sight position estimation process by the line-of-sight position estimation unit 132 can be expressed by the following equations (2) to (5).

In equation (2), a is the addressee class; a classification class that indicates whether the speaker's line of sight is a target in the image x _i , the photographer, or something else. , I ₁ indicates the feature amount obtained by the saliency map S (x _i ), I ₂ indicates the speaker feature amount obtained from the gaze mask map G (x _h , x _p ), and θ is the LSTM language model The model parameters in p(a|I ₁ , I ₂ , S ₁ , ..., S _T ) are shown. Words S ₁ -S _T forming the utterance content are indicated by vector values. The words S ₁ -S _T are extracted from a large number of candidates prepared in advance. Further, argmax is a function for determining a parameter that maximizes the target term, and p() is an expression indicating a line-of-sight position estimation model, which includes one or more modality streams. A modality stream is a component of a language expression model that expresses the speaker's judgment and feelings about the content being spoken. The modality stream is, for example, what is called a saliency-estimation-feature stream, a speaker-appearance-feature stream, or an utterance-based-feature stream. The utterance content estimation unit 140, for example, selects a method for extracting words S ₁ -S _T to be reflected in the utterance content according to the speaker's facial expressions and gestures, or selects a word group to be an extraction source from the utterance estimation information 171. I do things. θ* indicates the parameter θ that maximizes p(). Further, T is a positive integer indicating the number of words in the utterance content. In order to model the relationship shown in equation (2), the utterance content estimation unit 140 models each modality stream one by one using equations (3) to (5), for example.

In equations (3) to (5), W ₁ and W ₂ are network parameters, b ₁ and b ₂ are bias values, ReLU (Rectified Linear Unit) is a normalized linear function, and concat is It is a combination function. The utterance content estimation unit 140 processes the saliency map S(x _i ) using Equation (3), the gaze mask map G(x _h , x _p ) using Equation (4), and converts the processing results into Equation (5). Integrate with. Concat in Equation (5) indicates a concatenate function, which is a function that combines vectors.

[Utterance content estimation]
The utterance content estimation process by the utterance content estimation unit 140 is realized, for example, by the following two-step procedure.
(e) Whole image analysis (f) Utterance content estimation

Each of the steps (e) to (f) will be explained below.

(e) Whole image analysis The utterance content estimation unit 140 analyzes targets such as a person, target, background, etc. reflected in the image x _i , for example. The utterance content estimation unit 140 may analyze the target using the derivation results of the feature value derivation unit 120.

The utterance content estimation unit 140 analyzes the portion of the image x _i in which the speaker is reflected, based on the result of the speaker estimation unit 130 estimating the speaker from the person reflected in the image x _i , for example. Note that the utterance content estimation unit 140 may use part or all of the result of estimating the line of sight position by the line of sight position estimation unit 132 to predict the content of the utterance, or may use a part or all of the result of estimating the line of sight position by the line of sight position estimating unit 132, or may use the input of the line of sight from the requesting terminal 200. You may use the results.

(f) Utterance content estimation The utterance content estimation unit 140 composes a sentence likely to be uttered by the speaker based on, for example, the words indicating the target analyzed in step (e) and the words associated with these words. In this way, the content of the speaker's utterance can be inferred.

The utterance content estimation by the utterance content estimation unit 140 is realized by, for example, an LSTM (Long short-term memory) language model. The LSTM language model is a model realized by a recurrent neural network (RNN) that utilizes learning content over a long period of time, and for example, calculation processes are stored cumulatively and used for subsequent calculations. A method of performing processing using the LSTM language model is sometimes referred to as an image captioning (image captioning automatic generation) method. Note that the LSTM language model may be extended with an attention model (for example, complementary information that directs attention to the speaker or a portion of the image x _i indicating the speaker's line of sight).

FIG. 4 is a diagram showing the structure of the utterance content estimated by the utterance content estimation unit 140. For example, based on the image x _i shown in FIG. 3, the utterance content estimation unit 140 assumes that two people are on board the ship and are preparing to depart, and calculates the utterance content as "Are you ready for leave port?" The method for estimating this will be explained.

For example, when a sentence is constructed based on the general neural network language model shown in the upper part of Figure 4, the fourth word S ₄ (fourth word from the beginning of the sentence) that forms the sentence is derived. In the process, only the third word S ₃ (the third word from the beginning of the sentence) "ready" used immediately before the fourth word is used as input data.

On the other hand, when a sentence is constructed based on the LSTM language model shown in the lower part of Figure 4, the first to third words (“Are you ready”) are input in the process of deriving the fourth word that makes up the sentence. The fourth word is derived as data. Therefore, the LSTM language model can infer the next word to appear based on the sequence of consecutive words, and can improve the validity of the sequence of words and sentences.

The utterance content estimation process by the utterance content estimation unit 140 can be expressed by, for example, the following equations (6) to (8).

In Equation (6), t is time, x _t is the input information at time t (i.e., the gaze destination position estimation result by the gaze destination position estimation unit 132 or word S _t ), and u _t is the input information at time t. is a vector, and h _t-1 indicates the hidden state at time t-1. The right side of equation (6) indicates that u _t and h _t-1 are calculated using the LSTM language model. The right side of Equation (8) indicates a softmax function that adds the bias value b _fu to the product of the network parameter W _fu and the input information x _fu . The softmax function calculates the following equation (9). In equation (9), v is a vector.

[Guess who is speaking]
For example, when the line-of-sight position of the speaker has not been estimated by the line-of-sight position estimating unit 132, the speaking partner estimation unit 150 estimates the speaking partner based on the utterance content estimated by the utterance content estimation unit 140.

The estimation of the speaking partner by the speaking partner estimation unit 150 can be expressed by the following equations (10) to (15).

Equations (14) and (15) are equations for deriving the status c _t and h _t of the LSTM language model. In equation (14), _it is an input gate, _gt is an input modulation gate, _ot is an output gate, _{and ft} is a forgetting gate ( forgetting gate). Further, the sigmoid function σ() in equations (10) to (12) shows the relationship σ(x)=(1+e ^−x ) ⁻¹ . Furthermore, φ() in equations (13) and (15) indicates a hyperbolic tangent function of a complex number, and φ(x)=(e ^x −e ^−x )/(e ^x +e ^{− x} ). The symbol containing a black circle within a circle on the right side of Equation (14) and Equation (15) indicates the product of two vector values for each element (Adamar product).

Note that if the line-of-sight position of the speaker has been estimated by the line-of-sight position estimating unit 132, the speaking partner estimating unit 150 may estimate the speaking partner based on the estimation result. For example, if the speaker's line-of-sight position has been estimated by the line-of-sight position estimating unit 132 and a person appears in the position of the speaker's line of sight in image x _i , the speaker's line-of-sight position estimation unit 150 determines that The person at the line of sight is inferred as having a high possibility of being the conversation partner. For example, the speech partner estimation unit 150 may detect that the line-of-sight position of the speaker has been estimated by the line-of-sight position estimating unit 132, and that a person is reflected in the line-of-sight position of the speaker in the image x _i . is directed toward the speaker (as in FIG. 4, the eyes of the people meet), the person at the speaker's line of sight is inferred to be the person with whom the speaker is speaking.

Furthermore, the speech partner estimation unit 150 may verify the estimation accuracy of the speech partner by comparing the estimation result of the speech partner with a target at the speaker's line-of-sight position by the line-of-sight position estimating unit 132. . For example, when the speech partner estimation unit 150 does not match the estimation result of the speech partner and the target ahead of the speaker's line of sight according to the line-of-sight position estimating unit 132, the speech partner estimation unit 150 returns the processing to the speech content estimation unit 140, The utterance content estimation may be performed again.

FIG. 5 is a diagram schematically showing the flow of processing by the line-of-sight position estimation unit 132 and the utterance content estimation unit 140. Note that the upper part of FIG. 5 shows details of the line-of-sight position estimation process shown in FIG. 2.

The line-of-sight position estimating unit 132 uses a convolutional neural network such as VGGNet for the entire image x _i and information on the position of the speaker's head or pupil whose line-of-sight position has been estimated by the line-of-sight position estimating unit 132. Line of sight estimation is performed by combining the calculation results obtained by machine learning. Next, the utterance content estimation unit 140 estimates the utterance content using the LSTM language model. The image analysis device 100 integrates these calculation results and outputs them to the requesting terminal 200.

[Processing flow]
FIG. 6 is a flowchart showing an example of the flow of estimation processing by the image analysis device 100. First, the acquisition unit 110 acquires the image x _i from the requesting terminal 200, and outputs it to the feature amount derivation unit 120 (S100). Next, the feature amount deriving unit 120 derives the feature amount of the image x _i (S102). Next, the line-of-sight position estimating unit 132 estimates the speaker in the image x _i and estimates the line-of-sight position of the speaker (S104). Next, the utterance content estimation unit 140 estimates the utterance content (S106). Next, the speaking partner estimation unit 150 guesses the speaking partner (S108). Next, the output unit 160 outputs the estimation result to the request source terminal 200 (S110). This concludes the explanation of the processing of this flowchart.

As described above, the image analysis device 100 includes the acquisition unit 110 that acquires the image x _i of the speaker reflected in the image, and the line-of-sight position estimating unit that estimates the position ahead of the line of sight of the speaker reflected in the image x _i . 132, and an utterance content estimation unit 140 that estimates the utterance content of the speaker based on the image x _i and the estimated position ahead of the line of sight _. can be inferred appropriately.

The image analysis device 100 also includes an acquisition unit 110 that acquires an image x _i of the speaker, and a target utterance uttered by the speaker based on the prediction result of the utterance content by the image xi and the utterance content estimation unit 140. By including the speaking partner estimating unit 150 that estimates the speaking partner, the speaking partner can be appropriately estimated from the image x _i and the content of the speaker's utterance.

<Second embodiment>
FIG. 7 is a schematic diagram of an image analysis apparatus 100A according to the second embodiment. The image analysis device 100A differs from the image analysis device 100 of the first embodiment in that it further includes a related behavior estimation unit 180. Therefore, the related behavior estimation unit 180 will be mainly described below.

The related behavior estimation section 180 estimates the behavior of the speaker and the speech partner based on the estimation results of the gaze destination position estimation section 132 and the estimation results of the utterance content estimation section 140 and the speech partner estimation section 150. guess. Behaviors associated with speech include gestures such as nodding, waving, and clapping, changes in facial expressions such as smiling and glaring, and movements of the speaker or the speaking partner such as walking away, turning around, and sitting down. include.

Since human reactions are not necessarily limited to verbal communication, the related behavior estimation section 180 estimates the speaker's behavior even in situations where it is difficult for the utterance content estimation section 140 to estimate the utterance, so that the image x _i From this, it is possible to infer more natural situations and communication on the spot.

[Processing flow]
FIG. 8 is a diagram showing an example of the flow of estimation processing by the image analysis device 100A. The flowchart shown in FIG. 8 is the flowchart shown in FIG. 6 with S112 added. Therefore, S112 will be explained below.

After the process of S108, the related behavior estimation unit 180 estimates the behavior accompanying the utterance of the speaker (S112), and advances the process to S110. This concludes the explanation of the processing of this flowchart.

Note that in the image analysis devices 100 and 100A described above, detection of the position of the line of sight by the line of sight position estimating unit 132 and estimation of the content of the utterance by the utterance content estimation unit 140 may be omitted. In that case, as a substitute for the line-of-sight position estimated by the line-of-sight position estimation unit 132, the acquisition unit 110 inputs the user's line-of-sight detection result via the requesting terminal 200 (text information may be used, or a mark indicating the line-of-sight is added to the image). ) or the input of the content of the statement (text information or audio data may be used).

Although the mode for implementing the present invention has been described above using embodiments, the present invention is not limited to these embodiments in any way, and various modifications and substitutions can be made without departing from the gist of the present invention. can be added.

100, 100A...Image analysis device, 110...Acquisition unit, 120...Feature derivation unit, 130...Speaker estimation unit, 132...Gaze position estimation unit, 140...Utterance content estimation unit, 150...Speech partner estimation unit, 160 ...Output section, 170...Storage section, 171...Utterance inference information, 180...Related behavior inference section, 200...Request source terminal

Claims (7)

a speaker estimation unit that detects one or more persons included in the image and estimates that a person with an open mouth among the detected persons is the speaker;
a line-of-sight position estimating unit that estimates the estimated position of the speaker's line of sight;
a speech partner estimation unit that estimates a speech partner to whom the speaker is speaking based on the estimated position ahead of the line of sight;
Equipped with
The line-of-sight position estimating unit is
detecting a target from the entire image excluding the part where the speaker is reflected in the image;
detecting the position of the speaker's eyes within the image;
Estimating the position ahead of the speaker's line of sight based on the detected target and the position of the pupil ,
The speech partner estimation unit is
If one person is included in the detected target, the one person is inferred to be the utterance partner;
If the detected target includes two or more people, inferring the speaking partner based on the estimated position ahead of the line of sight;
If the detected target does not include a person, the person who took the image or a person who is not included in the image and who exists in the real space assumed from the image is presumed to be the utterance partner;
Image analysis device. The line-of-sight position estimating unit derives a saliency map indicating a detection result of the target in the image using a first derivator using a convolutional neural network.
The image analysis device according to claim 1. The line-of-sight position estimating unit derives a line-of-sight mask map, which is a spatial map different from the saliency map in the image, using a second derivator that uses a convolutional neural network.
The image analysis device according to claim 2. The line-of-sight position estimating unit estimates the position of the speaker's line-of-sight by calculating the product of the saliency map and the line-of-sight mask map for each element.
The image analysis device according to claim 3. The speaking partner inferring unit estimates the person as the speaking partner when a person is reflected in the image at a position ahead of the estimated line of sight of the speaking person.
An image analysis device according to any one of claims 1 to 4. The computer is
Detecting one or more people included in the image, estimating that a person with an open mouth among the detected people is the speaker,
estimating the estimated position ahead of the speaker's line of sight;
Presuming a speaking partner to whom the speaker is speaking based on the estimated position ahead of the line of sight;
An image analysis method, comprising:
detecting a target from the entire image excluding the part where the speaker is reflected in the image;
detecting the position of the speaker's eyes within the image;
Estimating the position ahead of the speaker's line of sight based on the detected target and the position of the pupil ,
If one person is included in the detected target, the one person is inferred to be the utterance partner;
When the detected target includes two or more people, inferring the speaking partner based on the estimated position ahead of the line of sight;
If the detected target does not include a person, the person who took the image or a person who is not included in the image and exists in the real space assumed from the image is inferred to be the utterance partner;
Image analysis method. to the computer,
detecting one or more people included in the image, and estimating that a person with an open mouth among the detected people is the speaker;
estimating the estimated position ahead of the speaker's line of sight;
Based on the estimated position ahead of the line of sight, the person to whom the speaker is speaking is inferred ;
A program,
Detecting a target from the entire image excluding the part where the speaker is reflected in the image,
detecting the position of the speaker's eyes within the image;
Estimating the position ahead of the speaker's line of sight based on the detected target and the position of the pupil;
If one person is included in the detected target, the one person is inferred to be the utterance partner;
When the detected target includes two or more people, the speaker is inferred based on the estimated position ahead of the line of sight;
If the detected target does not include a person, the person who took the image or a person who is not included in the image and exists in the real space assumed from the image is presumed to be the utterance partner;
program.

Images