JP7093935B2 - Estimating system, estimation device, estimation method and computer program - Google Patents

Description

The present invention relates to an estimation system, an estimation device, an estimation method and a computer program.

Conventionally, there is a technique of estimating the direction in which a person is gazing from a color image showing a stationary human head (see, for example, Non-Patent Document 1).

Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba, “Gaze360: Physically Unconstrained Gaze Estimation in the Wild”

However, although the conventional technique can estimate the gaze direction of a passerby, it cannot estimate the position in the real space where the gaze starts, and it estimates the estimation result which is impossible even as the range of motion of the neck. Sometimes. Therefore, there is a problem that the estimation accuracy is lowered.

In view of the above circumstances, it is an object of the present invention to provide a technique capable of improving the estimation accuracy of the gaze start position and the gaze direction of a passerby.

One aspect of the present invention is to photograph a region including a body part of a passerby, a first imaging device that generates one or more first images, and a region including the body part of the passerby. A second photographing device that generates one or more second images in which information on the distance in the depth direction is stored, and the person area by inputting information on the person area and three-dimensional data corresponding to the person area. Generated from the information of the person area extracted from the first image and the second image for the trained model trained to output the information of the gaze start position and the gaze direction of the person indicated by. It is an estimation system including an estimation unit that estimates a gaze start position and a gaze direction of the passerby by inputting point group data or a depth image which are three-dimensional data.

One aspect of the present invention is the estimation system, in which a depth image cutting unit that extracts a person area from the second image and a restoration unit that restores the three-dimensional data based on the extracted person area. And further prepare.

One aspect of the present invention is the estimation system described above, wherein the region including the body portion of the passerby is a region including the body portion above the position of the waist of the passerby, from the first image. The first portion of the body part detection unit that detects the area including the body part of the passerby and the detected area including the body part of the passerby, including at least the head and shoulder areas of the person. The depth image cutting section further includes an image cutting section for extracting from an image, and the depth image cutting section includes a partial region including at least a body part of a person among the detected body parts of the passerby as the second image. Extract from.

One aspect of the present invention is the above estimation system, in which information on a person area and the three-dimensional data corresponding to the person area are input with teacher data in which information on a gaze start position and gaze direction is associated with each other. Further, a learning unit for generating the trained model is provided.

One aspect of the present invention stores one or more first images of a region including a passerby's body part and information on the distance in the depth direction with respect to the region including the passerby's body part1. An acquisition unit that acquires two or more second images, information on the person area, and three-dimensional data corresponding to the person area are input, and information on the gaze start position and gaze direction of the person indicated in the person area is output. For the trained model trained to do so, the information of the person area extracted from the first image acquired by the acquisition unit and the three-dimensional data generated from the second image . It is an estimation device including an estimation unit that estimates the gaze start position and gaze direction of the passerby by inputting group data or a depth image .

One aspect of the present invention stores one or more first images of a region including a passerby's body part and information on the distance in the depth direction with respect to the region including the passerby's body part1. The acquisition step of acquiring two or more second images, the information of the person area and the three-dimensional data corresponding to the person area are input, and the information of the gaze start position and the gaze direction of the person shown in the person area is output. For the trained model trained to do so, the information of the person area extracted from the first image acquired by the acquisition unit and the three-dimensional data generated from the second image . It is an estimation method including an estimation step for estimating a gaze start position and a gaze direction of the passerby by inputting group data or a depth image .

One aspect of the present invention stores one or more first images of a region including a passerby's body part and information on the distance in the depth direction with respect to the region including the passerby's body part1. The acquisition step of acquiring two or more second images, the information of the person area and the three-dimensional data corresponding to the person area are input, and the information of the gaze start position and the gaze direction of the person shown in the person area is output. For the trained model trained to do so, the information of the person area extracted from the first image acquired by the acquisition unit and the three-dimensional data generated from the second image . It is a computer program for causing a computer to perform an estimation step of estimating a gaze start position and a gaze direction of the passerby by inputting group data or a depth image .

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to improve the estimation accuracy of the gaze start position and the gaze direction of a passerby.

It is a block diagram which shows the structure of the estimation system in embodiment of this invention. It is a block diagram which shows the specific example of the functional structure of the learning apparatus in this embodiment. It is a figure which shows an example of the learning model used in this embodiment. It is a block diagram which shows the specific example of the functional structure of the estimation apparatus in this embodiment. It is a flowchart which shows the flow of the estimation process of the estimation device in this embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram showing a configuration of an estimation system 100 according to an embodiment of the present invention. The estimation system 100 is a system that estimates the gaze start position and gaze direction of a passerby. The passerby in the present invention is a person passing through an indoor passage. The gaze start position is a reference position for estimating the direction in which the passerby is gaze. For example, the gaze start position is the position between the eyebrows of a person. The gaze direction is the direction in which a passerby is gaze. It should be noted that gaze has a different meaning from gaze in that the angle of the eyes is not measured. In addition, gaze has a different meaning from face orientation in that it does not consider whether or not it is being viewed.

The estimation system 100 includes a learning device 10, an image sensor 20 (first imaging device), a depth sensor 30 (second imaging device), and an estimation device 40. The learning device 10 and the estimation device 40 are communicably connected by wire or wirelessly. The image sensor 20, the depth sensor 30, and the estimation device 40 are connected to each other so as to be communicable by wire or wirelessly. The image sensor 20 and the depth sensor 30 are installed in the same direction so that an area including a body part of a passerby can be photographed. The area including the body part of the passerby is the area including the body part above the position of the waist of the passerby, and includes, for example, at least the area of the face and shoulders of the passerby. The area above the waist of a passerby may be an area 3/4 or more from the position of the waist, for example, based on the line segment connecting the position of the midpoint of both shoulders of the person, the position of the neck, and the position of the waist. .. It is more preferable that the image sensor 20 and the depth sensor 30 are arranged close to each other. The region including the body part of the passerby may be the region including the whole body of the passerby.

The learning device 10 generates a trained model by learning a learning model by inputting teacher data. The teacher data is data for learning used for supervised learning, and is data represented by a combination of input data and output data that is assumed to have a correlation with the input data. The teacher data input to the learning device 10 is data in which a color image and a point cloud data in which an area above the waist of a passerby is photographed are associated with information on a gaze start position and a gaze direction of the passerby. .. The information on the gaze start position of a passerby in the teacher data is a start vector represented by the position (x, y, z) between the eyebrows of the person. The information on the gaze direction of a passerby in the teacher data is a direction vector represented by the gaze direction (x, y, z) of the person.

The learning device 10 inputs a color image and a point cloud data to the learning model, and generates a trained model trained to output information on a passerby's gaze start position and gaze direction. Here, learning is optimizing the coefficients used in the machine learning model. For example, learning is adjusting the coefficients used in a machine learning model so that the loss function is minimized. The coefficients used in the machine learning model are, for example, weight values and bias values.

It is assumed that the coordinate systems of the color image and the point cloud data in the teacher data are associated in advance on a pixel-by-pixel basis. It is desirable that the color image and the point cloud data are information obtained by photographing the same person from the same direction.

The image sensor 20 generates a color image (first image) by photographing an area including a body part of a passerby. The image sensor 20 outputs the generated color image to the estimation device 40. The color image contains time information regarding the time when the image was taken.

The depth sensor 30 generates a depth image (second image) by photographing an area including a body part of a passerby. The depth image is an image in which information on the distance in the depth direction from a measuring device (for example, a camera) is stored in each pixel value of the image. The depth sensor 30 outputs the generated depth image to the estimation device 40. The depth image contains time information about the time when the image was taken.

The estimation device 40 uses the trained model output from the learning device 10, the color image generated by the image sensor 20, and the depth image generated by the depth sensor 30 to capture a person imaged in the image. Estimate the gaze start position and gaze direction. The estimation device 40 is configured by using an information processing device such as a personal computer.

FIG. 2 is a block diagram showing a specific example of the functional configuration of the learning device 10 in the present embodiment.
The learning device 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes a program. The learning device 10 functions as a device including a learning model storage unit 11, a teacher data input unit 12, and a learning unit 13 by executing a program. All or part of each function of the learning device 10 may be realized by using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field-Programmable Gate Array), and the like. .. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a flexible disk, a magneto-optical disk, a portable medium such as a ROM (Read Only Memory) or a CD-ROM, or a storage device such as a hard disk built in a computer system. The program may be transmitted over a telecommunication line.

The learning model storage unit 11 is configured by using a storage device such as a magnetic storage device or a semiconductor storage device. The learning model storage unit 11 stores the learning model in machine learning in advance. Here, the learning model is information indicating a machine learning algorithm used when learning the relationship between the input data and the output data. In this embodiment, a neural network such as a multi-layer perceptron is used as a learning model for machine learning. In this embodiment, for example, ResNet (Residual Network) and PointNet are used as an example of the neural network.

The teacher data input unit 12 has a function of inputting teacher data. In the present embodiment, the combination of the color image and the point cloud data captured in the area above the waist of the passerby is used as the input data, and the information on the gaze start position and the gaze direction of the passerby captured in the image is output. Let it be data. Then, a combination of the input data and the output data is used as one sample data, and a set of a plurality of sample data is generated in advance as teacher data.

For example, the teacher data input unit 12 is communicably connected to an external device (not shown) that stores the teacher data generated in this way, and inputs teacher data from the external device via the communication interface. do. Further, for example, the teacher data input unit 12 may be configured to input teacher data by reading the teacher data from a recording medium that stores the teacher data in advance. The teacher data input unit 12 outputs the teacher data input in this way to the learning unit 13.

FIG. 3 is a diagram showing an example of the learning model 53 used in the present embodiment.
The learning model 53 is composed of a first feature extraction network 531, a second feature extraction network 532, and fully connected layers 533 to 538. The learning model 53 is stored in the learning model storage unit 11. The learning model 53 takes the color image 51 and the point cloud data 52, which are teacher data, as inputs, and outputs information on the gaze start position and the gaze direction. In the state before learning, a coefficient initialized with a random value is set in the learning model 53.

The first feature extraction network 531 is a convolutional neural network having the same configuration as ResNet. The first feature extraction network 531 is, for example, a network called ResNet18. ResNet18 has a layer called Residual block composed of 18 layers, and outputs a feature amount (tensor on the first floor) by inputting information (tensor on the fourth floor) in which color images are arranged. Hereinafter, the feature amount extracted by the first feature extraction network 531 will be referred to as a first feature amount. In the state before learning, a coefficient initialized with a random value is set in the first feature extraction network 531.

The second feature extraction network 532 is a convolutional neural network having a configuration similar to that of PointNet. PointNet takes point cloud data (third-order tensor) as input and outputs features (first-order tensor). In the present embodiment, the second feature extraction network 532 utilizes the global feature in PointNet as a feature quantity. Hereinafter, the feature amount extracted by the second feature extraction network 532 will be referred to as a second feature amount. In the state before learning, a coefficient initialized with a random value is set in the second feature extraction network 532.

The first feature amount extracted from the first feature extraction network 531 and the second feature amount extracted from the second feature extraction network 532 are input to the fully connected layer 533 and the fully connected layer 536, respectively. The fully connected layers 533 to 535 are used to determine the gaze start position. The fully connected layers 536 to 538 are used to determine the gaze direction.

The fully-bonded layer 533 and the fully-bonded layer 536 bind the input first feature amount and the second feature amount. The fully connected layers 534 and 535 are connected to the subsequent stage of the fully connected layer 533 in order to combine the fully connected layers 533 to determine the gaze start position. Then, the gaze start position (first-order tensor) is obtained from the fully connected layer 535. As described above, the output of the fully connected layer 535 is an estimation start vector representing the information of the gaze start position obtained by estimation. The estimation start vector is a start vector represented by the position (x, y, z) between the eyebrows of the person estimated by the learning model 53.

The fully connected layers 537 and 538 are connected to the subsequent stage of the fully connected layer 536 in order to combine the fully connected layers 536 to determine the gaze direction. Then, the gaze direction (first-order tensor) is obtained from the fully connected layer 538. As described above, the output of the fully connected layer 538 is an estimated direction vector representing the information of the gaze direction obtained by the estimation. The estimation direction vector is a direction vector represented by the gaze direction (x, y, z) estimated by the learning model 53. In the state before learning, a coefficient initialized with a random value is set in the fully connected layers 533 to 538.

As mentioned above, the learning model 53 is designed to extract features from the inputs in ResNet and PointNet, from which the outputs to the features are learned in the fully connected layer.

Returning to FIG. 2, the description of the configuration of the learning device 10 will be continued.
The learning unit 13 generates a trained model by learning the teacher data output from the teacher data input unit 12 based on the learning model. The generated trained model is input to the estimation device 40. The input of the trained model to the estimation device 40 may be performed via communication between the learning device 10 and the estimation device 40, or may be performed via a recording medium on which the trained model is recorded.

Next, a specific learning process of the learning unit 13 will be described. First, the learning unit 13 calculates a start position error between the gaze start position (estimated start vector) obtained by inputting the teacher data into the learning model and the gaze start position information (start vector) included in the teacher data. .. Next, the learning unit 13 calculates a direction error between the gaze direction information (estimated direction vector) obtained by inputting the teacher data into the learning model and the gaze direction information (direction vector) included in the teacher data. do. Then, the learning unit 13 updates the coefficient used in the learning model 53 by solving the minimization problem for the objective function determined based on the start position error and the direction error. Specifically, the learning unit 13 updates the coefficients of the first feature extraction network 531, the second feature extraction network 532, and the fully connected layers 533 to 538 used in the learning model 53. The learning unit 13 repeatedly updates the coefficients until the coefficients used in the learning model 53 are optimized, or a predetermined number of times. The coefficients of the learning model 53 are estimated by the error backpropagation method and the stochastic gradient descent method (SGD). As the optimization method, an optimization algorithm other than the stochastic gradient descent method may be used as long as the error back propagation method and the following optimization algorithm are combined. Optimization algorithms other than the stochastic gradient descent method include, for example, Adam, Adamax, Adagrad, RMSProp, Addaleta and the like.

The learning unit 13 outputs the coefficient obtained by the above processing and the learning model 53 to the estimation device 40 as a trained model.

FIG. 4 is a block diagram showing a specific example of the functional configuration of the estimation device 40 in the present embodiment.
The estimation device 40 includes a communication unit 41, a display unit 42, an operation unit 43, a storage unit 44, and a control unit 45.
The communication unit 41 communicates with other devices. Other devices are, for example, a learning device 10, an image sensor 20, and a depth sensor 30. The communication unit 41 receives the trained model transmitted from the learning device 10. The communication unit 41 receives the color image transmitted from the image sensor 20. The communication unit 41 receives the depth image transmitted from the depth sensor 30. When the trained model is recorded on an external recording medium such as a USB memory or an SD card, the communication unit 41 receives the trained model via the external recording medium.

The display unit 42 is an image display device such as a liquid crystal display, an organic EL (Electro Luminescence) display, and a CRT (Cathode Ray Tube) display. The display unit 42 displays the estimation result according to the operation of the user. The display unit 42 may be an interface for connecting the image display device to the estimation device 40. In this case, the display unit 42 generates a video signal for displaying the estimation result, and outputs the video signal to the image display device connected to the display unit 42.

The operation unit 43 is configured by using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), a touch panel, and buttons. The operation unit 43 is operated by the user when inputting the user's instruction to the estimation device 40. For example, the operation unit 43 receives an input of an instruction to start the estimation process. Further, the operation unit 43 may be an interface for connecting the input device to the estimation device 40. In this case, the operation unit 43 inputs the input signal generated in response to the user's input in the input device to the estimation device 40.

The storage unit 44 stores the trained model 441, the sensor parameter 442, and the estimation result 443. The storage unit 44 is configured by using a storage device such as a magnetic storage device or a semiconductor storage device.
The trained model 441 is a trained model trained by the learning device 10. The trained model is associated with the training model 53 and the latest coefficient information.

The sensor parameter 442 is a camera parameter of the depth sensor 30. Specifically, the sensor parameter 442 includes an internal parameter of the depth sensor 30 (center coordinate position of the x-axis and y-axis sensors, a parameter for correcting distortion) and an external parameter of the depth sensor 30 (relative to the world coordinate system). Three-dimensional position of the sensor, sensor rotation information).

The estimation result 443 is information on the gaze start position and gaze direction of the passerby estimated using the trained model 441.

The control unit 45 controls the entire estimation device 40. The control unit 45 is configured by using a processor such as a CPU and a memory. By executing the program, the control unit 45 realizes the functions of the acquisition unit 451, the body part detection unit 452, the color image cutting unit 453, the depth image cutting unit 454, the point cloud restoration unit 455, and the estimation unit 456.

A part or all of the functional parts of the acquisition unit 451, the body part detection unit 452, the color image cutting unit 453, the depth image cutting unit 454, the point cloud restoration unit 455, and the estimation unit 456 are hardware such as ASIC, PLD, and FPGA. It may be realized by ware, or it may be realized by collaboration between software and hardware. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is a non-temporary storage medium such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, or a storage device such as a hard disk built in a computer system. The program may be transmitted over a telecommunication line.

Some of the functions of the acquisition unit 451, the body part detection unit 452, the color image cutting unit 453, the depth image cutting unit 454, the point cloud restoration unit 455, and the estimation unit 456 do not need to be mounted on the estimation device 40 in advance. , An additional application program may be implemented by installing it in the estimation device 40.

The acquisition unit 451 acquires various data. For example, the acquisition unit 451 acquires the trained model. Further, for example, the acquisition unit 451 acquires a color image and a depth image.

The body part detection unit 452 detects a person's body part (based on the skeleton model) in pixel units from the color image acquired by the acquisition unit 451. Specifically, the body part detection unit 452 detects a person's body part on a pixel-by-pixel basis by using BodyPix (see Reference 1) in the Semantic Segmentation. Hereinafter, the region of the body part of the person detected by the body part detection unit 452 will be referred to as the body part region. The body part detection unit 452 outputs the information of the body part region to the color image cutting unit 453 and the depth image cutting unit 454. BodyPix is a technique that predicts body parts on a pixel-by-pixel basis from the relationship with peripheral pixels using a neural network learned in advance.
(Reference 1: “Overview of BodyPix: Real-time human segmentation using a browser and TensorFlow.js”, <URL: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html>)

The color image cutting unit 453 extracts the body part region from the color image based on the color image acquired by the acquisition unit 451 and the information of the body part region output from the body part detection unit 452. The body part region extracted by the color image cutting unit 453 is, for example, a region above the waist of a passerby. The color image cutting unit 453 extracts a body part region in a rectangular region, for example, a square region. This is because the background is ignored in the ResNet process, so there is no problem even if the extracted area contains the background portion. The color image cutting unit 453 may extract an area along the body of a person.

The depth image cutting unit 454 extracts the body part region from the depth image based on the depth image acquired by the acquisition unit 451 and the information of the body part region output from the body part detection unit 452. The body part region extracted by the depth image cutting unit 454 extracts, for example, the region of the body part of a passerby. The area of the body part of the passerby does not have to include the area of the face of the person, and may include at least the area of the shoulder. The depth image cutting unit 454 extracts an area along the body of a person. This is because the background cannot be ignored in the PointNet process.

The point cloud restoration unit 455 restores the point cloud data based on the information of the region extracted by the depth image cutting unit 454 and the sensor parameter 442. Restoration of the point cloud is realized by a process called Deprojection or backprojection.

The estimation unit 456 inputs the color image of the region above the waist of the passerby extracted by the color image cutting unit 453 and the point cloud data restored by the point cloud restoration unit 455 into the trained model 441. Estimates the gaze start position and gaze direction of passersby. Specifically, the estimation unit 456 outputs an estimation start vector and an estimation direction vector, which are estimation results of the gaze start position and the gaze direction of a passerby.

FIG. 5 is a flowchart showing a flow of estimation processing of the estimation device 40 in the present embodiment.
The acquisition unit 451 acquires a color image output from the image sensor 20 (step S101). The acquisition unit 451 outputs the acquired color image to the body part detection unit 452 and the color image cutting unit 453. The acquisition unit 451 acquires the depth image output from the depth sensor 30 (step S102). For example, the acquisition unit 451 acquires a depth image at the same time as the time information included in the color image. The acquisition unit 451 outputs the acquired depth image to the depth image cutting unit 454.

When real-time performance is not required in the processing of steps S101 and S102, the acquisition unit 451 may acquire a color image and a depth image stored in an external recording medium, or may acquire a color image and a depth image. Color images and depth images may be acquired from other stored devices.

The body part detection unit 452 detects the body part of a person in pixel units from the color image acquired by the acquisition unit 451 (step S103). The body part detection unit 452 outputs the detected information of the body part region to the color image cutting unit 453 and the depth image cutting unit 454. The color image cutting unit 453 extracts the body part region from the color image based on the color image acquired by the acquisition unit 451 and the information of the body part region output from the body part detection unit 452 (step S104). ..

When the color image cutting unit 453 extracts a body part area from a color image in a square area, the area corresponding to the pixel indicated by the information of the body part area in the extracted square area is the pixel of the color image. Pixel values are assigned, and a value of 0 is assigned to the area corresponding to the pixel not indicated by the information of the body part area. The color image cutting unit 453 outputs the information of the extracted body part region to the depth image cutting unit 454 and the estimation unit 456.

The depth image cutting unit 454 extracts the body part region from the depth image based on the depth image acquired by the acquisition unit 451 and the information of the body part region output from the body part detection unit 452 (step S105). .. Depth information is assigned to the body part region extracted by the depth image cutting unit 454 instead of the pixel value. The depth image cutting unit 454 outputs the information of the extracted body part region to the point cloud restoration unit 455.

The point cloud restoration unit 455 restores the point cloud data based on the information of the body part region extracted by the depth image cutting unit 454 and the sensor parameter 442 (step S106). The point cloud restoration unit 455 outputs the restored point cloud data to the estimation unit 456, and the estimation unit 456 is restored by the color image of the body part region extracted by the color image cutting unit 453 and the point cloud restoration unit 455. Import the point cloud data. The estimation unit 456 estimates the gaze start position and gaze direction of the passerby based on the captured color image of the body part region and the point cloud data (step S107).

Specifically, first, the estimation unit 456 reads out the learned model 441 stored in the storage unit 44. Next, the estimation unit 456 inputs a color image of the body part region and point cloud data to the learning model 53 included in the read learned model 441. For example, the estimation unit 456 inputs a color image of the body part region to the first feature extraction network 531 and inputs the point cloud data to the second feature extraction network 532. As a result, the estimation start vector and the estimation direction vector are output from the learning model 53. The estimation unit 456 acquires the estimation start vector and the estimation direction vector output from the learning model 53 as the estimation results of the gaze start position and the gaze direction.

According to the estimation system 100 configured as described above, it is possible to improve the estimation accuracy of the gaze start position and the gaze direction of a passerby. Specifically, the estimation system 100 captures an image sensor 20 that captures an area including a passerby's body part to generate a color image, and captures an area including a passerby's body portion to obtain a distance in the depth direction. Learning to input a depth sensor 30 that generates a depth image in which information is stored, a color image including a person area, and point group data, and output information on a passerby's gaze start position and gaze direction. The completed model is provided with an estimation unit 456 that estimates the gaze start position and gaze direction of a passerby by inputting information on a person area extracted from a color image and point group data generated from a depth image. .. This makes it possible to perform gaze estimation by taking into account not only the head of a passerby but also the region including the body part. Therefore, it is possible to reduce the estimation of the direction that is impossible as the range of motion of the neck as the gaze direction as in the conventional case. Furthermore, the trained model is used to estimate the gaze start position and gaze direction of passersby. Therefore, it is possible to improve the estimation accuracy of the gaze start position and the gaze direction of the passerby.

Further, the estimation system 100 includes a depth image cutting unit 454 that extracts a person area from a depth image, and a restoration unit 455 that restores three-dimensional data based on the extracted person area. As a result, it is possible to input the three-dimensional data of only the person area not including the background portion into the trained model. Therefore, it is possible to suppress a decrease in estimation accuracy caused by inputting three-dimensional data including a background portion into the trained model. Therefore, it is possible to improve the estimation accuracy of the gaze start position and the gaze direction of the passerby.

Further, in the estimation system 100, the region including the body portion of the passerby is the region including the body portion above the position of the waist of the passerby, and the region including the body portion of the passerby is detected from the color image. Further includes a detection unit 452 and a color image cutting unit 453 that extracts at least a partial region including a head and shoulder region of a person from a color image among the regions including the detected body parts of a passerby. Then, the depth image cutting unit 454 extracts from the depth image a partial region including at least the body portion of the person among the detected regions including the body portion of the passerby. This makes it possible to detect and extract a region including a passerby's body part from a color image taken in real time. Therefore, it is possible to estimate the gaze start position and gaze direction of a passerby in real time. Therefore, it becomes possible to improve convenience.

<Modification example>
In the above embodiment, a person who is passing through an indoor passage as a passerby has been described as an example, but the passerby may include a person walking outdoors.
The learning device 10 and the estimation device 40 may be integrated and configured. For example, the estimation device 40 may be configured to have the function of the learning device 10. When configured in this way, the estimation device 40 has a learning mode and an inference mode. Switching between the learning mode and the inference mode is performed by the operation unit 43. When the estimation device 40 operates in the learning mode, the estimation device 40 executes the process performed by the learning device 10. More specifically, the estimation device 40 performs learning of the learning model. When the estimation device 40 operates in the inference mode, the estimation device 40 executes the inference process shown in FIG.

In the above embodiment, the estimation device 40 inputs one color image 51 and point cloud data 52 to estimate the gaze start position and gaze direction of a passerby. The estimation device 40 may be configured to input a plurality of color images 51 and a plurality of point cloud data 52 that are continuous in time to estimate a gaze start position and a gaze direction of a passerby. The plurality of color images 51 that are continuous in time are a plurality of color images arranged in a time series constituting a so-called moving image. In this case, the learning device 10 also inputs the series data of the combination of the color image and the point cloud data to perform learning. The estimation device 40 uses the input plurality of color images 51 and the plurality of point cloud data 52 to estimate the gaze start position and gaze direction of the passerby imaged in each of the color images 51.

In the above embodiment, the point cloud data has been described as an example as the input data for the second feature extraction network 532, but the point cloud data represented by a mesh or a voxel may be used. When the point cloud data represented by the mesh is used, MeshNet (see Reference 2) is used instead of PointNet as the second feature extraction network 532. In order to use MeshNet, it is necessary to generate point cloud data represented by a mesh. In order to generate the point cloud data represented by the mesh, it is necessary to perform meshing after restoring the depth image to the point cloud data. Therefore, the point cloud restoration unit 455 performs a meshing after restoring the point cloud data based on the information of the region extracted by the depth image cutting unit 454 and the sensor parameter 442. As a specific algorithm of Meshing, the technique of Ball-Pivoting Algorithm described in Reference 3 may be used.
When using the point cloud data represented by the mesh, the point cloud data represented by the generated mesh is input to the second feature extraction network 532, and after obtaining the first-order tensor as output, the same as PointNet. It may be connected.
(Reference 2: Yutong Feng et al., “MeshNet: Mesh Neural Network for 3D Shape Representation”, AAAI 2019, <URL: https://arxiv.org/pdf/1811.11424.pdf>)
(Reference 3: Bernardini, F .; Mittleman, J .; Rushmeier, H .; Silva, C .; Taubin, G. “The ball-pivoting algorithm for surface reconstruction” Transactions on Visualization and Computer Graphics 1999, 5, 349 -359, <URL: https://www.cs.jhu.edu/~misha/Fall13b/Notes/Bernardini99.notes.pdf>)

When using the point cloud data case represented by voxels, 3D ShipeNet (see Reference 4) is used instead of PointNet as the second feature extraction network 532. In order to use 3D ShapeNet, it is necessary to generate point cloud data expressed in voxels. In order to generate the point cloud data expressed by voxels, it is necessary to restore the depth image to the point cloud data and then perform discretization at 1 cm square. Therefore, the point cloud restoration unit 455 performs discretization at a 1 cm square after restoring the point cloud data based on the information of the region extracted by the depth image cutting unit 454 and the sensor parameter 442.
When using the point cloud data represented by voxels, the point cloud data represented by the generated voxels is input to the second feature extraction network 532, and after obtaining the first-order tensor as output, the same as PointNet. It may be connected.
(Reference 4: Z. Wu et al., “3D ShapeNets: A Deep Representation for Volumetric Shape Modeling”. CVPR, 2015, <URL: https://pdfs.semanticscholar.org/3ed2/3386284a5639cb3e8baaecf496caa766e335.pdf>)

In the above embodiment, the configuration using PointNet as the second feature extraction network 532 is shown. In this case, the input data for the second feature extraction network 532 is point cloud data. The input data for the second feature extraction network 532 is not limited to the point cloud data, and other three-dimensional data may be used. Other 3D data is, for example, a depth image. When a depth image is used as input data for the second feature extraction network 532, the estimation device 40 does not have a point cloud restoration unit 455, and as the second feature extraction network 532 in the learning model 53, Deepth CNNs (see Reference 5). Is configured to use. In Depth CNNs, the depth image is input and the first-order tensor is output. When Deep CNNs are used, the depth image is HHA (Horizontal disparity, Height above ground, and Angle with gravity) coding in the depth image cutting unit 454. After that, the depth image cutting unit 454 inputs the HHA-coded depth image to the second feature extraction network 532. Then, in the learning model 53, a first-order tensor (corresponding to 512 fc shown in Figure 5 of Reference 5) obtained by the second feature extraction network 532 is used. When configured in this way, the learning device 10 uses a combination of a color image and a depth image as teacher data.
(Reference 5: Xinhang Song et al., “Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs”, AAAI, 2017 ”, <URL: https://arxiv.org/pdf /1801.06797.pdf ＞))

In the above embodiment, as an example of the neural network, a configuration using ResNet and PointNet is shown, but the neural network used is not limited to this. For example, as the first feature extraction network 531, another convolutional neural network may be used instead of ResNet. Examples of other convolutional neural networks include MobileNetV1, V2, V3, EffectNet, NASNet (Neural Architecture Search Net), MNASNet (Mobile Neural Architecture Search Net), and the like.
Further, as the second feature extraction network 532, another graph neural network may be used instead of PointNet. Examples of other graph neural networks include PointNet ++ and ShellNet.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

10 ... learning device, 20 ... image sensor, 30 ... depth sensor, 40 ... estimation device, 11 ... learning model storage unit, 12 ... teacher data input unit, 13 ... learning unit, 41 ... communication unit, 42 ... display unit, 43 ... operation unit, 44 ... storage unit, 45 ... control unit, 451 ... acquisition unit, 452 ... body part detection unit, 453 ... color image cutting unit, 454 ... depth image cutting unit, 455 ... point group restoration unit, 456 ... estimation Part, 531 ... 1st feature extraction network, 532 ... 2nd feature extraction network, 533-538 ... Fully connected layer

Claims (7)

A first imaging device that captures an area including a passerby's body part and generates one or more first images.
A second photographing device that photographs an area including a body part of a passerby and generates one or more second images in which information on the distance in the depth direction is stored.
For the trained model trained to output the gaze start position and gaze direction information of the person indicated in the person area by inputting the information of the person area and the three-dimensional data corresponding to the person area, the first The gaze start position and gaze direction of the passerby are estimated by inputting the information of the person area extracted from the image 1 and the point group data or the depth image which is the three-dimensional data generated from the second image. And the estimation part to do
An estimation system equipped with. A depth image cutting unit that extracts a person area from the second image,
A restoration unit that restores the three-dimensional data based on the extracted person area, and
The estimation system according to claim 1, further comprising. The region including the body part of the passerby is a region including at least the body part above the position of the waist of the passerby.
A body part detection unit that detects a region including the body part of the passerby from the first image, and a body part detection unit.
Further provided with an image cutting portion for extracting at least a partial region including the head and shoulder regions of the person from the first image among the detected regions including the body parts of the passerby.
The estimation system according to claim 2, wherein the depth image cutting unit extracts a partial region including at least the body portion of the person from the detected region including the body portion of the passerby from the second image. Further provided with a learning unit that generates the trained model by inputting teacher data in which the information of the person area and the three-dimensional data corresponding to the person area and the information of the gaze start position and the gaze direction are associated with each other. The estimation system according to any one of claims 1 to 3. One or more first images in which an area including a passerby's body part is captured, and one or more second images in which information on the distance in the depth direction with respect to the area including the passerby's body part is stored. And the acquisition department to acquire,
The acquisition of the trained model trained to output the gaze start position and gaze direction information of the person indicated in the person area by inputting the information of the person area and the three-dimensional data corresponding to the person area. Gaze of the passerby by inputting the information of the person area extracted from the first image acquired by the unit and the point group data or the depth image which is the three-dimensional data generated from the second image. An estimation unit that estimates the start position and gaze direction,
Estimator equipped with. One or more first images in which an area including a passerby's body part is captured, and one or more second images in which information on the distance in the depth direction with respect to the area including the passerby's body part is stored. And get the get steps and
The acquisition of the trained model trained to output the gaze start position and gaze direction information of the person indicated in the person area by inputting the information of the person area and the three-dimensional data corresponding to the person area. Gaze of the passerby by inputting the information of the person area extracted from the first image acquired by the unit and the point group data or the depth image which is the three-dimensional data generated from the second image. An estimation step for estimating the start position and gaze direction, and
Estimating method. One or more first images in which an area including a passerby's body part is captured, and one or more second images in which information on the distance in the depth direction with respect to the area including the passerby's body part is stored. And get the get steps and
The acquisition of the trained model trained to output the gaze start position and gaze direction information of the person indicated in the person area by inputting the information of the person area and the three-dimensional data corresponding to the person area. Gaze of the passerby by inputting the information of the person area extracted from the first image acquired by the unit and the point group data or the depth image which is the three-dimensional data generated from the second image. An estimation step for estimating the start position and gaze direction, and
A computer program that lets your computer run.

Images