JP7258426B2 - Simulation system, simulation program and learning device - Google Patents

Description

The present invention relates to a simulation system, and more particularly to a simulation system, a simulation program and a learning device for simulating a robot and a human agent in an environment where a robot such as a communication robot and a human coexist.

Conventional simulation devices of this type are disclosed in Non-Patent Documents 1 and 2. With the simulation devices of these non-patent documents 1 and 2, by simulating and visualizing the sensors and actuators, it is possible to more easily simulate the adaptation of the robot in the coexistence environment with humans.

As a calibration technique for a sensor network installed in an environment, Patent Documents 1 and 2 are known, which specify the positional relationship between sensors by moving people and objects in the environment and recording them at the same time. .

One way to improve the accuracy of human orientation and gesture (movement) recognition in sensor networks is to collect a large amount of behavioral data from many people with different genders, heights, body shapes, etc., and adjust the parameters of the recognition algorithm. It is to implement. However, the placement of sensors in the sensor network is fixed in the actual field environment, and there are many variations depending on the installation situation. , incurs a great deal of cost.

On the other hand, non-patent document 3, non-patent document 4, etc. have known an approach in which a large amount of data is generated by simulation and used for learning.

JP 2017-96813 (G01S 7/497 G01S 17/42) Patent No. 6447706 (G01B 11/00 G06T 7/70 G06T 7/80)

https://www.openrobots.org/morse/doc/stable/morse.html http://gazebosim.org/ David Silver et al., Mastering the game of Go with deep neural networks and tree branch, Nature, Vol.529, 484-503, 2016 Mnih V et al., Human-level control through deep reinforcement learning, Nature, Vol.518, No.7540, 529-533, 2015

Approaches such as Non-Patent Document 3 and Non-Patent Document 4 are aimed at objects that can be easily reproduced in a virtual world, such as Go (Non-Patent Document 3) and computer games (Non-Patent Document 4), and are intended to simulate human behavior. It is difficult to apply to the real-world sensing system of recognition.

SUMMARY OF THE INVENTION Therefore, a primary object of the present invention is to provide a novel simulation system.

Another object of the present invention is to provide a simulation system capable of generating human data that can be used in real-world sensing systems.

In order to solve the above problems, the present invention employs the following configurations. It should be noted that reference numerals in parentheses, supplementary explanations, etc. indicate the correspondence with the embodiments described later in order to facilitate understanding of the present invention, and do not limit the present invention in any way.

A first invention is a human agent generation means for introducing a human agent into a virtual simulation space simulating the real world in which a plurality of three-dimensional range image sensors are arranged at respective predetermined positions, and updating the human agent in the virtual simulation space. an updating means, a 3D model database having a plurality of 3D models indicative of a three-dimensional shape of the human agent, and an objective model database having objectives of the human agent, wherein the human agent generating means applies a selected 3D model from the 3D model database. generating a human agent having a shape according to the objective, the updating means moving the human agent according to the objective, and further including at least the position, orientation and gesture of each human agent as seen from each of the plurality of three-dimensional range image sensors. true value label creation means for creating a true value label; sensor data acquisition means for acquiring, as sensor data , a range image of each human agent by a plurality of three-dimensional range image sensors when the human agent is moved by the update means; A simulation system comprising output means for outputting a data set combining value labels and sensor data.

In the first invention, a simulation system (10: reference numerals illustrating corresponding parts in the embodiments; the same shall apply hereinafter) includes human agent generation means (26, 110), and this human agent generation means is a 3D model A human agent is generated according to a 3D model selected from a plurality of 3D models set in a database (111) and put into a virtual agent simulation space (22). An updating means (26, 114) updates the position of the human agent according to the selected objectives from the objective model database (116). A sensor data acquiring means (S33) acquires sensor data obtained by simulating a plurality of three-dimensional range image sensors when the human agent is moved by the updating means. A true value label creation means (S31) creates a true value label of an agent (position, orientation, gesture , etc. of a human agent), and an output means (S35) outputs a data set combining the true value label and sensor data. do. By inputting this data set into the learning machine, it becomes possible to calibrate the 3D range image sensor.

According to the first invention, the simulation system can efficiently output a large amount of data sets.

A second invention is according to the first invention, and is the simulation system, wherein the human agent generating means generates different types of human agents each having a shape according to a 3D model.

In the second invention, human agent generation means (110) generates human agents of different types, such as adult, child, female, and male, according to 3D models.

According to the second invention, since various types of human agents can be introduced into the virtual simulation space, various data sets can be output.

A third invention is according to the first or second invention, further comprising a gesture database setting different gestures for the human agent, wherein the updating means causes the human agent to make a gesture selected from the gesture database. It is a simulation system.

In the third invention, the gesture database (113) has a plurality of gestures, and the updating means causes the human agent to select gestures from the gesture database.

According to the third invention, since the human agent is made to make a gesture, it is possible to output various data sets close to the real world.

A fourth invention is dependent on any one of the first to third inventions and further comprises a social force model database (118) setting a social force model, wherein the updating means is configured to update the human agent according to the social force model. It is a simulation system that updates the position.

According to the fourth invention, since the human agent is moved according to the social force model, it is possible to output various data sets that are closer to the real world.

A fifth invention is a simulation system for executing a simulation in a virtual simulation space simulating the real world in which a plurality of three-dimensional range image sensors are arranged at respective predetermined positions , wherein the simulation system comprises a three-dimensional shape of a human agent. a 3D model database having a plurality of 3D models and an objective model database having human agent objectives, wherein the computer of the simulation system is placed in a virtual simulation space to create a shape according to a 3D model selected from the 3D model database. human agent generation means for inputting a human agent, updating means for updating the human agent by moving the human agent according to the objective in the virtual simulation space, at least the position of each human agent viewed from each of a plurality of three-dimensional range image sensors, When the human agent is moved by the true value label creating means and the updating means for creating the true value label of the human agent , including orientation and gestures, a range image of each human agent is acquired as sensor data by a plurality of three-dimensional range image sensors. It is a simulation program that functions as sensor data acquisition means for acquiring data, and output means for outputting a data set that combines true value labels and sensor data.

According to the fifth invention, the same effects as those of the first invention can be expected.

A sixth aspect of the invention is a sensor calibration or recognition model learning device comprising a learning device for learning the data set output by the output means of any one of the first to fifth aspects of the invention.

According to the sixth aspect, since a large amount of data sets can be efficiently obtained from the output means, sensor calibration and recognition model learning can be efficiently executed.

According to this invention, a large amount of various sensor data can be obtained by reproducing various people (pedestrians) in the virtual simulation space. Therefore, by inputting a large amount of sensor data into a learning device, sensor calibration and recognition model learning can be performed efficiently.

The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

FIG. 1 is a functional block diagram showing a simulation system of one embodiment of the invention. FIG. 2 is an illustrative view showing one example of a robot to which the present invention is applied. FIG. 3 is a block diagram showing the electrical configuration of the robot shown in FIG. 2. As shown in FIG. 4 is an illustrative view showing one example of the virtual simulation space of FIG. 1. FIG. FIG. 5 is a functional block diagram showing details of the human simulator of FIG. FIG. 6 is an illustrative view showing one example of a basic objective model of a human agent. FIG. 7 is an illustrative view showing another example of the basic objective model of the human agent. FIG. 8 is an illustrative view showing another example of the human agent objective model. 9 is an illustrative view showing an example of movement of a human agent in a part of the virtual simulation space of FIG. 4; FIG. FIG. 10 is a flow diagram showing an example of basic simulation operation in the embodiment of FIG. FIG. 11 is a flowchart showing in detail an example of a pedestrian (person) simulation operation in the FIG. 1 embodiment. FIG. 12 is a flow chart showing an example of the operation of creating a pedestrian in the embodiment of FIG. 11. FIG. FIG. 13 is a flow chart showing an example of the operation of updating pedestrians in the FIG. 11 embodiment. FIG. 14 shows an example of sensing data in FIG. 11 embodiment, FIG. 14(A) shows sensing data (distance image) of each sensor AJ, and FIG. 14(B) shows sensing data integrating them. . FIG. 15 is an illustrative view showing an arrangement example of a three-dimensional range image sensor in a real world environment corresponding to the virtual simulation space of the embodiment.

Referring to FIG. 1, a simulation system 10 of this embodiment comprises a robot 12, an operating device 14 for remotely controlling the robot 12, a robot agent simulating the robot 12, and a human agent simulating a person (pedestrian). A simulation device 16 for simulating is included. The simulation system 10 is implemented by one or more computers (processors), except for the robot 12 .

The robot 12 of this embodiment is a robot that works in an environment 200 coexisting with humans as shown in FIG. 15, like the communication robot shown in FIG. 2, for example. Since such a robot 12 is affected by the behavior of a person existing in the environment, in the simulation device 16 of this embodiment, when simulating the behavior of the robot 12 according to a robot application program (robot behavior determination program), By also simulating human behavior, the development of safe robot application programs for the robot 12 is aided.

In an environment (real world) 200 where the robot 12 is active and where various people come and go, such as a shopping mall, as shown in FIG. are placed. In this example, each sensor 202 is assumed to be a three-dimensional range image sensor.

Here, with reference to FIGS. 2 and 3, the configuration of the robot 12 will be described to the extent necessary for understanding the present invention. The robot 12 includes a carriage 30 with two wheels 32 and one follower wheel 34 provided on the underside of the carriage 30 for moving the robot 12 . The two wheels 32 are independently driven by wheel motors 36 (see FIG. 3) to move the carriage 30, that is, the robot 12, in any direction, forward, backward, leftward, or rightward.

A cylindrical sensor mounting panel 38 is provided on the carriage 30, and a large number of distance sensors 40 are mounted on the sensor mounting panel 38. As shown in FIG. These distance sensors 40 measure the distance to objects (people, obstacles, etc.) around the robot 12 using, for example, infrared rays or ultrasonic waves.

A body 42 is provided upright on the sensor mounting panel 38 . Further, the above-described distance sensor 40 is further provided in the front central upper part of the body 42 (position corresponding to the chest of the person), and measures the distance in front of the robot 12, mainly to the person. Further, the body 42 is provided with a support 44 extending from substantially the center of the side upper end portion thereof, and an omnidirectional camera 46 is provided above the support 44 . The omnidirectional camera 46 captures the surroundings of the robot 12 and is distinguished from the eye camera 70 described later. As the omnidirectional camera 46, for example, a camera using a solid-state imaging device such as CCD or CMOS can be adopted.

An upper arm 50R and an upper arm 50L are provided on both side upper ends (positions corresponding to human shoulders) of the torso 42 by shoulder joints 48R and 48L, respectively. Although not shown, the shoulder joint 48R and the shoulder joint 48L each have degrees of freedom of three orthogonal axes. That is, the shoulder joint 48R can control the angle of the upper arm 50R around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 48R is an axis parallel to the longitudinal direction (or axis) of the upper arm 50R, and the other two axes (pitch axis and roll axis) are perpendicular to that axis from different directions. It is the axis that Similarly, the shoulder joint 48L can control the angle of the upper arm 50L around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 48L is an axis parallel to the longitudinal direction (or axis) of the upper arm 50L, and the other two axes (pitch axis and roll axis) are perpendicular to that axis from different directions. It is the axis that

Further, elbow joints 52R and 52L are provided at the distal ends of the upper arms 50R and 50L, respectively. Although not shown, the elbow joint 52R and the elbow joint 52L each have a degree of freedom of one axis, and can control the angles of the forearms 54R and 54L around this axis (pitch axis).

Hands 56R and 56L, which correspond to human hands, are provided at the tips of the forearms 54R and 54L, respectively. Although detailed illustration is omitted, these hands 56R and 56L are configured to be openable and closable, whereby the robot 12 can grip or pinch an object using the hands 56R and 56L. However, the shape of the hands 56R and 56L is not limited to the shape of the embodiment, and may be made to have a shape and functions very similar to those of human hands.

In addition, although not shown, the front surface of the carriage 30, the portion corresponding to the shoulder including the shoulder joint 48R and the shoulder joint 48L, the upper arm 50R, the upper arm 50L, the forearm 54R, the forearm 54L, the hand 56R and the hand 56L are provided with , a contact sensor 58 (generically shown in FIG. 3) is provided. A contact sensor 58 on the front of the truck 30 detects contact of a person or other obstacle with the truck 30 . Therefore, if the robot 12 comes into contact with an obstacle during its own movement, it can detect it and immediately stop driving the wheels 32 to abruptly stop the movement of the robot 12 . Further, other contact sensors 58 detect whether or not each part is touched.

A neck joint 60 is provided in the upper central portion of the body 42 (a position corresponding to a human neck), and a head 62 is provided thereon. Although not shown, the neck joint 60 has three degrees of freedom and can be angularly controlled around each of the three axes. One axis (yaw axis) is an axis directed directly above the robot 12 (vertically upward), and the other two axes (pitch axis and roll axis) are orthogonal axes in different directions.

A speaker 64 is provided on the head 62 at a position corresponding to a person's mouth. Speakers 64 are used by robot 12 to communicate audibly to people in its vicinity. A microphone 66R and a microphone 66L are provided at positions corresponding to human ears. Hereinafter, the right microphone 66R and the left microphone 66L may be collectively referred to as the microphone 66. FIG. A microphone 66 captures ambient sounds, especially the human voice with which the communication is being carried out.

Further, a right eyeball portion 68R and a left eyeball portion 68L are provided at positions corresponding to human eyes. Right eyeball portion 68R and left eyeball portion 68L include right eye camera 70R and left eye camera 70L, respectively. Hereinafter, the right eyeball portion 68R and the left eyeball portion 68L may be collectively referred to as an eyeball portion 68. As shown in FIG. Also, the right eye camera 70R and the left eye camera 70L may be collectively referred to as an eye camera 70. FIG.

The eye camera 70 photographs the face of a person approaching the robot 12, other parts or objects, and captures the corresponding video signal. In this embodiment, the robot 12 detects the line-of-sight directions (vectors) of each of the left and right eyes of a person from the video signal from the eye camera 70 .

Also, the eye camera 70 can be the same camera as the omnidirectional camera 46 described above. For example, the eye camera 70 is fixed within the eyeball portion 68, and the eyeball portion 68 is attached at a predetermined position within the head 62 via an eyeball support (not shown). Although illustration is omitted, the eyeball supporting portion has two degrees of freedom and can be angularly controlled around each of these axes. For example, one of the two axes is the upward direction axis (yaw axis) of the head 62, and the other is orthogonal to one axis and orthogonal to the direction in which the front side (face) of the head 62 is directed. is the axis (pitch axis) in the direction to By rotating the eyeball supporting portion around each of these two axes, the eyeball portion 68 or the tip (front) side of the eye camera 70 is displaced, and the camera axis, that is, the line of sight direction is moved. The installation positions of the speaker 64, the microphone 66, and the eye camera 70 described above are not limited to these parts, and may be provided at appropriate positions.

As described above, the robot 12 of this embodiment has independent two-axis drive of the wheels 32, three degrees of freedom of the shoulder joints 48 (six degrees of freedom on the left and right), one degree of freedom on the elbow joints 52 (two degrees of freedom on the left and right), It has a total of 17 degrees of freedom, including 3 degrees of freedom of the neck joint 60 and 2 degrees of freedom of the eyeball supporting portion (4 degrees of freedom in left and right).

FIG. 3 is a block diagram showing the electrical configuration of the robot 12. As shown in FIG. Referring to FIG. 3, robot 12 includes one or more processors 80 . Processor 80 is connected via bus 82 to memory 84 , motor control board 86 , sensor input/output board 88 and audio input/output board 90 .

Although not shown, the memory 84 includes ROM, HDD and RAM. Various programs are stored in advance in the ROM and HDD.

The motor control board 86 is composed of, for example, a DSP, and controls the driving of each shaft motor of each arm, neck joint 60, eyeball portion 68, and the like. That is, the motor control board 86 receives control data from the processor 80, and controls two motors (in FIG. 3, collectively referred to as "right eyeball motor 92") for controlling the respective angles of the two axes of the right eyeball portion 68R. to control the rotation angle of the Similarly, the motor control board 86 receives control data from the processor 80 and controls two motors (collectively referred to as "left eye motor 94" in FIG. 3) for controlling the respective angles of the two axes of the left eyeball 68L. (shown) to control the angle of rotation.

Also, the motor control board 86 receives control data from the processor 80, and controls three motors for controlling the angles of the three orthogonal axes of the shoulder joint 48R and one motor for controlling the angle of the elbow joint 52R. It controls the rotation angles of four motors (shown collectively as "right arm motor 96" in FIG. 3). Similarly, the motor control board 86 receives control data from the processor 80, three motors controlling the angles of the three orthogonal axes of the shoulder joint 48L and one motor controlling the angle of the elbow joint 52L. (collectively referred to as "left arm motor 98" in FIG. 3).

Furthermore, the motor control board 86 receives control data from the processor 80 and controls three motors (collectively referred to as "head motors 100" in FIG. ) to control the rotation angle. The motor control board 86 receives control data from the processor 80 and controls the rotation angles of the two motors that drive the wheels 32 (collectively referred to as "wheel motors 36" in FIG. 3).

A hand actuator 101 is further coupled to the motor control board 86, and the motor control board 86 receives control data from the processor 80 and controls opening and closing of the hands 56R and 56L.

The sensor input/output board 88, like the motor control board 86, is composed of a DSP, takes in signals from each sensor, and supplies them to the processor 80. FIG. That is, data relating to the reflection time from each of the range sensors 40 are input to the processor 80 through this sensor input/output board 88 . Also, the video signal from the omnidirectional camera 46 is input to the processor 80 after undergoing predetermined processing by the sensor input/output board 88 as necessary. A video signal from the eye camera 70 is similarly input to the processor 80 . Signals from the plurality of contact sensors 58 (collectively referred to as “contact sensors 58 ” in FIG. 3 ) are provided to processor 80 via sensor input/output board 88 . The voice input/output board 90 is also composed of a DSP in the same way, and the speaker 64 outputs voice according to voice synthesis data given from the processor 80 . Audio input from microphone 66 is also provided to processor 80 via audio input/output board 90 .

Processor 80 is also connected to communication LAN board 102 via bus 82 . Communication LAN board 102 is composed of a DSP, for example, and provides transmission data given from processor 80 to wireless communication module 104, and wireless communication module 104 transmits the transmission data to a server (not shown) or the like via a network. do. Communication LAN board 102 also receives data via wireless communication module 104 and provides the received data to processor 80 .

The robot to which the simulation system 10 of the embodiment can be applied is not limited to the robot 12 shown in FIGS. 2 and 3, and this simulation apparatus can also be applied to communication robots of other types and structures. .

Returning to FIG. 1, the operation device 14 is basically composed of one or more computers or processors, and a robot application program 18 is set in the memory (not shown) of the processor. A program is given to the robot 12 or simulation device 16 from a computer, for example wirelessly. The robot application program 18 is an application program for the robot 12. For example, when the robot 12 is operated as a guide robot, the robot 12 executes operations for that purpose.

The memory of the operation device 14 further stores environmental data 20 in advance. The environment data 20 mainly includes map data of the virtual simulation space 22 shown in FIG. 4 and appearance rate data of human agents within the virtual simulation space 22 . The environment and objects within the environment are represented as 3D model data and visualized as a 3D virtual space. This three-dimensional virtual simulation space 22 reproduces the real world 200 described above.

The environment data 20 further includes the position, orientation, scanning range (sensing range), etc. of each sensor 202, such as a three-dimensional range image sensor, set in the real world 200 (FIG. 15).

A virtual simulation space 22 shown in FIG. 4 is assumed to be a three-dimensional closed space having an entrance 22a indicated by a dotted-line rectangle in the figure. In this virtual simulation space 22, a passage 22b indicated by a thick black line is set, and the robot 12 and the human agent pass through or exist in the passage 50, the square, or the like. These passages 22b and plazas are partitioned by objects such as buildings and walls, and diverge or merge at places indicated by circles in the drawing.

The simulation device 16 is basically composed of one or more processors and includes a robot simulator 24 that simulates the robot 12 and a human simulator 26 that simulates a human agent such as a pedestrian. is input to the physics engine 28 .

The robot simulator 24 is, for example, the MORSE simulator exemplified in Non-Patent Document 1 above, and this MORSE simulator provides many sensors, actuators and robot models similar to the robot 12, and a model for such a model. An API (Application Programming Interface) is also available.

For example, in the sensor simulation executed in step S33 of FIG. 11, for example, in the case of a range image sensor, optical calculation from the viewpoint of the camera is performed to generate image information for each viewpoint of the camera. In the case of a laser range finder, the distance is measured from the sensor in each direction by simulating laser scanning, and the results are stored as sensor values.

When developers prepare robot application programs for their robots, those programs access these APIs to read sensor data (e.g. reading distances from a laser rangefinder) and to send commands to actuators (e.g. movement speed).

The people simulator 26 periodically creates new pedestrians, updates their positions, and removes them when they leave the simulated environment (virtual simulation space 22). A pedestrian or person is represented as a three-dimensional object, placed in the virtual simulator space 22, and its walking motion is animated using an animation engine.

However, since the human simulator 26 is provided with environmental data 20 as shown in FIG. 1, the human simulator 26 periodically generates new pedestrians with a predefined probability called the occurrence rate contained in the environmental data. do. The pedestrian is then assigned to one of the doorways 22a shown in FIG. 4 and set to go to the other one.

Note that pedestrians are often generated in groups (like families or couples) and when they are displayed in groups, the size is defined by the distribution of group members, assuming that all members share the same purpose.

The physics engine 28 calculates what kind of movement path the robot or human agent will follow based on physical forces and laws. More specifically, the physics engine 28 performs processing that causes each agent (including robots) from the robot simulator 24 and the human simulator 26 to perform intended movements in the virtual simulation space 22 (FIG. 4). At this time, if there is no interference with other obstacles, structures, or other agents, it will generate the intended movement. However, in the presence of interference, the actual movement of each agent is processed according to the laws of physics (laws of mechanics). Then, the physics engine 28 outputs integrated simulation data.

The simulation device 16 is provided with a display 105, and the display 105 displays an image of a virtual simulation space, such as that shown in FIG. 4, together with simulated robots and human agents present therein. . For example, a human agent is represented as a three-dimensional object, placed in the virtual simulation space 22, and its walking motion is animated using an animation engine. A developer of the robot application program 18 creates and modifies an action determination program (application program) for the robot 12 while checking the image displayed on the display 105 .

Furthermore, the simulation device 16 is provided with an input device 107 . This input device 107 may be, for example, a keyboard or a touch panel. In the case of a touch panel, it may be incorporated in the display 105 and the display 105 may be configured as a touch display.

The person simulator 26 will be described in detail with reference to FIG. Human simulator 26 , like robot simulator 24 , is a function implemented by the computer or processor of simulation device 16 and includes human generation module 110 . This human generation module 110 creates new pedestrians (human agents) according to the occurrence rate data 108 contained in the environment data 20, as described above, in the virtual simulation space 22 represented by the map data 106, an example of which is shown in FIG. , is generated periodically.

Appearance rate data 108 is a database in which types (patterns) of people, such as adults and children, and the rate of appearance of each person are set in advance for human agents to be introduced into the virtual simulation space 22 of the simulation device 16. A human agent with a set shape is generated with a set probability.

A three-dimensional (3D) model database 111 is attached to the human generation module 110 . The 3D model database 111 holds 3D models of a plurality of different shapes, such as gender, height, build, etc., for each of the types of persons mentioned above. Therefore, the human generation module 110 generates in the virtual simulation space 22 a human agent of the type indicated by the appearance rate data 108 and according to the 3D model of the type randomly selected from the 3D model database 111 .

It should be noted that the appearance rate data 108 may hold different data sets for each day of the week and time so that the simulation can be performed by designating the day of the week and time. By doing so, various simulations can be performed, such as what happens when the robot is moved at a certain time.

However, if the appearance rate data 108 indicates that the human agents to be generated form a group, the human generation module 110 generates a plurality of human agents in a group having a member configuration according to the group member data 112 set as a simulation parameter. to generate human agents. Again, their body shape follows the 3D model selected from the 3D model database 111 .

The movement module 114 moves the human agent generated by the human generation module 110 within the virtual simulation space 22 ( FIG. 4 ) based on objectives preset in the objective model database 116 . Here, the objective is the way a human agent (not shown) reacts when a robot agent (not shown) enters the field of view of the human agent (not shown) in the virtual simulation space 22, that is, the human agent reacts. It is an instruction set that defines possible action patterns against robots in real space, and possible action patterns in the environment (even when robots are not present) (for example, stopping in front of a product shelf at a convenience store, etc.). For example, when a person walks, the instruction set includes a movement target and a movement speed as parameters.

In this embodiment, four typical reaction modes (objectives) described later are set as models, and the movement module 114 moves the human agent according to the objectives indicated by the models. In other words, the objective 116 functions as a first instruction set that causes the human agent to react to the presence of the robot in the virtual simulation space. These objectives are stored in a first memory (not shown) of simulation system 10 .

The available objectives preset in the objective model 116 can be read out from the first memory and displayed as a list on the display 105, for example, so that the developer who intends to debug can know them. can be done. The developer can select any objective in it and change (update) that objective in the script (122) as described below. At this time, the developer can use an input device (not shown) such as a keyboard provided in the simulation system 10 to change the portion to be changed. The modified (updated) objective is then also stored in the first memory. In other words, since the objective model 116 is designed to have extensibility, it is effective to be able to know the objectives updated by someone else as list information.

For example, behavior patterns (objectives) that a person takes when a robot enters his/her field of view include a pattern of interacting with the robot and a pattern of observing the robot, as shown in FIGS.

FIG. 6 illustrates an objective that stops around robot 12 a stopping distance Dstop from robot 12 and interacts with robot 12 when a person (indicated by "i") discovers robot 12 within his/her field of view. are doing.

FIG. 7 shows that when a person (indicated by "i") discovers the robot 12 in his field of view, he stops around the robot 12 with an observation distance Dobserve slightly larger than the stopping distance Dstop from the robot 12, and the robot 12 illustrates an objective observing 12;

Objectives include other objectives that pass by the robot, and objectives that slow down and pass by the robot while looking at it. In the movement module 114, the human agent moves according to any registered objective. simulate as

However, the locomotion module 114 simulates locomotion of the human agent taking into account the respective model data from the Social Force Model database 118 and the Interaction with robot model database 120 included in the simulation parameters. to The social force model is a model that expresses how pedestrians affect each other and behavior changes, as shown in FIG. 8, which will be described later. For example, it is known as repulsion or reaction force.

In the virtual simulation space 22, the behavior of the human agent changes under the influence of other agents (for example, people avoid each other, approach robots when they see them, etc.).

For example, as shown in FIG. 8, two people i and j may influence each other to determine their movement position in the next step, since human agent i originally moves in the direction of di,j. However, since the human agent j is about to move in the direction of the angle θi,j with respect to the human agent i at the speed vi,j, the human agent i ends up moving in the direction of d'i,j. .

The interaction model 120 is a model in which patterns of human interaction with the robot are set in advance.

FIG. 9 schematically shows a part of the virtual simulation environment (space) shown in FIG. there is However, all objects 124 are written in the map data 106 as obstacles.

The robot simulator 24 simulates the action of the robot in the next step according to the robot application program 18 based on data such as the position of the robot (robot agent) included in the environment data 20 . Here, the robot application program 18 is a program that controls each motor, that is, actuators, and determines the action of the robot according to the sensor output input to the sensor input/output board 88 of FIG.

In the example of FIG. 9, the robot application program 18 is set so that the robot 12 enters the virtual simulation space 22 through one doorway 22a and moves along the path 22b until it exits through the other doorway 22a. It is Incidentally, reference numeral 126 in FIG. 9 indicates a human agent put into this virtual simulation space 22 for simulation.

Human simulator 26 also performs simulations according to scripts 122 rather than objectives 116 when necessary. The objective 116 is a preset, fixed response pattern to the robot, as described above, while the script 122 is an instruction that causes the simulation device 16, ie, the human simulator 26, to perform a non-fixed action. Here, a script is a simple program (instruction set) that can be easily interpreted and executed without conversion into a computer-readable language. When giving instructions to a computer in software design, etc., it is usually necessary to convert a language that humans can normally read and write into a language that computers can understand, so-called machine language. It is a simple program that can be easily interpreted and executed by omitting the conversion work. Although these scripts fall under the second set of instructions, they are stored in a second memory (not shown) of simulation system 10 . However, the first memory and the second memory may be different memory locations in the same memory.

A gesture model database 113 is also attached to the human simulator 26 . A plurality of types of gestures (movements) that a person (walker) can take are registered in this gesture model database 113 . The gestures include, for example, looking at an object, walking while looking at an object, picking up an object (right and/or left hand), walking (normally or slowly, fast), stopping, crouching, and the like. Gesture data may include data generated using a motion capture system in order to generate data similar to human gestures occurring in the real world.

Therefore, the movement module 114 of the human simulator 26 causes the human agent performing the gesture randomly selected from the gesture model database 113 to move in a movement pattern or movement path according to the objective model 116 or the script 122 .

Here, for the purpose of facilitating understanding of the embodiment of the present invention shown in FIG. 11, first, referring to FIG. 10, an example of operation in the simulation device 16 shown in FIG. 1 according to the objective model 116 will be described. . However, the simulation shown in FIG. 10 executes a plurality of steps (S3-S17) as one step. In practical operation, one step of the simulation can be executed in units of 0.1 milliseconds at the fastest, but here it is set to units of 10 milliseconds, and execution of the robot simulation program 18 can be executed in units of 100 milliseconds. It is possible. In this example, the robot application program 18 is executed once every 10 steps executed by the simulation device 16 . In other words, in this embodiment, it is assumed that the period of the robot application program is longer than the execution period of one step of the simulation.

In the first step S1, the robot application program 18, which is set to move on the path 22b shown in FIG. 4 (and FIG. 9) as an example, is read, and the environmental data 20 is read. That is, parameters (appearance frequency , movement route, number of people in the group, adult/child/male/female ratio, height, etc.) and location data of each agent (including the robot 12). This virtual simulation space 22 is displayed on the display 105 together with the robot 12 and the human agent.

Furthermore, in this step S1, the simulation is initialized. That is, the execution speed is set to s=1.0, and the number of basic execution steps fps is set to fps=30.0. However, fps is a numerical value indicating how many steps of simulation are executed per second, and in the initial setting of this embodiment, it is set so that 30 steps of simulation are executed per second.

In the next step S3, the human generation module 110 of the human simulator 26 generates 1 or 2 human agents in the virtual simulation space 22 according to the above-mentioned parameters for human agent generation presented by the environment data 20 and the human pattern data from the group. Generate and place more human agents.

In step S5, the human simulator 26 determines the human agent's next step action based on the current positions of the human agent and the robot 12 indicated by the environment data 20 and on the behavior pattern with respect to the robot indicated by the objective model 116. decide. A list of the four basic objectives described above is provided in a form that can be freely accessed by the developer (operator of the operation device 14). Additionally, other objective models can be employed.

In step S7, the robot simulator 24 determines the behavior of the robot 12 in the next step according to the robot application program 18 based on the current human agent indicated by the environmental data 20 and the position of the robot 12. FIG.

Specifically, in step S7, if the robot 12 is a robot whose main purpose is to go to a destination, such as a carrier robot or a boarding robot, the other people's current positions and movement speeds are used as much as possible. processing to calculate a movement route that does not interfere with the movement of In addition, when the robot 12 is a robot whose purpose is to provide services to people, for example, in order to approach a person who is likely to be interested in the robot, in a simple case, a person standing near the robot choose a place where people tend to gather and wait for the person who comes, and do not approach a place where people pass excessively so as not to cause congestion. We also take steps to prevent it from happening.

In the case of this step S7, the action of the human agent in the next step in step S5 has been determined, so the above processing is executed in consideration of that. Therefore, it is possible to judge the suitability of a robot application program that operates in a real space coexisting with people in the virtual simulation space.

Then, in step S9, the physics engine 28 implements the actions of each agent. More specifically, in step S9, as described above, each agent (including robots) performs the intended movement within the virtual simulation space 22, and other obstacles, structures, and other agents Each agent behaves as intended if there is no interference with For example, if the agent tries to take one step forward, the agent takes one step forward as intended.

However, in the presence of interference, the actual movement of each agent is processed according to the laws of physics (laws of mechanics). In a simple case, for example, if you try to move towards a wall, you will hit the wall, you will not be able to move further, and you will receive a reaction force according to a certain coefficient of restitution. In the case of a collision between agents, the forward movement and the movement due to the reaction force overlap, and for example, if we consider a simple example, if the agent collides from the side while moving forward, it will actually move diagonally forward. .

In the next step S11, environmental data, sensor data, etc. are updated.

Then, in step S13, it is determined whether or not to end the simulation, and if the simulation is to be continued, the process returns to the previous step S3.

In this way, the simulator 16 performs a simulation to reproduce the behavior of people around the robot according to the objective 116, and by confirming the state on the display 105 (FIG. 1), the behavior of the robot can be simulated. It is possible to efficiently develop a robot application program 18 that works in a real space coexisting with people without actually moving the robot in the real space.

FIG. 11 is a flow diagram primarily showing the operation of the human simulator 26 in this embodiment, and in the first step S21 of this FIG. 11, the computer determines whether the simulation is running. If "NO", the process ends as it is.

The features of this embodiment are that the human agent is generated by the human simulator 26 in step S3 of FIG. It is to acquire simulation data corresponding to the three-dimensional range image sensor 202 (FIG. 15) during action. This will be explained below.

When "YES" is determined in step S21, the computer determines in the next step S23 whether or not the pedestrian (person) generation condition is established. This pedestrian is generated with a certain probability according to the appearance rate data 108 (FIG. 5) when the number of pedestrians existing in the virtual simulation space 22 is less than a predetermined maximum number. Therefore, the pedestrian generation condition is a condition that the appearance rate is reached.

If "YES" is determined in step S23, a pedestrian is generated by the human generation module 110 in step S25. Specifically, pedestrians are generated according to the flowchart shown in FIG.

In step S41 of FIG. 12, the computer, that is, the human generation module 110 selects a pedestrian model from the 3D model database 111 at random or according to a predetermined rule.

Then, in step S43, an objective (behavior type) is selected from the objective model 116 at random or according to a predetermined rule, and simulation parameters are deterministically or predetermined according to the group member data 112, the social force model 118, and the interaction model 120. determined according to the law of In this step S43, a plurality of objectives can be selected, in which case the objectives are executed in the order of selection.

In step S45, for each objective, a gesture to be executed is selected from the gesture database 113 at random or according to a predetermined rule, and set to the objective. Gestures include, as described above, "take an object". Multiple gestures may be selected for each objective. Then return.

Thus, in step S25 of FIG. 11, a human agent (pedestrian) can be generated in the virtual simulation space 22 (FIG. 4). Therefore, step S25 and the computer that executes it can be called human agent generating means.

At step S27 of FIG. 11, a loop is executed for each pedestrian Pi, and at step S29 the computer or movement module 114 updates the pedestrian Pi. Specifically, the flowchart shown in FIG. 13 is followed.

In step S51 of FIG. 13, the computer determines whether the objective has ended. If "YES", delete (delete) the pedestrian Pi from the environment, that is, the virtual simulation space 22, and return. In other words, I set a movement sequence for each pedestrian, so that they disappear from the environment when it's done.

On the other hand, when "NO" is determined in step S51, the position of the pedestrian is updated in step S55 according to the social force model previously set in step S43. Then, in step S57, the gesture of the pedestrian Pi is reproduced according to the gesture set in step S45, and then the process returns.

In the updating process in step S27, if there are a plurality of objectives, the updating process for the next objective is sequentially executed. The update in step S27, that is, step S29, updates the position of the human agent by applying the movement sequence of the human agent in the virtual simulation space. be able to.

When the updating process for each pedestrian Pi is finished, in step S31 of FIG. 11, a true value label for each pedestrian is created. A true value label is how each pedestrian (person) is seen from each sensor, and includes, for example, the position of the pedestrian, the orientation of the pedestrian, gestures, and the like. Since the movement sequence of the pedestrian is simulated by the human simulator 26, the computer can easily acquire such true value labels from each model and parameter.

Next, in step S33, the computer creates sensor data. The sensor data is data obtained by simulating how the pedestrian Pi looks from each sensor when the pedestrian Pi is simulated as described above, such as a distance image. When a three-dimensional range image sensor is used as the sensor 202 (FIG. 15), as shown in FIG. 14A, respective range images of sensors AJ are created. By integrating the distance images for each of the sensors A to J in FIG. 14A, a distance image as shown in FIG. 14B is obtained. Since sensor noise exists in sensing in the real world, a sensor noise model for reproducing such sensor noise is applied when creating range image data. That is, the distance image is generated by adding (or subtracting) probabilistic sensor noise (such as Gaussian distribution) to the distance measured in the simulation. Such sensor noise models are preset, for example, in environmental data 20 (FIG. 1), together with data from three-dimensional range image sensors placed in the real world.

In the next step S35, the computer outputs a data set obtained by combining the true value label data acquired in step S31 and the distance image as shown in FIG. 14 created in step S33.

By inputting this data set into a learning device (not shown), sensor calibration and recognition model learning can be performed. However, sensor calibration means determining the calculation method of sensor data indicating what kind of calculation should be performed for each sensor 202 to improve the sensing accuracy. A large number of data sets are required, for example over a million.

Acquiring an extremely large number of data sets in the real world involves collecting a large amount of data that behaves in various places (environments) and various patterns, as well as their label sets, which incurs a great deal of cost.

On the other hand, if pairs of true value labels and sensor data are generated by simulation as in this embodiment, various human behaviors are generated by a human (pedestrian) behavior model in the simulation, and their By combining human 3D models, human motion (gesture) data, and sensor noise models for actions, it is possible to acquire precision improvement data that can be used in real-world sensor systems at low cost and in a short time.

Apart from such a learner that learns the parameters of the recognition model, the approach of creating a large amount of data sets by simulation in this embodiment can also be applied to deep learning (so-called deep learning). Deep learning is, so to speak, a learner that learns the recognition model itself. Therefore, the data set obtained by the embodiment can be used for sensor calibration and a learning device (deep learning) aiming at acquiring a recognition model.

In the above-described embodiment, the human agent to be introduced into the virtual simulation space is shaped according to the 3D model, moved according to the objective, and made to perform motion according to the gesture model. Since a simulation that closely resembles a person (pedestrian) in the world can be executed, the sensing accuracy of the 3D range image sensor that senses such a human agent can be reproduced with the same accuracy as that in the real world. Therefore, the accuracy of a learner (not shown) that learns with the data set output in step S35 is also improved.

However, if it is possible to achieve sufficient accuracy simply by shaping the human agent to be inserted into the virtual simulation space according to the 3D model and moving it according to the objective, it is possible to prevent the human agent from making gestures in the virtual simulation space. is.

Similarly, a method of learning without assigning simulation parameters to the human agent is also possible.

10 ... Simulation system 12 ... Robot 14 ... Operation device 16 ... Simulation device 18 ... Robot application program 22 ... Virtual simulation space 24 ... Robot simulator 26 ... Human simulator 28 ... Physics engine 105 ... Display 202 ... Range image sensor

Claims (6)

A human agent generating means for inserting a human agent into a virtual simulation space simulating the real world in which a plurality of three-dimensional range image sensors are arranged at respective predetermined positions ;
update means for updating the human agent in the virtual simulation space;
a 3D model database having a plurality of 3D models indicative of a three-dimensional shape of a human agent; and an objective model database having objectives of the human agent;
The human agent generating means generates a human agent having a shape according to the 3D model selected from the 3D model database, the updating means moves the human agent according to the objective, and
true value label creation means for creating a true value label of the human agent, including at least the position, orientation and gesture of each human agent viewed from each of the plurality of three-dimensional range image sensors;
sensor data acquisition means for acquiring, as sensor data , a range image of each human agent captured by the plurality of three-dimensional range image sensors when the human agent is moved by the updating means; and combining the true value label and the sensor data. A simulation system comprising output means for outputting a data set. 2. The simulation system according to claim 1, wherein said human agent generating means generates different types of human agents, each having a shape according to a 3D model. further comprising a gesture database setting different gestures for the human agent;
3. The simulation system according to claim 1, wherein said updating means causes said human agent to make a gesture selected from said gesture database. further comprising a social force model database setting the social force model,
4. The simulation system according to any one of claims 1 to 3, wherein said updating means updates the position of said human agent according to said social force model. A simulation system for executing a simulation in a virtual simulation space simulating the real world in which a plurality of three-dimensional range image sensors are arranged at respective predetermined positions ,
The simulation system comprises a 3D model database having a plurality of 3D models that indicate a three-dimensional shape of a human agent, and an objective model database having objectives of the human agent, the computer of the simulation system comprising:
human agent generation means for inputting a human agent having a shape according to the 3D model selected from the 3D model database into the virtual simulation space;
updating means for updating the human agent by moving the human agent according to the objective within the virtual simulation space;
true value label creation means for creating a true value label of the human agent, including at least the position, orientation and gesture of each human agent viewed from each of the plurality of three-dimensional range image sensors;
sensor data acquisition means for acquiring, as sensor data , a range image of each human agent captured by the plurality of three-dimensional range image sensors when the human agent is moved by the updating means; and combining the true value label and the sensor data. A simulation program that functions as an output means for outputting datasets. 6. A learning device for sensor calibration or recognition model, comprising a learning device for learning a data set output by said output means according to any one of claims 1 to 5.

Images