JP6604832B2 - Machine learning support device - Google Patents

Description

  The present invention relates to a machine learning support device, and more particularly to a machine learning support device that supports machine learning and operation verification of an artificial intelligence device.

  In recent years, an image recognition system using a neural network for recognizing a subject (such as a person) in a video or an image has been developed.

  In this system, the neural network can recognize the subject only by learning the neural network in advance using image data including the subject to be recognized as input. For this reason, it is not necessary to perform an object feature extraction operation that has been performed manually by experts.

  However, the pre-learning requires a large amount of images of tens of thousands to hundreds of thousands, and in some cases, millions or more. In order to prepare images, they are also classified (tagged) in advance as teacher images. ) Takes a lot of effort and time.

  In order to solve this problem, for example, in Patent Document 1, an original image showing a subject is acquired, and a plurality of teacher images are created while performing image processing on the original image. And then. A technique for learning a neural network using image data of a plurality of teacher images having different contents of image processing as input is disclosed.

  Further, for example, in Patent Document 2, a subject (person) to be recognized is photographed in a real space, and a human body model (three-dimensional shape data) of the subject (person) is created from the acquired original image. A person is generated and displayed in a virtual space based on the dimensional shape data. Next, in the virtual space, when the user designates a change in the posture, movement, or physique of the person (using a mouse or the like), image processing corresponding to the designated content is added to the human body model (three-dimensional shape data), Based on the processed three-dimensional shape data, a teacher image including a person (subject) whose posture, motion, physique, etc. has changed is created. Then, according to the above procedure, a technique of creating a plurality of teacher images while variously changing the posture, motion, physique, etc. of a person (subject), and learning a neural network using the image data of those teacher images as input Is disclosed.

JP 2011-221840 A JP 2010-238134 A

  By the way, recently, an intelligent robot has been developed that has various adaptive feelings and an adaptive ability so that an appropriate treatment operation can be performed according to changes in the environment.

  In such an intelligent robot, it is important to have visual and tactile functions as sensory functions for recognizing the outside world. Vision is necessary for recognizing the entire environment including an object (object of operation such as lifting a robot), and tactile sensation is necessary for motion control in contact with the object.

  An intelligent robot generally includes an image sensor and a tactile sensor (including a force sensor) for these sensory functions. Therefore, it is necessary to process two sensory information (external recognition information) of image information and tactile information (including haptic information), and an intelligent robot uses a neural network to process the external recognition information. Many things are appearing.

  However, as described above, the conventional technology supports learning of image information among external recognition information, but does not support learning of tactile information (including haptic information).

  In addition, in the field of pharmaceuticals and foodstuffs, in addition to the inspection of wrinkles based on images of drugs and food (sensor information output from image sensors), in the field of pharmaceuticals and foodstuffs, detection of organic compounds and fluorescence using infrared spectroscopy Inspection of impurities and the like that cannot be understood only by an image, such as element detection by X-ray analysis, is performed together, and final inspection determination is performed by integrating the information.

  In this spectroscopic analysis, fluorescence spectrum information (sensor information) output from a multispectral camera (sensor) is used. Therefore, if the neural network learns various patterns of the sensor information, artificial intelligence devices including intelligent robots can make inspection decisions themselves, and rejects that contain impurities, etc. from the medicine or food to be examined You will be able to select and remove items.

  In the medical field, in the diagnosis of diseases such as cancer, fluorescence spectrum information (sensor information) output from a multispectral camera (sensor) is used to detect autofluorescence such as cancer. Furthermore, in surgery, images (sensor information) output from endoscopic cameras and operative field cameras (sensors) and measurements such as pulse, heart rate, blood pressure, and blood flow output from various measuring devices (sensors) during surgery Information (sensor information) is acquired, and the information is integrated to support surgery.

  Therefore, if the neural network learns various patterns of sensor information obtained from the plurality of sensors, the artificial intelligence device can support the diagnosis of disease and the performance of surgery. That is, in order to upgrade an artificial intelligence apparatus including an intelligent robot, it is necessary to make a neural network learn various patterns of sensor information (multi-sensor information) obtained from various sensors.

  However, as described above, the conventional technology supports learning of image information among multi-sensor information including external world recognition information, but does not support learning of other information. In addition, it is necessary to verify whether or not the operation of the artificial intelligence apparatus after the learning is assumed.

  An object of the present invention is to provide a machine learning support device that can support machine learning and operation verification of an artificial intelligence device for multi-sensor information.

In order to achieve the above object, a first machine learning support device according to the present invention includes an information generation unit that generates a plurality of virtual information corresponding to output information of a plurality of sensors, and the plurality of information generated by the information generation unit. A learning support unit that learns the artificial intelligence device using the virtual information, and a verification support unit that verifies the operation of the artificial intelligence device after the learning using the plurality of virtual information . The tactile information, which is one of the virtual information, includes the object position information indicating the position and posture of the object to be operated by the artificial intelligence device after learning in the virtual space, and the virtual information of the movable part of the artificial intelligence device. a movable section position information indicating the position and orientation in space, Ru is generated based on.

  According to the second machine learning support device of the present invention, in the first machine learning support device, the verification support unit performs the verification in the first machine learning support device in order to achieve the machine learning support of the above-described artificial intelligence device. A reinforcement learning unit is provided for further reinforcement learning of the artificial intelligence device after learning.

According to a third machine learning support device of the present invention, in order to achieve the machine learning support of the above-described artificial intelligence device, in the first or second machine learning support device, the information generation unit is known. the object position information together with generating an image of the virtual space on the basis of including the operation subject to the object of the learning after the artificial intelligence system to the information, indicating the position and orientation in the virtual space of the object based on the known information a visual information generation unit that generates, and the object position information, previously envisaged, it said indicating the position and orientation in the virtual space of the movable portion of the learning after AI the artificial intelligence system to be output from the device and a touch information generating unit that generates the haptic information based on a movable portion position information. The learning support unit learns the visual function of the artificial intelligence device using the image of the virtual space and the object position information as input, and uses the tactile information as input for the artificial intelligence device. Learn tactile functions.

In order to achieve the verification of the operation of the above-described artificial intelligence device, the fourth machine learning support device according to the present invention is characterized in that in the first or second machine learning support device, the information generation unit includes: A visual information generation unit that generates an image of a virtual space including an object to be operated by the artificial intelligence device after learning based on known information; and the virtual space of the object output from the artificial intelligence device after learning based on the first object position information indicating the position and orientation, and the movable portion position information indicating the position and orientation in the virtual space of the movable portion of the artificial intelligence system, which is output from the artificial intelligence system after the learning in and a touch information generating unit for generating the haptic information. The visual information generation unit updates the position and orientation of the object in the virtual space based on the first object position information and the movable part position information. Then, the verification support unit outputs the second object position information to the neural network of the learned artificial intelligence device by giving the image of the virtual space as an input, and also gives the tactile information as an input. Thus, based on the second object position information and the tactile determination information, the neural network of the artificial intelligence apparatus after learning outputs tactile determination information indicating whether or not the operation on the object is appropriate. The learning is performed so that the operation of the movable part of the artificial intelligence apparatus after the learning is verified, the tactile information generation unit is configured to generate the tactile information, and the visual information generation unit is configured to perform the update. A later artificial intelligence apparatus is made to output the first object position information and the movable part position information.

  In the machine learning support device of the present invention, the information generation unit generates a plurality of virtual information corresponding to the output information (multi-sensor information) of the plurality of sensors. Then, the learning support unit causes the artificial intelligence apparatus to perform machine learning using the plurality of virtual information generated by the information generation unit. Further, the verification support unit verifies the operation of the artificial intelligence apparatus after machine learning using the plurality of virtual information generated by the information generation unit.

  Therefore, according to the machine learning support apparatus of the present invention, it is possible to support machine learning of the artificial intelligence apparatus for multi-sensor information. Further, according to the machine learning support apparatus of the present invention, it is possible to support verification of whether or not the operation of the artificial intelligence apparatus after the machine learning is assumed.

It is a block diagram of the machine learning assistance apparatus which is embodiment of this invention. It is an example of the image of the virtual space containing the components piled up. It is a block diagram of the machine learning assistance apparatus which is a modification of embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, an example will be described in which machine learning and operation verification are performed on an intelligent robot that is a type of artificial intelligence device. Therefore, as the multi-sensor information, two sensory information of image information and tactile information (including haptic information), which are external world recognition information important for the intelligent robot, are used.

  FIG. 1 is a configuration diagram of a machine learning support apparatus according to an embodiment of the present invention. As shown in FIG. 1, the machine learning support device 100 includes a virtual space generation device 10, a virtual intelligent robot device 20, and a system control device 30.

  The virtual space generation apparatus 10 includes a CG unit 11 and a force generation unit 12.

  The CG unit 11 generates a virtual space of a three-dimensional orthogonal coordinate system corresponding to a real space by using conventional techniques such as CG (Computer Graphics), VR (Virtual Reality), and MR (Mixed Reality) and Manage spatial information. Furthermore, the CG unit 11 generates and appears in the virtual space an object that should exist in the operating environment of the intelligent robot or an object that is a target of an operation such as lifting the intelligent robot, and 3D corresponding to the object. Manage model information.

  In addition, the CG unit 11 generates an image of the virtual space including the appearing object and outputs the image information. Further, the CG unit 11 generates and outputs 6-dimensional position information J (x, y, z, θx, θy, θz) indicating the position and orientation of the object in the virtual space for the object in the generated image. To do. Note that x, y, and z represent the coordinate position of the object on three axes in the three-dimensional orthogonal coordinate system, and θx, θy, and θz represent the rotation angle of the object.

  The force generation unit 12 generates and outputs tactile information (including force information) corresponding to the output of the tactile sensor (including force sensor) provided on the finger, wrist, arm, etc. of the intelligent robot. To do. Specifically, if the tactile sensor of the intelligent robot is, for example, a 6-axis force sensor, the force generation unit 12 detects the 6-axis force component (fx, fy, fz, mx, my) to be detected by the sensor. , Mz) is generated by calculation, and the information is output as virtual six-axis force sense information. In this case, fx, fy, and fz represent triaxial force components, and mx, my, and mz represent triaxial moment components.

  Next, the virtual intelligent robot apparatus 20 includes a neural network 21, a neural network 22, a robot control unit 23, and an image processing unit 24.

  Here, the neural network will be briefly described. A neural network is an information processing system that models a cranial nervous system, and is configured by connecting a plurality of input / output elements called neurons with elements having a variable connection strength (coupling coefficient) called a synapse. Each neuron has a function of outputting “1 (true)” if the input is larger than the threshold value, and “0 (false)” if the input is smaller. Transfer functions such as a sigmoid function and a hyperbolic tangent (tanh) function are provided. It is realized using.

  Learning of the neural network is performed by adjusting the synapse coupling coefficient (weighting coefficient) between neurons and the threshold value of each neuron so that the output of the neural network approaches the target value. In the case of supervised learning, the learning of the neural network converges quickly by using the error back-propagation method (back propagation).

  In this embodiment, a multilayer feedforward neural network in which layers composed of a plurality of neurons are coupled in a direction from an input layer to an output layer via an intermediate layer is used.

  The neural network 21 receives the image information output from the CG unit 11 and recognizes an object (including an object to be operated by the intelligent robot) in the image of the virtual space, and also 6-dimensional position information of the recognized object. J (x, y, z, θx, θy, θz) is output. In order to obtain such an output, the correspondence between the image information of the virtual space including the object to be recognized and the six-dimensional position information J indicating the position and orientation of the object in the virtual space is stored in advance in the neural network 21. It is necessary to learn.

  For such a neural network 21 for image recognition, it is preferable to use a convolutional neural network (CNN) because of its high recognition rate.

  The neural network 22 receives virtual 6-axis force information (fx, fy, fz, mx, my, mz), which is force information corresponding to the output of the force sensor, output from the force sensor 12, Force sense determination information indicating whether or not an operation such as lifting an intelligent robot with respect to an object is appropriate is output. Specifically, the neural network 22 determines whether or not the input virtual 6-axis haptic information is within the range of the haptic information pattern in the case of an appropriate operation. Or information indicating failure is output as force sense determination information. In order to obtain such an output, it is necessary for the neural network 22 to learn in advance all patterns of virtual 6-axis force sense information that are appropriate operations (pass).

  The robot control unit 23 determines an operation such as lifting the intelligent robot with respect to the object based on the 6-dimensional position information J indicating the position and orientation of the object in the virtual space output from the neural network 21, and details of the determined operation The movement (movement) of the arm, wrist, finger, etc. of the intelligent robot is controlled accordingly.

  Further, the robot control unit 23 may finish the operation and move to the next operation based on the force sense determination information indicating whether the operation such as lifting the intelligent robot output from the neural network 22 is appropriate. Alternatively, it is determined whether the operation (movement) of the arm, wrist or finger of the intelligent robot is corrected and the operation is continued.

  When the robot controller 23 controls the movement (movement) of the arm, wrist, finger, and the like of the intelligent robot, the robot finger 6-dimensional position information R (x, x, which indicates the position and posture of the finger in the virtual space after the control is controlled. y, z, θx, θy, θz) are output. At this time, the robot control unit 23 indicates the position and posture in the virtual space after the control of the object that is the target of the operation such as lifting, together with the 6-dimensional position information R of the finger of the robot. The dimension position information K (x, y, z, θx, θy, θz) is also output. Note that the 6-dimensional position information R of the finger of the robot corresponds to each finger, for example, if the robot has five fingers and each finger is equipped with a tactile sensor (including a force sensor). The five pieces of information “6-dimensional position information R1 of the robot finger” to “6-dimensional position information R5 of the robot finger” are output. The 6-dimensional position information K is obtained by performing calculations based on the 6-dimensional position information J indicating the position and orientation of the object output from the neural network 21 and the 6-dimensional position information R of the robot finger. be able to.

  The 6-dimensional position information R of the finger of the robot and the 6-dimensional position information K of the operation target object output from the robot control unit 23 are input to the CG unit 11 and the force generation unit 12. Based on the input information, the CG unit 11 detects a change in the six-dimensional position of an object such as a component that appears in the virtual space managed by the CG unit 11, and reflects the detected change in the virtual space. To do. The force generation unit 12 generates 6-axis force information (fx, fy, fz, mx, my, mz) based on the input information, and the generated 6-axis force sense. Information is output as virtual 6-axis force information.

  The image processing unit 24 gives the image output from the CG unit 11 as it is as input to the neural network 21. However, when a reference image and a distance image are output from the CG unit 11, the reference image (an image equivalent to that captured by the camera of the intelligent robot) is provided as it is as an input to the neural network 21 and another sheet. The distance of the object is detected based on the distance image (parallax image), and the detected distance information is given to the robot controller 23.

  Then, the system control device 30 performs overall control of the virtual space generation device 10 and the virtual intelligent robot device 20, and executes machine learning and operation verification of the intelligent robot.

  The system control device 30 is realized by an information processing device such as a workstation. The virtual space generation device 10 may also be realized by a workstation or the like together with the system control device 30. In particular, an image processor that can perform GPU computing may be used for the graphics processing.

  In particular, at the time of verification, an actual image sensor (camera) and force sensor are used instead of the outputs of the CG unit 11 and the force generation unit 12 of the virtual space generation device 10 by switching according to an instruction from the system control device 30. Output can be used.

(Neural network learning)
Hereinafter, an example will be described in which the intelligent robot performs machine learning (supervised learning) of two sensory information of image information and tactile information (including haptic information) as external world recognition information.

  In order for the intelligent robot to work, two external world recognition functions are required. One is a function corresponding to human vision, and the other is a function corresponding to the tactile sensation (including force sensation) of the fingers of the hand.

(Learning image information / visual function)
The visual function necessary for the intelligent robot is to recognize an object from an image taken by a camera and detect 6-dimensional position information J (x, y, z, θx, θy, θz) indicating the position and orientation of the object. It is a function.

  While the intelligent robot advances the work according to the work procedure, there are many scenes in which the 6-dimensional position of an object (such as an object to be operated) must be detected.

  Therefore, the machine learning support apparatus according to the present embodiment generates the object by calculation in a virtual space of a three-dimensional orthogonal coordinate system corresponding to the real space as if it was photographed by an intelligent robot camera. Make it appear. The machine learning support apparatus according to the present embodiment includes an image of a virtual space including the object, and 6-dimensional position information J (x, y, z, θx, θy) indicating the position and orientation of the corresponding object in the virtual space. , Θz), and the neural network learns them as teacher information in advance.

  As a result, the intelligent robot can recognize the object in the work and detect the six-dimensional position of the object.

  The object mentioned here appears in the work space (= virtual space) according to the work procedure, such as parts, piled parts, semi-finished products, tip jigs, robot arms (arms), set stands, etc. All objects.

  By the way, the object appearing in the work space and the environment of the work space itself are all known, and the three-dimensional dimensions and the surface image of the appearing object can be acquired from the CAD information. Information on illumination in the workspace (information on illumination illuminance, direction, wavelength band, etc.) and camera information (camera position and direction, angle of view, lens zoom magnification, lens distortion, number of pixels, SN ratio, etc.) Information) can also be acquired in advance, including changes in their contents.

  In other words, based on these things, that is, based on known information, all objects that should exist in the work space of the intelligent robot in actual work can appear in the virtual space (= work space). it can. In this case, the 6-dimensional position of the appearing object is also known.

  Therefore, the CG unit 11 of the virtual space generation device 10 includes each object as a subject for all objects that should exist in the work space of the intelligent robot based on the known information in accordance with an instruction from the system control device 30. A spatial image is generated for each subject.

  Also, the CG unit 11 determines the position and orientation of the subject (object) in the virtual space corresponding to the image of the subject (object) in the virtual space based on the known information in accordance with an instruction from the system control device 30. 6-dimensional position information J (x, y, z, θx, θy, θz) is generated.

  Then, the system control device 30 learns the virtual network image of each object generated and output by the CG unit 11 and the corresponding 6-dimensional position information J to the neural network 21 of the virtual intelligent robot device 20 as teacher information. Let

  In the learning image set, images in which the 6-dimensional position of each object is changed little by little are prepared in consideration of the work error of the intelligent robot (position detection error due to camera accuracy, robot shake, etc.). . That is, the CG unit 11 generates an image in which each object is moved, rotated, enlarged, and reduced little by a work error, and this is used as an image set for learning.

  In this case, the change amount of the 6-dimensional position may be a roughness (step size) corresponding to the redundancy of pattern recognition of the neural network.

  An image corresponding to a change in illumination or a variation in illumination direction within an assumed range is also added to the learning image set.

  Further, an image of a 6-dimensional position that is considered to be clearly not present in the work procedure or operation step may be omitted from the learning set in order to shorten the learning time.

  In the neural network 21, the synapse coupling coefficient (weighting coefficient) between neurons obtained by learning, the threshold value of each neuron, and the like are not shown in FIG. Recorded in a storage device. Note that the coupling coefficient (weighting coefficient), neuron threshold value, and the like stored in the storage device are read from the storage device in advance by the robot control unit 23 when the intelligent robot executes the corresponding work procedure or operation step. The neural network 21 is loaded and used. Thereby, a local solution (local minimum) due to over-learning of the neural network can be suppressed, and recognition accuracy can be improved.

(Learning tactile information / tactile and haptic functions)
The tactile function, which is another external recognition function required for intelligent robots, is a tactile sensor (including a tactile sensor) that is output from multiple tactile sensors (including force sensors) provided on the fingers, wrists, and arms of the intelligent robot. This is a function for detecting whether or not the information is within the range of the tactile (including force) information pattern when the intelligent robot performs an appropriate operation or action.

  Therefore, the machine learning support apparatus according to the present embodiment performs tactile sensation (including force sensation) by calculation as if the tactile sensor (including force sensor) of the intelligent robot detects each operation or operation of the intelligent robot. ) Generate information. Specifically, the machine learning support apparatus has information on the 6-dimensional position of the object in the virtual space and the assumed 6-dimensional position indicating the position and posture of the intelligent robot such as fingers, wrists, and arms in the virtual space. Tactile (including haptic) information is generated by performing calculations based on information (x, y, z, θx, θy, θz). Then, the machine learning support device of the present embodiment generates all tactile (including haptic) information patterns (acceptance patterns) detected when the intelligent robot performs an appropriate operation on the operation target object. Then, the neural network is made to learn in advance as teacher information.

  As a result, the intelligent robot can determine whether or not the operation or action on the operation target object is appropriate in the work.

  Therefore, in accordance with an instruction from the system control device 30, the CG unit 11 of the virtual space generation device 10 manages the virtual space managed by the CG unit 11 based on the known information about an object such as a part to be operated. 6-dimensional position information J (x, y, z, θx, θy, θz) of the object is generated.

  Next, the force sense generation unit 12 receives the 6-dimensional position information J of the object generated by the CG unit 11 based on the above-described known information in accordance with an instruction from the system control device 30, and the virtual space assumed in advance. Tactile (including haptic) information is generated by performing calculations based on the position of the intelligent robot such as fingers, wrists, and arms and information on the 6-dimensional position indicating the posture. If the tactile sensor of the intelligent robot is, for example, a 6-axis force sensor, 6-axis force information (fx, fy, fz, mx, my, mz) is generated by calculation. At this time, the force generation unit 12 detects force information patterns detected when the intelligent robot performs an appropriate operation on the object to be operated (if there are n force sensors, 6 axes × (n = 6n patterns) are all generated.

  Then, the system control device 30 causes the neural network 22 of the virtual intelligent robot device 20 to learn the force information pattern (pass pattern) generated by the force sense generator 12 as teacher information.

  Note that the haptic information pattern that passes is variable in a certain range due to the work error of the intelligent robot (position detection error due to camera accuracy, robot shake, etc.). Therefore, the haptic information pattern corresponding to the variation amount is also generated by the haptic generating unit 12, and the generated pattern including the generated pattern is set as a pass pattern set.

  In this case, the amount of variation due to the workpiece error may be a roughness (step size) corresponding to the redundancy of pattern recognition of the neural network. This will reduce the learning time.

  In the neural network 22, the synapse coupling coefficient (weighting coefficient) between neurons obtained by learning, the threshold value of each neuron, and the like are not shown in FIG. Recorded in a storage device. Note that the coupling coefficient (weighting coefficient), neuron threshold value, and the like stored in the storage device are read from the storage device in advance by the robot control unit 23 when the intelligent robot executes the corresponding work procedure or operation step. And loaded into the neural network 22 for use.

(Example of verification of intelligent robot operation)
Hereinafter, an example of verifying the operation of an intelligent robot including a learned neural network will be described with reference to FIGS.

  Here, the operation of “random picking of parts” performed by the intelligent robot will be described as an example. Random picking is an operation in which an intelligent robot recognizes a part to be operated from a pile of parts and grabs the recognized part. Will be explained.

(Random picking by visual function)
Since the information about the three-dimensional structure of the part and the surface image can be obtained from CAD information that is known information, first, the CG unit 11 of the virtual space generator 10 shown in FIG. In response to the instruction, a part appears in the virtual space based on the known information. At this time, the CG unit 11 stacks a plurality of parts in the virtual space while randomly changing the position and orientation of the parts in the virtual space, that is, the six-dimensional position (x, y, z, θx, θy, θz) in the virtual space. To appear. Then, the CG unit 11 generates an image of a virtual space including the piled parts according to an instruction from the system control device 30. An example of this image is shown in FIG. In the example of FIG. 2, five parts (A, B, C, D, E) are stacked. This image is used as an image for verification.

  In random picking, the intelligent robot determines the distance between the gripping part of the part and an object (such as other parts in a pile) in order to determine whether or not its finger can easily grip the gripping part of the part. It needs to be detected. Therefore, in the machine learning support apparatus of the present embodiment, for example, a distance image is used based on the distance image using a reference image output from the distance image camera and a distance image of the distance image. However, in the verification, the CG unit 11 generates and outputs a reference image and a distance image. In order to further improve the accuracy of distance detection, an image obtained by projecting a special pattern onto the surface of an object such as a part by special illumination (such as infrared rays) may be used as the distance image.

  Next, in response to an instruction from the system control device 30, the CG unit 11 uses the previously generated verification image as a reference image, and based on the reference image, each component (for example, A and B in FIG. 2). , C, D, E), the distance image is generated by calculating 6-dimensional position information J (x, y, z, θx, θy, θz), and the reference image and the distance image are output. In addition, the process of this CG part 11 is performed based on the work procedure which is one of the known information.

  When the reference image and the distance image are input, the image processing unit 24 of the virtual intelligent robot apparatus 20 gives the reference image as it is to the learned neural network 21 of the virtual intelligent robot apparatus 20 as an input. . Prior to the input, the weight coefficient corresponding to the work procedure and the operation step, the threshold value of the neuron, and the like are read from the storage device and loaded into the neural network 21 by the robot control unit 23 as necessary. The

  By this input, the neural network 21 recognizes the part in which the entire image is shown from the reference images of the stacked parts, and the recognized parts, for example, parts A, B (C, D, E in FIG. , Each of which has a part of the image hidden by A or B), 6-dimensional position information J (x, y, z, θx, θy, θz) is output. This output is input to the robot controller 23.

  Further, the image processing unit 24 detects the distance of each component based on another distance image, and gives the detected distance information to the robot control unit 23.

  Based on the input 6-dimensional position information J indicating the position and posture of the part in the virtual space and the distance information of the part, the robot control unit 23 surrounds the gripping part of the part. It is determined whether or not the finger can be moved to the position of the part to be grasped without coming into contact with an object such as another part. The gripping part of the component is determined in advance for each component to be picked from the viewpoint of weight balance when gripped and whether the component is not damaged when gripped.

  Then, the robot control unit 23 selects parts that are easy to grasp from the parts to be operated (for example, parts A and B in FIG. 2). In the example of FIG. 2, the gripping part (g, g ′) of the part B cannot be gripped to surround the gripping part because the g-side surface is close to the part A. On the other hand, the gripping part (g, g ′) of the component A can be gripped because there is no object close to the surface. In this case, the part A is selected as an operation target by the robot control unit 23.

  In addition, the process of these robot control parts 23 is performed based on the work procedure which is one of the known information.

  Next, based on the 6-dimensional position information J of the selected part, the robot control unit 23 moves the intelligent robot's finger toward the position surrounding the gripping part of the part, the arm, wrist, and finger of the intelligent robot. Control the movement (movement). The processing of the robot control unit 23 is performed based on a work procedure that is one of known information. Further, when the intelligent robot itself is simulated in the virtual space, this operation is omitted because it is performed only by information processing.

  When the robot controller 23 reaches the position where the finger of the intelligent robot surrounds the gripping part of the operation target component by controlling its movement (movement), the robot finger 23 indicates the position and posture of the finger in the virtual space after the control. 6-dimensional position information R (x, y, z, θx, θy, θz) is output. In addition, at this time, the robot control unit 23 combines the 6-dimensional position information R of the finger of the robot with the 6-dimensional position information K (x indicating the position and orientation in the virtual space after the control of the operation target component. , Y, z, θx, θy, θz) are also output. The six-dimensional position information R and the six-dimensional position information K take into account work errors of the intelligent robot (position detection error due to camera accuracy, robot shake, etc.).

  6-dimensional position information R (x, y, z, θx, θy, θz) of the finger of the robot and 6-dimensional position information K (x, y, z, θx, θy, θz) of the operation target part are input. Then, the CG unit 11 of the virtual space generation device 10 detects a change in the six-dimensional position of an object such as a component that appears in the virtual space managed by the CG unit 11 based on these pieces of information. When the CG unit 11 detects a change, the CG unit 11 updates the position and orientation information of the object such as the changed part in the virtual space in order to reflect the change in the managed virtual space. It should be noted that here, the intelligent robot finger is only in a state where it has reached the position surrounding the gripping part of the operation target component, and the position (or 6-dimensional position) of the component has not changed, so the update (reflection) is Not done. The processing of the CG unit 11 is performed based on a work procedure that is one of known information.

(Random picking with haptic function)
In addition, the force generation unit 12 of the virtual space generation device 10 includes 6-dimensional position information R (x, y, z, θx, θy, θz) of the robot finger and 6-dimensional position information K (x , Y, z, θx, θy, θz), 6-axis force information (fx, fy, fz, mx, my, mz) is generated by calculation based on these pieces of information. For example, if there are n force sensors, “6-axis force information 1” to “6-axis force information n” are generated by calculation. Then, the force sense generating unit 12 outputs the generated 6-axis force sense information as virtual 6-axis force sense information. Note that the processing of the force generation unit 12 is performed based on a work procedure that is one of known information. The output virtual 6-axis force sense information is given as an input to the learned neural network 22 of the virtual intelligent robot apparatus 20. Prior to the input, the weighting coefficient, the neuron threshold value, and the like corresponding to the work procedure and the operation step are read from the storage device and loaded into the neural network 22 by the robot control unit 23 as necessary. The

  Next, whether the neural network 22 is appropriate for an operation or an operation on an object such as a component to be operated based on the input virtual 6-axis force sense information (fx, fy, fz, mx, my, mz). It judges whether or not, and haptic judgment information indicating pass or fail is output. This output is input to the robot controller 23.

  Based on the force determination information output from the neural network 22, the robot control unit 23 ends the operation and proceeds to the next operation, or corrects the motion (movement) of the intelligent robot finger and continues the operation. Judge whether to do. It should be noted that here, the intelligent robot finger has only reached the position surrounding the gripping part of the operation target component, and the intelligent robot finger does not touch the component. Therefore, since it is necessary to move to the operation (operation) of the next step for grasping the part, in this case, the haptic determination information indicating acceptance is output from the neural network 22. The processing of the robot control unit 23 is performed based on a work procedure that is one of known information.

  When the haptic determination information indicating acceptance is output from the neural network 22, the robot control unit 23 proceeds to an operation (operation) for gripping a component of the next step based on a work procedure that is one of known information. .

  The operation (operation) in which the finger of the intelligent robot grips the component is verified according to the following processing flow.

(Process 1)
When the robot controller 23 controls the motion (movement) of the intelligent robot finger to start gripping the part, the robot finger 6-dimensional position information R indicating the position and posture of the finger in the virtual space after the control is controlled. (X, y, z, θx, θy, θz) is output. In addition, at this time, the robot control unit 23 combines the 6-dimensional position information R of the finger of the robot with the 6-dimensional position information K (x indicating the position and orientation in the virtual space after the control of the operation target component. , Y, z, θx, θy, θz) are also output. As described above, the 6-dimensional position information R and the 6-dimensional position information K include work errors of the intelligent robot (position detection error due to camera accuracy, robot shake, etc.).

(Process 2)
6-dimensional position information R (x, y, z, θx, θy, θz) of the finger of the robot and 6-dimensional position information K (x, y, z, θx, θy, θz) of the operation target part are input. Then, the CG unit 11 of the virtual space generation device 10 detects a change in the six-dimensional position of an object such as a component that appears in the virtual space managed by the CG unit 11 based on these pieces of information. When the CG unit 11 detects a change, the CG unit 11 updates the position and orientation information of the object such as the changed part in the virtual space in order to reflect the change in the managed virtual space. In addition, the process of this CG part 11 is performed based on the work procedure which is one of the known information.

(Process 3)
In addition, the force generation unit 12 of the virtual space generation device 10 includes 6-dimensional position information R (x, y, z, θx, θy, θz) of the robot finger and 6-dimensional position information K (x , Y, z, θx, θy, θz), 6-axis force information (fx, fy, fz, mx, my, mz) is generated by calculation based on these pieces of information. Then, the force sense generating unit 12 outputs the generated 6-axis force sense information as virtual 6-axis force sense information. Note that the processing of the force generation unit 12 is performed based on a work procedure that is one of known information. The output virtual 6-axis force sense information is given as an input to the learned neural network 22 of the virtual intelligent robot apparatus 20. Prior to the input, the weighting coefficient, the neuron threshold value, and the like corresponding to the work procedure and the operation step are read from the storage device and loaded into the neural network 22 by the robot control unit 23 as necessary. The

(Process 4)
Next, whether the neural network 22 is appropriate for an operation or an operation on an object such as a component to be operated based on the input virtual 6-axis force sense information (fx, fy, fz, mx, my, mz). It judges whether or not, and haptic judgment information indicating pass or fail is output. This output is input to the robot controller 23.

(Process 5)
Based on the force determination information output from the neural network 22, the robot control unit 23 ends the operation and proceeds to the next operation, or corrects the motion (movement) of the intelligent robot finger and continues the operation. Judge whether to do. The processing of the robot control unit 23 is performed based on a work procedure that is one of known information.

(Process 6)
Here, the robot control unit 23 moves to the operation (operation) of the next step as described above when the haptic determination information indicates a pass, but if it indicates a failure, the robot controller 23 moves the finger of the intelligent robot. Correct (movement) and continue the operation. Specifically, the robot control unit 23 grasps the target component by controlling the movement (movement) of the finger of the intelligent robot by so-called servo control. At this time, since the finger of the intelligent robot is moved little by little so as to grasp the part by the servo control, the robot control unit 23 responds to the change in the movement of the finger every time feedback of the servo control is performed. Information on the position and orientation of the finger in the virtual space and information on the position and orientation of the part operated by the finger in the virtual space are output. Therefore, in the machine learning support device according to the present embodiment, every time feedback of servo control is performed until the component is properly gripped by the finger of the intelligent robot, the processing after the control in the above (Process 1) is The determination process up to process 5) is repeatedly executed.

  As described above, in the machine learning support apparatus according to the present embodiment, precise control is required for the force information, so that the robot control unit 23 performs the 6-dimensional position information R of the robot finger and the operation target component (object ) Is fed back to the force generation unit 12 to perform closed loop control. In this case, the neural network 22 has the same function as a so-called window comparator that determines whether or not the detected haptic information is within an appropriate range.

  Here, an example of intelligent robot finger servo control will be described regarding the gripping operation of the component.

  First, the gripping part of the part is surrounded by fingers of the intelligent robot to start a gripping operation, and the movement is temporarily stopped from the finger where the pattern of force information indicating that the part touching the gripping part is detected.

  Next, when it is detected that all the fingers have touched the gripped part based on the force information pattern, the gripping operation of each finger is resumed all at once.

  These gripping operations are repeated until a force information pattern suitable for gripping a component is obtained.

  In this way, the gripping operation of the component can be performed while absorbing the detection error (work error) of the six-dimensional position of the target component.

(Modification of the embodiment)
Hereinafter, a machine learning support apparatus that is a modification of the above embodiment will be described. The machine learning support device of this modification is obtained by adding elements for performing reinforcement learning to the configuration of the machine learning support device of the above embodiment. Therefore, in the following, only the additional elements will be described, and the same reference numerals are given to the constituent elements common to the above-described embodiment, and redundant description will be omitted.

  The difference between this modification and the above embodiment is that a reinforcement learning evaluation unit 40 is added as shown in FIG.

  Also in the machine learning support apparatus of this modification, the robot control unit 23 uses the 6-dimensional position information R of the finger of the robot and the operation target component (for the haptic information as in the machine learning support apparatus of the above embodiment). The closed loop control is performed by feeding back the six-dimensional position information K of the object) to the force generation unit 12. This closed loop can be used not only for verification but also as a control loop for reinforcement learning.

  By the way, the visual function of the outside world recognition function that is important for intelligent robots recognizes the entire environment including objects such as parts to be operated, and there is ambiguity in the accuracy of detection by the camera. No control is required. However, the tactile sensation (including force sense) function needs to be controlled precisely since it is necessary to control the operation in a contact state with an object.

  Therefore, in supervised learning that prepares information such as images and tactile sensations (including haptics) necessary for neural network learning in advance as teacher information, it is very difficult to perform learning for precise control. Accompany.

  Therefore, the tactile information (including the force sense) information can be solved by causing the neural network to learn by reinforcement learning.

  Therefore, in the machine learning support apparatus according to this modification, the reinforcement learning of the neural network for the tactile information (including the haptic information) is performed as follows.

  First, in order for reinforcement learning to be performed efficiently, the neural network 22 is subjected to supervised learning as much as necessary as prior learning, and then reinforcement learning is performed while giving a work error while repeatedly verifying the operation of the intelligent robot. To do.

  Specifically, first, prior learning is performed on the neural network 22. There are various methods for pre-learning, but here, it is assumed that one haptic information pattern that is close to ideal is learned as teacher information.

  That is, according to an instruction from the system control device 30, the CG unit 11 of the virtual space generation device 10 manages a virtual space managed by the CG unit 11 based on the above-described known information on an object such as a part to be operated. 6-dimensional position information J (x, y, z, θx, θy, θz) of the object is generated.

  At this time, together with the generation, the CG unit 11 causes an object such as the part to appear in the virtual space, manages the corresponding 3D model information, and also displays an image of the virtual space including the object such as the part. Is generated as a verification image.

  Next, the force sense generation unit 12 receives the 6-dimensional position information J of the object generated by the CG unit 11 based on the above-described known information in accordance with an instruction from the system control device 30, and the virtual space assumed in advance. 6-axis haptic information (fx, fy, fz, mx, my, mz) is generated by performing calculations based on the position of the intelligent robot's fingers, wrists, arms, etc. and 6-dimensional position information indicating the posture. .

  Then, the system control device 30 uses the pattern of the six-axis force sense information generated by the force sense generating unit 12, in this case “fx, fy, fz, mx, my, mz”, as the teacher information of the virtual intelligent robot device 20. The neural network 22 is trained.

  In this way, after the supervised learning with the neural network is performed in advance, the same verification processing as the above-described random picking is performed.

  In other words, in response to an instruction from the system control device 30, the CG unit 11 outputs the previously generated verification image. This image is input to the image processing unit 24 of the virtual intelligent robot apparatus 20.

  The image processing unit 24 gives the input verification image as it is to the neural network 21 that has been learned by the virtual intelligent robot apparatus 20 as input. Here, the neural network 21 learns in advance an object such as an operation target part included in the image by the learning method described in the visual function necessary for the intelligent robot.

  With this input, the neural network 21 recognizes an object such as a component from the verification image and outputs 6-dimensional position information J (x, y, z, θx, θy, θz) of the recognized object such as the component. To do. This output is input to the robot controller 23.

  Based on the 6-dimensional position information J of an object such as an input part, the robot control unit 23 moves the intelligent robot's finger toward the position surrounding the gripping part of the part, the arm, wrist, and finger of the intelligent robot. Control the movement (movement).

  The subsequent verification processing is the same as the processing described in the above-described random picking portion, and thus a duplicate description is omitted. However, regarding the operation (operation) in which the finger of the intelligent robot grips an object such as a component, the following (Process 3-1) and (Process 3-) are performed after the above (Process 3) is performed. The process 2) is additionally executed.

(Process 3-1)
The reinforcement learning evaluation unit 40 receives the virtual 6-axis force sense information (fx, fy, fz, mx, my, mz) output from the force sense generation unit 12 and sets the virtual 6-axis force sense information in advance. The evaluation is performed based on the rule and the output / non-output of the learning instruction signal to the neural network 22 is controlled based on the evaluation result. This learning instruction signal includes reward information. The rule set in advance is a pass range of a pattern of force information for evaluation that should be determined in advance.

(Process 3-2)
When the learning instruction signal is output from the reinforcement learning unit evaluation unit 40, the neural network 22 outputs the virtual output from the force generation unit 12 that is input to the neural network 22 in parallel with the input to the reinforcement learning evaluation unit 40. Six-axis force sense information (fx, fy, fz, mx, my, mz) is learned as a pattern of pass force information. In this learning, only when the value of the reward information included in the learning instruction signal is larger than that in the previous learning (there is more reward), the input virtual 6-axis force information is learned as a pattern of passing force information. To do.

  In the above description, as a prior learning of the neural network, one ideal force information pattern (ideal force information pattern) is learned as teacher information. Therefore, as an acceptable range of the haptic information pattern, for example, the ideal haptic information pattern is, for example, a median value, and a range indicated by a value larger by a predetermined amount than the median value and a value smaller by a predetermined amount. It becomes an acceptable range. In this case, the reward increases as it approaches the median, and decreases as it goes away from the median.

  Therefore, in the machine learning support device of this modification, in the reinforcement learning of the haptic information, the robot control unit 23 can obtain more rewards, that is, the virtual 6-axis haptic information pattern has a median value side. Closed loop control is performed while correcting the movement of the intelligent robot's finger so that As a result, the recognition accuracy of the neural network 22 is improved, so that the intelligent robot can recognize appropriate force information.

  In the neural network 22, the synapse connection coefficient (weighting coefficient) between neurons obtained by reinforcement learning, the threshold value of each neuron, etc. are not shown in FIG. It is recorded in the illustrated storage device. Note that the coupling coefficient (weighting coefficient), neuron threshold value, and the like stored in the storage device are read from the storage device in advance by the robot control unit 23 when the intelligent robot executes the corresponding work procedure or operation step. And loaded into the neural network 22 for use.

(Use of other sensor information)
In the above, an example in which two sensory information of image information and tactile information (including haptic information) is used as multisensor information has been described. However, the sensor information is not limited to the two.

  For example, an example in which auditory information output from an auditory sensor (such as a microphone) is used will be described. An abnormal sound pattern generated from an arm (arm) of an intelligent robot is generated by the virtual space generator 10 and the generated pattern is generated. The pattern is learned in advance by the neural network 21 or 22.

  Next, when virtual auditory information indicating an abnormal sound is output from the virtual space generation device 10 as if it is output from an auditory sensor provided in the intelligent robot during operation of the intelligent robot, the virtual auditory information is input. The neural network 21 or 22 thus recognized recognizes the occurrence of an abnormal sound by pattern recognition and outputs failure information indicating that the abnormal sound has occurred.

  Then, the robot controller 23 to which the failure information is input controls the operation of the arm of the intelligent robot so as to urgently stop the arm (arm) of the intelligent robot.

  As described above, in the above machine learning support device, it is as if the external world recognition sensor detects external world recognition information such as images and tactile sensation (including force sense) necessary for the external world recognition of an artificial intelligence device such as an intelligent robot. It is generated in a virtual space by calculation, and the information is used to verify (simulate) the operation of an artificial intelligence device such as an intelligent robot. Therefore, according to the machine learning support device described above, an artificial intelligence device such as an intelligent robot can be learned at a high speed, so that the labor and time required to start up an artificial intelligence device such as an intelligent robot can be greatly reduced. Can do.

  Moreover, according to said machine learning assistance apparatus, the machine learning of artificial intelligence apparatuses, such as an intelligent robot, can be supported about multisensor information.

  In addition, according to the machine learning support device described above, verification of whether or not the operation of an artificial intelligence device such as an intelligent robot after machine learning is assumed can be supported.

  In addition, according to the machine learning support device described above, even for functions that require precise control such as tactile sensation (including force sensation), learning of an artificial intelligence device such as an intelligent robot is facilitated by the above reinforcement learning. It can be carried out.

  Further, according to the machine learning support device, it is possible to cause an artificial intelligence device such as an intelligent robot to learn an operation that is difficult to program in a short time by the reinforcement learning. In addition, an artificial intelligence device such as an intelligent robot can be easily learned about a new operation corresponding to a change or addition of a work process.

  When the above learning and / or verification work is completed, in order to complete an artificial intelligence apparatus such as an intelligent robot, the weighting factors of the neural networks 21 and 22, the threshold values of the neurons, the neural networks 21 and 22 themselves, and the robot Part or all of the functions of the control unit 23 are transplanted to an artificial intelligence device such as an intelligent robot.

  In the above description, an artificial intelligence apparatus such as an intelligent robot has been described as a real machine. However, an artificial intelligence apparatus such as an intelligent robot may be a virtual apparatus on simulation instead of an actual machine.

  In the above machine learning support device, the intelligent robot is an external device, but a function of generating a virtual space may be included in an artificial intelligence device such as an intelligent robot. Then, in addition to the original processing of the artificial intelligence device, the above learning and verification, and the generation of virtual space image information and haptic information, etc. are collectively performed by an arithmetic device such as a GPU (Graphics Processing Unit). The system can be processed and the efficiency of the system can be improved.

  The machine learning support apparatus may be combined with a CAD (Computer Aided Design) system or the like as one function of a computer integrated manufacturing system CIM (Computer Integrated Manufacturing). By doing so, it is possible to perform assembly simulation simultaneously with product design, so that product design can be optimized.

(Example of application to inspection equipment)
In the inspection of pharmaceuticals, foodstuffs, materials, etc., in addition to inspection of sputum by images, inspection of impurities, etc. that are not understood by images, such as detection of organic compounds by infrared spectroscopic analysis and detection of elements by fluorescent X-ray analysis, etc. In addition, the final inspection determination is performed by integrating the information.

  These spectroscopic (spectral) data are called fingerprint regions, and the spectrum patterning is roughly performed. Based on this, virtual data for learning the artificial intelligence apparatus can be created.

  In this case, in the machine learning support apparatus described above, for example, a virtual data set for learning is created by randomly changing other waveform portions while retaining the characteristic spectrum portion of the spectral data.

  Next, the machine learning support apparatus learns the artificial intelligence apparatus in advance using the learning virtual data set.

  The machine learning support apparatus verifies the inspection process by causing objects to be inspected having various wrinkles to appear in the virtual space.

  Further, the machine learning support apparatus improves the inspection ability of the artificial intelligence apparatus by performing reinforcement learning (unsupervised learning) during the verification.

(Example of application to ME camera)
Endoscopic cameras and surgical field cameras require functions for supporting the performance of surgery in addition to the purpose of observing and recording images. For example, in cancer surgery, the camera performs enhancement processing on an image of a blood vessel structure unique to cancer, detects autofluorescence of cancer (using a fluorescent dye or a reagent), and contrasts blood vessels. This will support intraoperative diagnosis of cancerous lesions in the performance of surgery.

  By the way, cancer has a unique blood vessel structure because of its vigorous proliferative properties, and the blood vessel structure varies depending on the site of occurrence of cancer. In other words, because cancer has a unique pattern in the blood vessel structure depending on the site of occurrence, patterning for diagnosis has already been made, and based on this, virtual data for learning the artificial intelligence device can be created .

  In addition, autofluorescence is detected by a multispectral camera, etc., that is different from the camera for normal photography. The autofluorescence spectrum also has its own pattern depending on the cancer tissue. Virtual data for intelligent device learning can be created.

  Further, during the operation, measurement of pulse, heart rate, blood pressure, blood flow, and the like is performed by a measuring instrument, and the operation for integrating the operation is usually supported. Since these measurement data are also generally patterned, virtual data for learning of the artificial intelligence apparatus can be created based on this pattern.

  Therefore, in the machine learning support apparatus, for example, the vascular structure pattern, the autofluorescence spectrum pattern, and the measurement data pattern are variously changed within a predetermined range assumed in actual surgery to learn virtual data for learning. Create a set.

  Next, the machine learning support apparatus learns the artificial intelligence apparatus in advance using the learning virtual data set.

  The machine learning support apparatus verifies the intraoperative diagnosis in the performance of the operation by causing the internal model of the human body including the cancerous lesion to appear in the virtual space. In addition, the machine learning support apparatus improves the diagnosis ability of the artificial intelligence apparatus by performing reinforcement learning (unsupervised learning) during the verification.

(Other)
In the medical field, modeling of the inside of the human body is performed for simulation of endoscopes and robot surgeries. Therefore, if the modeled information is used to increase the level and variation of the model in the virtual space in the machine learning support device, the artificial intelligence device can be learned in the virtual space in a short time. become able to. This makes it possible to support safe insertion of an endoscope, support for surgery, and support for avoiding medical errors.

  In addition, since the modeling of the human body is also progressing, if the modeled information is used, various specimen models appear in the virtual space in the machine learning support apparatus, and verification and learning of the support for performing the surgery are performed. It can be carried out.

  Furthermore, if the information modeled inside the human body and the human body is used, in addition to the support for verification of the definitive diagnosis in the virtual space in the machine learning support device, the support for reducing the excision part in the operation, It can also be applied to support for ensuring safety, such as performing surgery while avoiding dangerous places such as arteries.

  Also, in the field of automobiles, information that models roads and the like is already in the navigation system and the like, and if the modeled information is used, learning of automatic driving of the automobile in the virtual space in the above machine learning support device Can be applied to.

DESCRIPTION OF SYMBOLS 10 Virtual space generator 11 CG part 12 Force sense generation part 20 Virtual intelligent robot apparatus 21 Neural network 22 Neural network 23 Robot control part 24 Image processing part 25 Reinforcement learning instruction part 30 System controller 40 Reinforcement learning evaluation part 100 Machine learning support apparatus

Claims (4)

An information generation unit that generates a plurality of virtual information corresponding to the output information of the plurality of sensors;
A learning support unit for learning the artificial intelligence device using the plurality of virtual information generated by the information generation unit;
A verification support unit that verifies the operation of the artificial intelligence apparatus after the learning using the plurality of virtual information ,
The tactile information, which is one of the plurality of virtual information, includes object position information indicating the position and orientation of the object to be operated by the artificial intelligence device after learning in the virtual space, and a movable part of the artificial intelligence device. Is generated based on the movable part position information indicating the position and orientation in the virtual space.
A machine learning support device. The machine learning support device according to claim 1,
A machine learning support apparatus, comprising: a reinforcement learning section that further learns the artificial intelligence apparatus after learning during the verification performed by the verification support section. The machine learning support apparatus according to claim 1 or 2,
The information generator is
And generates an image of the virtual space including the operation subject to the object of known based on information the learning after the artificial intelligence system, the object position indicating a position and a posture in the virtual space of the object based on the known information A visual information generator for generating information;
And the object position information, previously envisaged, on the basis of the said movable portion position information indicating the position and orientation in the virtual space of the movable portion of the artificial intelligence system to be output from the artificial intelligence system after the learning A tactile information generation unit that generates tactile information,
The learning support unit learns the visual function of the artificial intelligence device using the image of the virtual space and the object position information as input, and also uses the haptic information as input for the haptic function of the artificial intelligence device. Machine learning support device characterized by learning. The machine learning support apparatus according to claim 1 or 2,
The information generator is
A visual information generation unit that generates an image of a virtual space including an object to be operated by the artificial intelligence device after learning based on known information;
First object position information indicating the position and posture of the object in the virtual space output from the learned artificial intelligence device, and a movable part of the artificial intelligence device output from the learned artificial intelligence device. and a touch information generating unit that generates the haptic information based on said movable portion position information indicating the position and orientation in the virtual space,
The visual information generation unit updates the position and orientation of the object in the virtual space based on the first object position information and the movable part position information;
The verification support unit outputs the second object position information to the neural network of the learned artificial intelligence device by giving the image of the virtual space as an input, and also gives the tactile information as an input. , Causing the neural network of the learned artificial intelligence device to output tactile determination information indicating whether or not the operation on the object is appropriate, and learning based on the second object position information and the tactile determination information Verifying the operation of the movable part of the later artificial intelligence device, causing the tactile information generation unit to generate the tactile information, and causing the visual information generation unit to perform the update. An artificial intelligence device causes the first object position information and the movable part position information to be output. A machine learning support apparatus, comprising: