JP7798613B2 - Posture determination method - Google Patents

Description

The present invention relates to an attitude determination method.

In conventional end effector control, the wrist posture determined from the fingertip contact positions that allow stable grasping is stored in a database using polygon information (known objects) obtained offline. Additionally, online, wrist positions stored offline in the database are used through conditional search. Devices that manipulate the posture of robots have also been proposed (see, for example, Patent Document 1).

Patent No. 6476358

However, with conventional technology, when remotely operating an end effector to grasp an unknown object, the wrist posture data for known objects stored offline in a database does not contain information such as the orientation in which the unknown object can be grasped, making it impossible to determine the posture of the end effector, etc.

The present invention was made in consideration of the above problems, and aims to provide an attitude determination method that can determine attitude online.

(1) To achieve the above object, one aspect of the present invention provides an attitude determination method for a mobile mechanism that is capable of grasping and manipulating a target object, is connected to an end effector having multiple operation attitudes, and is capable of moving the position of the end effector, and includes a measurement step of measuring the target object using a sensor, a calculation step of calculating multiple points that indicate the center of the measured target object, a grasp representative point determination step of determining one of the multiple points as a grasp representative point corresponding to one of multiple selected operation shapes, and an attitude determination step of determining the attitude of the mobile mechanism from the grasp representative point.

(2) Furthermore, in an attitude determination method according to one aspect of the present invention, the attitude determination step may determine the attitude of the moving mechanism based on a taxonomy relating to the gripping of the object to be gripped, based on the attitudes of the target object and the moving mechanism.

(3) In addition, in the posture determination method according to one aspect of the present invention, the posture determination step may determine the grasp representative point using a model trained in advance using a pose scene graph that represents the relationship between the end effector and the grasped object, the relationship between the target object and objects surrounding the target object, and the posture of the target object using nodes and edges.

(4) In addition, in a posture determination method according to one aspect of the present invention, the nodes of the pose scene graph may be based on the coordinates of the target object as seen from the end effector, and the edges of the pose scene graph may be costs based on the relationship between the end effector and the grasped object, the relationship between the target object and objects surrounding the target object, and the posture of the target object.

(5) In addition, in the posture determination method according to one aspect of the present invention, the posture determination step may determine the grasp representative point using a model that has been trained in advance using a cost scene graph that represents the relative pose of the end effector and the target object, the target object and multiple points indicating its center, and possible postures of the mobile mechanism using nodes and edges.

(6) In addition, in a posture determination method according to one aspect of the present invention, the cost scene graph may include, as costs for edges, information based on the relative pose of the grip center with respect to a plurality of points indicating the center of the target object, information based on the difference between the curved surface of the target object around the plurality of points indicating the center of the target object and the curved surface of the finger portion of the end effector, and a Quality Measure value.

Based on (1) to (6), the posture can be determined online.

10A and 10B are diagrams for explaining an example of a method for determining a point indicating a grip center according to an embodiment. 10A and 10B are diagrams illustrating examples of center lines when the size of an object to be grasped changes. 10A and 10B are diagrams illustrating examples of centerlines for three-dimensional writing styles. FIG. 10 is a diagram for explaining selection of a grip representative point. FIG. 10 is a diagram for explaining determination of a grasp representative point in grasping an object. FIG. 10 is a diagram showing examples of names in a taxonomy. This is an image diagram of the context of the relationship between objects and the relationship between objects and actions. FIG. 1 is a diagram for explaining a scene. FIG. 10 is a diagram for explaining a graph. 10A and 10B are diagrams for explaining learning of a hand direction and an object posture according to an embodiment. FIG. 10 is a diagram for explaining an example of use of a learning result according to the embodiment. 1 is a diagram showing a scene of the relationship between a hand and an object and a simple graph representation thereof; FIG. 10 is a diagram illustrating an overview of a wrist posture estimation process. FIG. 10 is a diagram for explaining a cost scene graph. 1 is a diagram illustrating an example of the configuration of an end effector that performs work according to an embodiment. FIG. FIG. 2 is a diagram illustrating an example of the configuration of a control device according to the embodiment. 10 is a flowchart of a processing procedure performed by a wrist posture determination device during learning according to an embodiment. 10 is a flowchart of a processing procedure performed by a wrist posture determination device during work according to an embodiment. FIG. 10 is a diagram for explaining a method for determining a wrist posture when no model is used.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized.
In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted.
Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

<Overview>
In the posture determination method of the embodiment, the wrist posture that can be processed online is determined based on the geometric structure of the object (mainly skeleton information of unknown polygons), the grasp center of the taxonomy, the fingertip curvature during grasp, the quality measure around the fingertip contact point (minimum sum of squares of lateral forces (friction forces)), and a scene graph of the hand, object, and surrounding environment.
In addition, in the posture determination method of the embodiment, the relative pose between the hand and the grasped object, the relative pose between the grasped object and the surrounding environment, and the relative pose differences between the grasped object and the floor are defined as a pose graph. The posture determination method of the embodiment defines the relative pose of the grasp center with respect to the skeleton, the convolution of the difference between the surface curves of the objects around the skeleton and the finger curves, and the Quality Measure value as a cost. Furthermore, in the posture determination method of the embodiment, the wrist posture is determined by convolving the edges of two scene graphs around a node to find the node with the smallest cost.

<Example of method for determining the point indicating the grip center>
FIG. 1 is a diagram for explaining an example of a method for determining a point indicating a grip center according to this embodiment.
First, a black-and-white image g101 of the object to be grasped viewed from above is created to represent it as a polygon, as in image g100. Voronoi processing is then used to determine the center of the polygon using line g102, a set of neighboring points. An axis g103 is then defined, either perpendicular to line g102, a set of points indicating the center of the object, or perpendicular to the outer surface of the object to be grasped. These are used to calculate frictional forces, etc., when pinching the long and short axes of two fingers.

Image g110 is an example of a 2D or 3D copy of the centerline generated in this way.
Image g120 is an image (polygon) of a rock, which is an object to be grasped g121, viewed obliquely from the side. A set of multiple adjacent points g122 is the target grasp center, and point g123 is the grasp center of the end effector. In this embodiment, the multiple adjacent points g122 representing the center of the object are also referred to as a center line.

The rule for creation is that if the center line is short (for example, 5 samples or less), the object center frame is selected as the target grasp pose. Also, above the target grasp pose, only the quaternion (0,0,0,1) is used, and the finger direction is the same as the x-axis of the body frame.

FIG. 2 is a diagram showing an example of the center line when the size of the object to be grasped changes.
As shown in Figure 2, the smaller the target object, the shorter the center line. If the target object is a sphere or a circle, the center line will have only one point because multiple points converge at the center. The center line, which is made up of multiple adjacent points, also contains information about the size, longitudinal direction, and orientation of the object.
In FIG. 2, reference symbol g11 denotes the target grip center of the target object, and reference symbol g12 denotes the grip center of the end effector.

Figure 3 shows an example of a center line for a three-dimensional writing style. Center line detection involves converting the image to black and white and creating a center line, as described above. Image g201 is an example of a detected center line for a torus-shaped three-dimensional object. Reference symbol g202 is a three-dimensional object, line g203 represents a center line candidate, and line g204 represents example center line information obtained by averaging line g202. Image g205 is an example of a detected center line for a screw-shaped three-dimensional object. Reference symbol g206 is a three-dimensional object, line g207 represents a center line candidate, and line g208 represents example center line information obtained by averaging line g207. Center lines can be generated using well-known techniques for objects wrapped around thin strings, doll-like objects, animals, and even non-continuous objects.

<Determination of gripping representative point>
As described above, the center line is made up of multiple points. When grasping, it is necessary to select one point (the grasp representative point) from these multiple points. The selection of the grasp representative point will be explained below.
4 is a diagram illustrating the selection of the grasp representative point. When a trained model is not used, a point is selected from multiple points that is near the center of gravity of the object model, in the direction in which the end effector is approaching, and that is likely to provide a stable grasp without tilting when grasped.

However, in reality, it may be difficult to determine the grip representative point using only such a center line.
Fig. 5 is a diagram for explaining how to determine a grasp representative point when grasping an object. Fig. 6 is a diagram showing examples of names in the taxonomy.

Image g221 is an example of a medium-wrap gripping the side of a bottle from the side in the taxonomy (see Reference 1).
Image g222 is an example of a bottle being held in a holder or the like and grasped from the side of the upper part of the side of the bottle using a medium-wrap in the taxonomy.
Image g223 is an example of pinching the side of a bottle from above with Precision-3 fingers in the taxonomy when the bottle is lying down.
Image g224 is an example of grasping a bottle from above using a Tripod in the taxonomy.
Image g225 is an example of a medium-wrap in the taxonomy supporting a bottle from below.

In images g221 to g225, the areas indicated by symbols g231 to g235 are areas that are easy to grasp. This area cannot be determined solely by the area surrounding the center; it is necessary to consider the state of the target object and the direction from which the end effector is approaching. For this reason, grasping requires a function that can understand the context and change the way of holding.

Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human Grasp Types” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

As explained using Figure 5, the grasping scenes include holding a bottle, holding a bottle in a holder, holding a bottle on the ground, pinching the bottle cap, and supporting the bottom of the bottle. These common elements indicate the need to understand two contexts: the relationship between objects and the relationship between objects and actions.

FIG. 7 is an image diagram of the context of the relationship between objects and the relationship between objects and actions.
Image g251 is an image of the context of the relationship between objects. Image g252 is an image of the context of the relationship between objects and actions.
As in image g251, there are also relationships between objects, such as when an object is sandwiched between something else or placed on a desk or floor. Such relationships (states) are meanings, and meanings are added as costs to the relationships between objects.
For example, in the case of pinching an object as in image g252, there is an object and the meaning of the action of pinching. The meaning is taken into account as a cost for the relationship between the object and the action.

<Scene graph>
Next, the scene graph will be described.
FIG. 8 is a diagram for explaining a scene. In this embodiment, a scene graph is used to represent the environment surrounding an object. For example, as shown in FIG. 8, a scene represents a state in which a bottle, a glass, and a bowl are placed on a desk. This scene can be represented graphically as shown in image g271 in FIG. 9. The bottle has a lid. FIG. 9 is a diagram for explaining a graph. The graph represents the relationship between objects. For example, when viewed from the bottle, the lid is on top, and when viewed from the lid, the bottle is on the bottom. Furthermore, when viewed from the bottle, the glass is on the right, and when viewed from the glass, the bottle is on the left. The bottle, glass, and bowl are on the desk.

The relationship between objects in image g271 in Figure 9 is represented by nodes, edges, and nodes, as in image g272 in Figure 9, and these edges correspond to the feature, which is the cost. Note that the words that express the relationship are expressed numerically as labels. In this way, in a scene graph, the relationship between objects can be represented graphically using nodes and labels.

<Learning the end effector and object pose>
FIG. 10 is a diagram illustrating learning of the hand direction and object posture according to this embodiment. As shown in image g301, this embodiment learns the orientation of the hand relative to the object in order to perform online processing on an unknown object. Furthermore, for example, if a glass is overturned, the meaning of "up" may not be clear from the object's orientation, such as whether the top is on top or whether the side of the glass is on top. For this reason, this embodiment learns the direction and object posture separately. Therefore, in this embodiment, the relative poses, the hand direction relative to the object and the object posture, are learned in advance. Note that nodes are determined by captured images and sensor values. Edges are generated using a trained model. The graph representation is for the object as seen from the hand.

Image g302 is an image of learning the direction of the hand from the perspective of the target object (mass system). Note that the feature quantities of the edge are, for example, the distance between nodes, the angle between nodes, and the position vector.
Image g303 is an image diagram of object posture learning. The object posture may be, for example, an object standing, an object not touching the floor, an object lying down (also referred to as "lying down" in the embodiment), or an object lying down and not touching the ground. The edge feature quantities in this case include, for example, the object quaternion, the contact point and normal in the object coordinate system, and the gravitational acceleration.

In this embodiment, the results of this learning are used as shown in FIG.
FIG. 11 is a diagram for explaining an example of using the learning result according to this embodiment.
As a result of learning using a scene graph, the mathematical model can be used as follows. The relationship between the object and the hand is, for example, "the direction is that the hand is on top of or beside the object" and "the posture is that the object is standing." In this case, the grasping information is to place a capsule, which is a grasping candidate area, on the lid or side. Also, the relationship between the object and the hand is, for example, "the direction is that the hand is on top of or beside the object" and "the posture is that the object is lying down." In this case, the grasping information is to place a capsule on the side.

<Scene graph of hand-object relationships>
Next, the relationship between the hand and the object can be expressed in a scene graph as shown in Figure 12. Figure 12 is a diagram showing a scene of the relationship between the hand and the object and a simple graph representation. Note that the simple graph representation in Figure 12 shows a simplified version of the link arrows between the nodes in Figure 9. In this embodiment, such a scene graph is called a pose scene graph. Note that the configuration of the pose graph is such that the nodes are, for example, the coordinates of the object as seen from the hand, and the edges are the coordinates of the relative pose between the hand and the object to be grasped, the relative pose between the object to be grasped and the surrounding environment, and the relative pose between the object to be grasped and the floor.

<Wrist Posture Estimation Processing>
Next, an outline of the wrist posture estimation process will be described. Fig. 13 is a diagram showing an outline of the wrist posture estimation process. As shown in Fig. 13, in this embodiment, the above-mentioned pose scene graph and cost scene graph are used to estimate the wrist posture.

Now, let's explain the cost scene graph. Figure 14 is a diagram for explaining the cost scene graph. Image g351 shows a state in which the hand is on top of an object (bowl). Image g352 shows a state in which the hand is on the side of an object (bottle). Image g353 is a graph representation. In such cases, the relationship between the hand and the object requires other information (hand position, finger angle, hand pose, friction, etc.). As a result, multiple states may be possible for a single node, as shown in image g354 in image g253. In this embodiment, a cost is assigned for each of these multiple states. The cost is calculated using the Quality Measure (minimum sum of squares of lateral forces) value around the fingertip contact point. The cost graph is constructed by multiplying the points that make up the center line by the wrist pose, and the edges are coordinates obtained by adding the relative pose of the grasp center with respect to the skeleton, the convolution of the difference between the object surface curve around the skeleton and the finger curve, and the Quality Measure value. Note that nodes are, for example, positions that can be grasped with the hand, or positions that can be grasped. The number of nodes is the number of points that make up the center line multiplied by the number of possible wrist poses. The reason for this is that even if the hand and target are positioned on an object and the grasping point is fixed, the way the object is grasped will change depending on the wrist posture.

In this embodiment, the pose graph is used to create the relative pose between the hand and the object to be grasped, or the relative pose between the object to be grasped and the surrounding environment, or the relative pose between the object to be grasped and the floor surface.
Next, in this embodiment, a cost graph is used to create a scene graph in which edges are assigned costs based on the relative pose of the grasp center with respect to the skeleton, the convolution of the difference between the object (or surface, curved surface) around the skeleton and the finger curved surface (whether or not a finger is near the object, and if so, which of the multiple points constituting the center line should be used as the grasp representative point), and the Quality Measure value.
Furthermore, in this embodiment, the wrist posture is determined by folding the edge costs and finding the node that can have the minimum cost.
In addition, in this embodiment, training data, which is the result of actually operating the end effector and gripping a target object, is stored in a database.

In this embodiment, the system uses a pose scene graph and a cost scene graph to determine the grip style online, and includes a network layer (model) that uses accumulated training data as correct answer data and includes attention nodes that are associated with the correct answer data.

During operation, the actual current wrist posture can be input into this network (a trained model) to estimate the wrist posture during grasping.

<Example of end effector configuration>
Fig. 15 is a diagram showing an example of the configuration of an end effector that performs work according to this embodiment. As shown in Fig. 15, end effector 1 (hand) includes finger portions 101, 102, 103, and 104, and a base 111. End effector 1 is connected via a joint to arm 121, which is a movement mechanism that can move the position of end effector 1.

Furthermore, finger unit 101 is equipped with a force sensor 141, for example, at its fingertip. Finger unit 102 is equipped with a force sensor 142, for example, at its fingertip. Finger unit 103 is equipped with a force sensor 143, for example, at its fingertip. Finger unit 104 is equipped with a force sensor 144, for example, at its fingertip. Finger unit 101 corresponds to, for example, a human thumb, finger unit 102 corresponds to, for example, a human index finger, finger unit 103 corresponds to, for example, a human middle finger, and finger unit 104 corresponds to, for example, a human ring finger.
The end effector 1 has at least two fingers, but may have three or more fingers.

The arm 121 is also a mechanism that can change the wrist position of the end effector 1 .
16, the end effector 1 and the arm 121 are controlled by a control device 200. Fig. 16 is a diagram showing an example of the configuration of the control device according to this embodiment.

The control device 2 is connected to an imaging device 5 (sensor), an instruction device 7, an end effector 1, and an arm 121 by wire or wirelessly.
The end effector 1 includes fingers 101 to 104 (FIG. 15) and a base 111 (FIG. 15), as well as an actuator 161 and a sensor 151.
The arm 121 includes an actuator 171 and a sensor 181 .

The indicator device 7 includes, for example, a sensor 71 and a communication unit 72.

The control device 2 includes an acquisition unit 21 , a control unit 22 , an end effector driving unit 23 , an arm driving unit 24 , and a wrist posture determination device 3 .
The wrist posture determination device 3 includes a center line creation unit 31, a grasp representative determination unit 32, a learning unit 33, a storage unit 34, a determination unit 37, and a taxonomy determination unit 38. The storage unit 33 stores a model 35 and training data 36.

The instruction device 7 is a data glove that is worn by the worker. The instruction device 7 may include, for example, an HMD (head mounted display) that detects the worker's line of sight.
The sensor 71 is, for example, a six-axis sensor, a gyro sensor, a pressure sensor, etc. The sensor 71 detects at least the positions of the fingers and the wrist and their trajectories.
The communication unit 72 transmits the sensor value detected by the sensor 71 to the control device 2 .

The image capture device 5 is an RGB-D camera that is also capable of depth measurement. The image capture device 5 determines the position of the target object using the captured image. The image capture device 5 uses the captured image to determine the position of the environment, such as the floor or ground, where the target object is located.

The actuators 161 are provided, for example, on each of the fingers 101 to 103 of the end effector and on each of the joints.
The sensor 151 is, for example, a six-axis sensor, a position sensor, a pressure sensor, or the like.

The actuator 171 is provided, for example, at a connection portion with the end effector 1 .
The sensor 181 is, for example, a six-axis sensor, a position sensor, a pressure sensor, or the like.

The acquisition unit 21 acquires images captured by the imaging device 5. The acquisition unit 21 acquires sensor values detected by the sensor 151. The acquisition unit 21 acquires sensor values detected by the sensor 181. The acquisition unit 21 acquires the sensor values from the indicator device 7.

The control unit 22 generates an arm drive command using the information acquired by the acquisition unit 21 and the information determined by the wrist posture determination device 3. The control unit 22 generates an end effector drive command using the information acquired by the acquisition unit 21.

The end effector driving unit 23 controls the end effector 1 based on the end effector driving command generated by the control unit 22.

The arm driving unit 24 controls the arm 121 based on the arm driving command generated by the control unit 22.

Next, the wrist posture determination device 3 will be described.
The center line creating unit 31 performs image processing on the captured image to create a center line that indicates the center and includes at least one point.

The grasp representative determination unit 32 determines the grasp representative point using the trained model 35.

During learning, the learning unit 33 generates a pose scene graph and a cost scene graph as described above. During learning, the learning unit 33 learns a model 35 using training data 36. The learning unit 33 stores the learned model 35 in the memory unit 34.

The determination unit 37 acquires the actual wrist posture from the sensor 181 during work. The determination unit 37 inputs the acquired actual wrist posture into the trained model 35 to determine the wrist posture online.

The taxonomy determination unit 38 estimates the position, orientation, etc. of the target object based on the captured image. The taxonomy determination unit 38 determines the taxonomy based on the estimation results.

<Example of processing procedure>
Next, an example of the processing procedure performed by the wrist posture determination device 3 during learning will be described. Fig. 17 is a flowchart of the processing procedure performed by the wrist posture determination device during learning according to this embodiment.

(Step S101) The imaging device 5 captures an image of the target object.

(Step S102) The image capture device 5 determines the position of the target object and the position of the environment.

(Step S103) The learning unit 33 creates a pose scene graph using, for example, the coordinates of the object as seen from the hand as nodes, and the coordinates of "the relative pose between the hand and the grasped object, the relative pose between the grasped object and the surrounding environment, and the relative pose between the grasped object and the floor" as edges.

(Step S104) The learning unit 33 creates a cost scene graph using the points that make up the center line multiplied by the wrist pose for the nodes, and the edges using the coordinates of "the relative pose of the grasp center with respect to the skeleton, the convolution of the difference between the surface curve of the object around the skeleton and the finger curve, and the Quality Measure value."

(Step S105) The learning unit 33 uses the training data 36, the pose scene graph, and the cost scene graph to train the model 35, and stores the trained model 35 in the memory unit 34.

Next, we will explain an example of the processing procedure performed by the online wrist posture determination device 3 during work. Figure 18 is a flowchart of the processing procedure performed by the wrist posture determination device during work according to this embodiment.

(Step S201) The imaging device 5 captures an image of the target object.

(Step S202) The taxonomy determination unit 38 estimates the position, orientation, etc. of the target object based on the captured image. The taxonomy determination unit 38 determines a taxonomy based on the estimation results.

(Step S203) The center line creation unit 31 performs image processing on the captured image to create a center line that indicates the center and includes at least one point. Note that the wrist posture determination device 3 performs the processes of steps S201 to S203 at the start of grasping.

(Step S204) During work, the actual wrist posture is acquired from sensor 181.

(Step S205) The grasp representative point determination unit 32 determines the grasp representative point for the unknown object online by inputting the actual current wrist posture into the learned model 35.

(Step S205) The determination unit 37 determines the wrist posture of the hand with respect to the unknown object so that the target object is grasped with the determined grasp representative point as the grasp center. The determination unit 37 determines the wrist posture online, for example, by inputting the scene graph, cost scene graph, and actual current wrist posture into the trained model 35. The determination unit 37 determines the wrist posture by folding the edges of the two scene graphs around a node and finding the node that can provide the minimum cost.

As described above, in this embodiment, a wrist posture that can be processed online is determined online by inputting the scene graph, cost scene graph, and actual current wrist posture into the trained model 35. When model 35 is used, the grasp representative point is determined by model 35, and the optimal wrist posture is then determined using model 35 from among the postures that the wrist of the end effector can take as it approaches the target object.

In this embodiment, the position is determined based on the geometric structure of the object (mainly skeleton information of unknown polygons), the taxonomy grasp center, the fingertip curvature during grasp, the Quality Measure (minimum sum of squared lateral forces) around the fingertip contact point, and a scene graph of the hand, object, and surrounding environment. Furthermore, in this embodiment, a pose graph is created based on the relative pose between the hand and the grasped object, the relative pose between the grasped object and the surrounding environment, and the relative pose between the grasped object and the floor.

In this embodiment, a cost scene graph is created using the relative pose of the grasp center (grasp representative point) relative to the skeleton, the convolution of the difference between the surface curve of the object around the skeleton and the finger curve, and the Quality Measure value as the cost. The relative pose of the grasped object and the surrounding environment is calculated by determining a provisional wrist posture and calculating the difference in distance between the finger and the provisional grasp center (grasp representative point) based on the provisional grasp center between the fingers. The wrist posture determination device 3 calculates the cost for each of multiple points on the center line. The wrist posture determination device 3 also convolves multiple differences between the curved surface of the object surface around the center line of the object and the curved surface of the fingers of the end effector 1 to generate a single numerical value so that friction occurs. When the grasp center is near the center line, it is necessary to consider the balance of the total lateral friction force at the contact point. Therefore, the wrist posture determination device 3 calculates this as the Quality Measure value and uses it as the cost. In this embodiment, the wrist posture is determined by folding the edges of the two scene graphs around a node to find the node with the lowest possible cost.

As a result, this embodiment allows the wrist posture to be determined online. Furthermore, this embodiment allows the most appropriate conditions to be selected from multiple constraint conditions, thereby determining the method of movement of the wrist of the end effector 1. As a result, this embodiment makes it easier for processing to converge.

Note that while an example of the model 35 used to determine the grasp representative point and wrist posture has been described using both a pose scene graph and a cost scene graph, this is not limiting. Depending on the shape of the target object, the posture of the target object, the current posture of the wrist, etc., the model 35 may be a pose scene graph only.

<Attitude determination method without using a model>
In the above example, the wrist posture is determined using the model 35, but the present invention is not limited to this. Fig. 19 is a diagram for explaining a method of determining the wrist posture without using a model.

19, image g300 is an example in which the end effector 1 and the wrist are positioned above the target object obj and are then moved closer to the target object obj. Image g410 is an example in which the end effector 1 and the wrist are positioned diagonally to the upper right of the target object obj and are then moved closer to the target object obj.
When the model 35 is not used, the wrist posture determination device 3 selects one of the multiple points constituting the center line from the multiple selected operation shapes (possible postures of the wrist of the end effector approaching the target object) according to the taxonomy and determines it as the grasp representative point. Then, the wrist posture is tilted within an angle range that prevents the end effector 1 from colliding with the floor surface on which the target object obj is placed. Alternatively, the range within which the grasp center can be tilted is determined. In FIG. 19 , symbols g401 and g4111 indicate the grasp centers of the target object. Symbols g402 and g412 indicate the grasp center of the end effector 1. The grasp center of the end effector 1 is calculated based on the sensor value detected by the sensor 151 provided in the end effector 1.

In addition, a program for implementing some or all of the functions of the control device 200 and wrist posture determination device 3 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the control device 200 and wrist posture determination device 3. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. It also includes a WWW system equipped with a homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves within the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded on the computer system.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1... end effector, 101, 102, 103, 104... finger portion, 111... base body, 121... arm, 5... imaging device, 7... instruction device, 151... sensor, 161... actuator, 171... actuator, 181... sensor, 71... sensor, 72... communication unit,
21...acquisition unit, 22...control unit, 23...end effector driving unit, 24...arm driving unit, 3...wrist posture determination device, 31...center line creation unit, 32...grasp representative determination unit, 33...learning unit, 34...storage unit, 37...determination unit, 38...taxonomy determination unit, 35...model, 36...training data

Claims (5)

1. A method for determining an attitude of a moving mechanism that is capable of gripping and manipulating a target object , is connected to an end effector having a plurality of operation attitudes, and is capable of moving the position of the end effector, comprising:
a measuring step of measuring the target object by a sensor;
a calculation step of calculating a plurality of points indicating the center of the measured object;
a grip representative point determination step of determining one of the plurality of points as a grip representative point corresponding to one of the plurality of selected operation shapes;
and a posture determination step of determining a posture of the moving mechanism from the grip representative point ,
The posture determining step determines the grip representative point using a model that has been trained in advance using a pose scene graph that represents the relationship between the end effector and the target object, the relationship between the target object and objects in the periphery of the target object, and the posture of the target object using nodes and edges.
Posture determination method. the attitude determination step determines a taxonomy related to grasping the target object based on the position of the target object and the attitude of the moving mechanism, and determines the attitude of the moving mechanism based on the taxonomy.
2. The method of claim 1. The nodes of the pose scene graph are based on the coordinates of the target object as seen from the end effector,
The edges of the pose scene graph are costs based on the relationship between the end effector and the target object , the relationship between the target object and objects in the vicinity of the target object, and the posture of the target object.
2. The method of claim 1 . The posture determination step determines the grip representative point using a model that has been trained in advance using a cost scene graph that represents the relative pose between the end effector and the target object , the target object and a plurality of points indicating the center of the target object, and possible postures of the moving mechanism using nodes and edges.
2. The method of claim 1 . The cost scene graph has, as costs on edges, information based on a relative pose of a grip center with respect to a plurality of points indicating the center of the target object, information based on a difference between a curved surface of the target object around the plurality of points indicating the center of the target object and a curved surface of a finger unit of the end effector, and a Quality Measure value.
5. The method of claim 4 .