JP7660387B2 - Object-to-robot pose estimation from a single RGB image - Google Patents

Description

This application is a continuation-in-part of U.S. patent application Ser. No. 16/405,662 (Reference: 741851/18-RE-0161-US02) entitled "DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL" filed on May 7, 2019, and is a continuation-in-part of U.S. patent application Ser. No. 16/405,662 (Reference: 741851/18-RE-0161-US02) entitled "DETECTION AND POSE ESTIMATION OF HOUSEHOLD OBJECTS FOR HUMAN-ROBOT" filed on May 17, 2018. This application claims priority to U.S. Provisional Patent Application No. 62/672,767 (Reference: 18-RE-0161US01), entitled "INTERACTION," which is incorporated herein by reference in its entirety.

This application is a continuation-in-part of U.S. Patent Application No. 16/657,220 (Reference: 1R2674.006901/19-SE-0341US01), entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS," filed on October 18, 2019, and is hereby incorporated by reference in its entirety.

This disclosure relates to a system and method for pose estimation.

Pose estimation generally refers to a computer vision technique that determines the Euclidean position and orientation of an object, usually relative to a particular camera. Pose estimation has many applications, but is particularly useful in the context of robotic manipulation systems. Until now, robotic manipulation systems have required a camera mounted on the robot itself (i.e., a handheld camera) to capture images of the object, and/or a camera external to the robot to capture images of the object. In both cases, the camera must be calibrated to the robot in order to then estimate the pose of the captured object relative to the robot.

Although the calibration of a local camera only needs to be performed once to determine the camera's position relative to the robot, due to the rigid mounting of the camera on the robot, this camera has a limited field of view, which prevents it from seeing the surrounding context and being easily adjustable when an object moves out of the camera's field of view. An external camera can have a wider field of view and can optionally assist the local camera, but it needs to be calibrated every time it is moved even slightly. However, calibration is typically a tedious, delicate, and error-prone offline process. Note that while these issues are described with particular reference to robotic manipulation systems, the same limitations apply to any pose estimation system operable to estimate the pose of one object relative to another object (i.e., which may or may not be moving).

There is a need to address these and/or other problems associated with the prior art.

A method, computer readable medium, and system are disclosed for object-to-object pose estimation from images. In use, images of a first object and a target object are identified, and the images are captured by a camera external to the first object and the target object. Additionally, the images are processed using a first neural network to estimate a first pose of the target object relative to the camera. Furthermore, the images are processed using a second neural network to estimate a second pose of the first object relative to the camera. Furthermore, a third pose of the first object relative to the target object is calculated using the first pose and the second pose.

1 is a diagram of a method for object-to-object pose estimation from images, according to one embodiment. 1 is a diagram of a system for object-to-object pose estimation from images, according to one embodiment. FIG. 3 is a block diagram associated with the first neural network of FIG. 2, according to one embodiment. FIG. 3 is a block diagram associated with the second neural network of FIG. 2, according to one embodiment. FIG. 1 is a diagram of a robotic grasping system controlled using a robot-to-object pose estimation system, according to one embodiment. FIG. 1 illustrates inference and/or training logic, according to at least one embodiment. FIG. 1 illustrates inference and/or training logic, according to at least one embodiment. FIG. 1 illustrates training and deployment of a neural network, according to at least one embodiment. FIG. 1 illustrates an exemplary data center system, according to at least one embodiment.

FIG. 1 illustrates a method 100 for object-to-object pose estimation from images, according to one embodiment. Method 100 may be performed by any computing system, which may include one or more computing devices, one or more computer processors, non-transitory memory, circuitry, etc. In one embodiment, the non-transitory memory may store instructions executable by one or more computing devices and/or one or more computer processors to perform method 100. In another embodiment, a circuit may be configured to perform method 100.

As shown in operation 102, images of the first object and the target object are identified and the images are captured by cameras external to the first object and the target object. In the context of this description, the target object and the first object are physically separate objects (e.g., in some proximity to each other). In one embodiment, the first object may be a robotic gripping system having a robotic arm for gripping the object. In furtherance of this embodiment, the target object may be a known object that is gripped by the robotic gripping system. In another embodiment, the first object may be another autonomous object, such as an autonomous vehicle, that interacts with (or makes decisions related to) the target object, such as another car. However, of course, the first object and the target object may be any object for which a bilateral pose is desirable (e.g., for computer vision applications).

As described above, a camera external to the first object and the second object captures images of the first object and the target object. For purposes of this description, the camera is external to the first object and the target object by being positioned independent of the first object and the target object. For example, the camera may not be mounted on either the first object or the target object. In one embodiment, the camera may be mounted on a tripod or other rigid surface to capture images of the first object and the target object.

The image of the target object may be a red-green-blue (RGB) image in one embodiment, or a grayscale image in another embodiment. The image may or may not include depth. In another embodiment, the image may be a single image. In a further embodiment, the image may be a combination of images obtained from various types of sensors, such as a six-channel image that may include red, green, blue, infrared, ultraviolet, and radar. Furthermore, the camera may be a monocular RGB camera. However, of course, the camera may capture any number of channels of any wavelength of light (visible or invisible to humans), and even non-light wavelengths. For example, the camera may utilize infrared, ultraviolet, microwave, radar, sonar, or other technologies to capture the image.

Additionally, as shown in operation 104, the image is processed using a first neural network to estimate a first pose of the target object relative to the camera. The first neural network may be trained to output two-dimensional (2D) image locations (x, y coordinates) of key points of the target object. These 2D image locations may then be used to estimate the pose of the target object relative to the camera along with the 3D coordinates of the target object model. In one embodiment, a PnP algorithm may be used to calculate the pose of the target object relative to the camera. In another embodiment, the first pose of the target object relative to the camera may include a three-dimensional (3D) rotation and translation of the target object relative to the camera.

By way of example, the first neural network may be one disclosed in U.S. Patent Application No. 16/405,662, entitled "DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL," filed May 7, 2019 (Reference: 741851/18-RE-0161-US02), which is incorporated by reference herein in its entirety. Further details relating to an embodiment of the first neural network are described below with reference to FIG. 3.

Further, as shown in operation 106, the image is processed using a second neural network to estimate a second pose of the first object relative to the camera. The second neural network may be trained to output two-dimensional (2D) image locations (x, y coordinates) of key points of the first object. These 2D image locations may then be used to estimate the pose of the first object relative to the camera along with the 3D coordinates of the first object model. In one embodiment, a perspective-n-point (PnP) algorithm may be used to calculate the pose of the first object relative to the camera. In another embodiment, the second pose of the first object relative to the camera may include a 3D rotation and translation of the first object relative to the camera.

As an option, the second neural network can also perform online calibration of the camera, allowing the camera to be moved during run-time, including during operation of, for example, a robotic gripping system or other autonomous system.

By way of example, the second neural network may be one disclosed in U.S. Patent Application No. 16/657,220, entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS," filed on October 18, 2019 (Reference: 1R2674.006901/19-SE-0341US01), which is incorporated by reference in its entirety. Further details relating to an embodiment of the second neural network are described below with reference to FIG. 4.

Still further, as shown in operation 108, a third pose of the first object relative to the target object is calculated using the first pose and the second pose. Thus, a pose estimate from the first object to the target object may be calculated. In one embodiment, the third pose may be the pose of the target object relative to the coordinate frame of the first object. In another embodiment, the third pose may be calculated by multiplying the inverse of an output of one of the neural networks by an output of another one of the neural networks, such as the inverse first pose multiplied by the second pose, or the inverse of the second pose multiplied by the first pose, etc.

In the context where the first object is a robotic gripping system, the robotic gripping system may further be caused to grip a target object (i.e., a known object) using a third pose. For example, a third pose may be provided to the robotic gripping system to enable the robotic gripping system to locate and grip the target object. Of course, in other embodiments, the third pose may be used for other purposes, including, for example, controlling an autonomous vehicle or having an autonomous vehicle make decisions based on the relative location of the target object.

As an option, the first pose and/or the second pose can be refined prior to calculating the third pose. For example, the first pose can be refined by iteratively matching the image with a synthetic projection of the model in the first pose and then adjusting parameters of the first pose based on the results of the iterative matching. Similarly, the second pose can be refined by iteratively matching the image with a synthetic projection of the model in the second pose and then adjusting parameters of the second pose based on the results of the iterative matching.

For this purpose, the pose between the first object and the target object may be estimated using two neural networks. In particular, one neural network may estimate a first pose of the first object relative to the camera, and a second neural network may estimate a second pose of the first object relative to the camera. The outputs of the neural networks (i.e., the first pose and the second pose) may then be used as a basis for determining the pose between the first object and the target object.

Next, further explanatory information is described in relation to various optional architectures and features that can implement the aforementioned framework as desired by a user. It is particularly noted that the following information is described for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.

FIG. 2 illustrates a system 200 for object-to-object pose estimation from images, according to one embodiment. For example, system 200 may implement method 100 of FIG. 1. System 200, in one embodiment, may be located in the cloud and thus remote with respect to the object. In another embodiment, system 200 may be located on a computing device local to the object.

As shown, system 200 includes a first module 201 including a first neural network 202, a second module 203 including a second neural network 204, and a processor 206. Images are input to first neural network 202 and second neural network 204. The images are of a first object and a target object and are captured by a camera external to both the first object and the target object. The camera may be external to system 200. However, the images may be provided to system 200 by the camera, via a network, shared memory, or in any other manner.

The first neural network 202 receives an image as input and processes it to estimate a first pose of the target object relative to the camera (i.e., the pose from the target object to the camera). The first neural network 202 may be a deep neural network that estimates the 6DoF pose (e.g., 3D rotation and translation) of a known object relative to the camera. This network 202 may consist of a multi-stage convolutional network that converts the input RGB image into a set of belief maps, one for each key point. In one embodiment, n=9 key points are used to represent the vertices of a bounding cuboid along with the centroids. In addition to the belief maps, the network 202 may output n-1 affinity maps, one for each key point that is not a centroid. Each map is a 2D field of unit vectors pointing to the nearest object centroid. The maps can be used to individuate objects in a post-processing step, allowing the system to handle multiple instances of each object. The pose can be determined by a first module 201 that applies the PnP algorithm to key points detected as peaks in the belief map.

In one embodiment, the input to network 202 may consist of a 533x400 image processed by a VGG-based feature extractor, resulting in a 50x50x512 feature block. These features may be processed by a series of six stages (each containing seven convolutional layers) that output and refine the belief map described above.

The second neural network 204 receives an image as input and processes the image to estimate a second pose of the first object relative to the camera (i.e., the pose from the first object to the camera). In one embodiment, the second neural network 204 may be a deep neural network that estimates the 6DoF pose of the robot relative to the camera. This network 204 may consist of an encoder-decoder that converts the input RGB image into a set of belief maps, one for each key point. Since there is only one robot in a scene, an affinity field may not be required.

In one embodiment, key points placed on the robot joints may be defined to achieve pose stability when the arm is mostly outside the camera's field of view, which occurs when the camera is viewing the scene from a close distance. In one embodiment, the input to the network 204 is a 400x400 image, downsampled and center cropped from the original 640x480. The layers of the network 204 may be as follows: the encoder follows the same layer structure as VGG, while the decoder is built from 4 upsampling layers followed by 2 convolution layers to give a belief map of key points. The second module 203 applies the PnP algorithm to the key points detected as peaks in the belief map.

As an option, the first pose and/or the second pose can be refined to reduce errors in the estimation due to limited training data and/or network capacity. This refinement can be performed by adjusting the pose parameters by iteratively matching the input image with a synthetic projection of the model with the current pose.

Since the first neural network 202 and the second neural network 204 do not require each other's outputs, the first neural network 202 and the second neural network 204 may or may not operate in parallel if desired. The outputs of each of the first neural network 202 and the second neural network 204 are provided to a processor 206. The processor 206 uses the first pose and the second pose to calculate a third pose of the first object relative to the target object. The third pose may be calculated using Equation 1 described according to one embodiment.

Where:

is the pose of the object in the robot frame,

is the pose of the robot in the camera frame (computed by the second network 204),

is the pose of the object in the camera frame (computed by the first network 202).

In an exemplary embodiment where the first object is a robotic grasping system and the target object is a known object, the processor 206 can now use the third pose to cause the robotic grasping system to grasp the target object. The processor 206 can communicate with the robotic grasping system over a network, such as the Internet (e.g., if the system 200 is located remotely relative to the object) or Ethernet (e.g., if the system 200 is located locally relative to the object).

Because the first neural network 202 only estimates the pose of the target object relative to the camera, only the first neural network 202 (and not necessarily the second neural network 204) may be trained on the target object. In this manner, the target object may be "known" to the first neural network 202, or in other words, a "known object." Of course, the first neural network 202 may also be trained on other categories or types of target objects.

Similarly, only the second neural network 204 (and not necessarily the first neural network 202) may be trained on the first object, since the second neural network 204 only estimates the pose of the first object relative to the camera. In other words, the first object may be "known" to the second neural network 204. However, it should be noted that the second neural network 204 may also be trained on other objects (e.g., other robots, or other autonomous objects).

To this end, in contrast to end-to-end learning, the modular approach presented in the present system 200 allows the system to be reused without the need to retrain all networks. For example, to apply the system 200 to a new robot, only the second network 204 needs to be retrained. Similarly, to apply the system 200 to a new object, only the first network 202 needs to be trained on this object, independent of the robot and other objects. Moreover, this modular approach allows for easier testing and refinement of the individual components to ensure accuracy and reliability.

3 is a block diagram associated with the first neural network 202 of FIG. 2 according to one embodiment. Of course, the block diagram is described as one possible embodiment associated with the first neural network 202 of FIG. 2. This embodiment is described in more detail in U.S. Patent Application No. 16/405,662, entitled "DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL," filed May 7, 2019 (Reference: 741851/18-RE-0161-US02), which is incorporated herein by reference in its entirety.

In the illustrated embodiment, the pose estimation system 300 includes a key point module 310, a set of multi-stage modules 305, and a pose unit 320. Although the pose estimation system 300 is described in the context of a processing unit, one or more of the key point module 310, the set of multi-stage modules 305, and the pose unit 320 may be implemented by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the key point module 310 may be implemented by a GPU, a central processing unit (CPU), or any processor capable of processing an image to generate key point data.

The pose estimation system 300 receives an image captured by a single camera. The image may contain one or more objects for detection. In one embodiment, in images without any depth data, the image contains color data per pixel. The pose estimation system 300 first detects key points associated with the object, and then estimates 2D projections of vertices that define a bounding volume surrounding the object associated with the key points. The key points may include the center of gravity of the object and the vertices of the bounding volume surrounding the object. The key points are not explicitly visible in the image, but are instead inferred by the pose estimation system 300. In other words, the object of interest is visible in the image except for parts of the object that may be occluded, and the key points associated with the object of interest are not explicitly visible in the image. The 2D locations of the key points are estimated by the pose estimation system 300 using only the image data. The pose unit 320 recovers the 3D pose of the object using the estimated 2D locations, the camera intrinsic parameters, and the object dimensions.

The key point module 310 receives an image containing an object and outputs image features. In one embodiment, the key point module 310 includes a multi-layer convolutional neural network (i.e., the first network 202). In one embodiment, the key point module 310 includes the first 10 layers of a Visual Geometry Group (VGG-19) neural network pre-trained using the ImageNet training database, followed by two 3x3 convolution layers to reduce the feature dimensionality from 512 to 256, and then from 256 to 128. The key point module 310 outputs three channel features, one per channel (e.g., RGB).

The image features are input to a set of multi-stage modules 305. In one embodiment, the set of multi-stage modules 305 includes a first multi-stage module 305 configured to detect the center of gravity of the object, and additional multi-stage modules 305 configured in parallel to detect vertices of a bounding volume surrounding the object. In one embodiment, the set of multi-stage modules 305 includes a single multi-stage module 305 that is used to process the image features in multiple passes to sequentially detect the center of gravity and vertices of the bounding volume. In one embodiment, the multi-stage module 305 is configured to detect vertices without detecting the center of gravity.

Each multi-stage module 305 includes T stages of belief map units 315. In one embodiment, the number of stages is equal to six (e.g., T=6). Belief map unit 315-1 is the first stage, belief map unit 315-2 is the second stage, and so on. Image features extracted by the key point module 310 are passed to each of the belief map units 315 in the multi-stage module 305. In one embodiment, the key point module 310 and the multi-stage module 305 include a feedforward neural network (i.e., the first neural network 202) that receives an RGB image of size wxhx3 as input and branches to generate two different outputs, such as belief maps. In one embodiment, w=640 and h=480. The stages of the Belief Map Unit 315 operate sequentially, with each stage (Belief Map Unit 315) taking into account not only image features but also the output of the previous stage.

The stages of the belief map unit 315 in each multi-stage module 305 generate belief maps for the estimation of a single 2D location associated with an object in an image. A first belief map contains probability values for the object's centroid, and an additional belief map contains probability values for the vertices of a bounding volume that encloses the object.

In one embodiment, the 2D locations of the detected vertices are the 2D coordinates of the 3D bounding vertices that each enclose the object and are projected into image space for the scene. Representing each object by a 3D bounding box defines an abstract representation of each object that is sufficient for pose estimation, yet is independent of the details of the object's shape. When the bounding volumes are 3D bounding boxes, nine multi-stage modules 305 can be used to generate belief maps for the centroid and eight vertices in parallel. The pose unit 325 estimates the 2D coordinates of the 3D bounding box vertices projected into image space, and then infers object locations and poses in 3D space from the perspective-n-point (PnP) using either traditional computer vision algorithms or another neural network. PnP estimates the pose of an object using a set of n locations in 3D space and a projection of the n locations in image space. In one embodiment, the pose estimation system 300 estimates the 3D pose of a known object in clutter from a single RGB image in real time.

In one embodiment, each stage of the Belief Map Unit 315 is a stage of a Convolutional Neural Network (CNN). When each stage is a CNN, each stage exploits an increasingly larger effective receptive field as data passes through the neural network. This property allows the stages of the Belief Map Unit 315 to resolve ambiguities by incorporating an increasingly larger amount of context in later stages.

In one embodiment, the stages of belief map units 315 receive the 128-dimensional features extracted by the key point module 310. In one embodiment, belief map unit 315-1 includes three 3x3x128 layers and one 1x1x512 layer. In one embodiment, belief map unit 315-2 is a 1x1x9 layer. In one embodiment, belief map units 315-3 through 315-T are identical to the first stage except that each receives a 153-dimensional input (128+16+9=153) and includes five 7x7x128 layers and one 1x1x128 layer before the 1x1x128 or 1x1x16 layers. In one embodiment, each of the belief map units 315 is of size _w/8 and _h/8 and is globally interleaved with rectified linear unit (ReLU) activation functions.

FIG. 4 is a block diagram associated with the use of the second neural network 204 of FIG. 2, according to one embodiment. Of course, the block diagram is described as one possible embodiment associated with the second neural network 204 of FIG. 2. This embodiment of the second neural network 204 is described in more detail in U.S. Patent Application No. 16/657,220, entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS," filed on October 18, 2019 (Reference: 1R2674.006901/19-SE-0341US01), which is incorporated herein by reference in its entirety.

As shown, a captured image 402 of a robot is provided as an input to the trained neural network 204. It should be noted that while a robot is described in the context of this example, the block diagrams described herein are equally applicable to other objects (e.g., other autonomous systems). As an option, some pre-processing or enhancement of the image may be performed prior to processing, such as to adjust resolution, color depth, or contrast. In at least one example, the network 204 may be trained specifically for the type of robot 204, since different robots may have different shapes, sizes, configurations, kinematics, and characteristics.

In use, the neural network 204 may analyze the input images 402 and output a set of belief maps 406 as a set of inferences. As an option, other dimension determining inferences may be generated to locate feature points. For example, the neural network 204 may infer one belief map 406 for each robot feature identified.

In at least one embodiment, a model of the robot used for training can identify specific features to be tracked. These features can be learned through a training process. In another embodiment, the features can be located on various moving parts or components of the robot such that a pose of the robot can be determined from the features. In a further embodiment, the features can be selected such that each pose of the robot corresponds to one and only one setting of the features, and each setting of the features corresponds to one and only one robot pose. This uniqueness can allow the camera-to-robot pose to be determined based on the unique orientation of the features as represented in the captured image data.

Regarding network 204, an autoencoder network can detect key points. In at least one embodiment, neural network 204 receives as input an RGB image of size wxhx3 and outputs n belief maps 406 of the form wxhxn. In at least one embodiment, RGBD or stereoscopic images can be received as input as well. Optionally, w=640 and h=480. In at least one embodiment, the output for each key point is a 2D belief map, with pixel values representing the likelihood that the key point is projected onto that pixel.

In one embodiment, the encoder of network 204 includes a VGG-19 convolutional layer pre-trained with ImageNet. In another embodiment, a ResNet-based encoder can be used. In a further embodiment, the decoder or upsampling component of network 204 is composed of four 2D transposed convolutional layers, each followed by a regular 3x3 convolutional layer and a ReLU activation function. Still further, the output head can be composed of three convolutional layers (3x3, stride=1, padding=1) with ReLU activation on 64, 32, and n channels, respectively. In at least one embodiment, there may be no activation layer after the last convolutional layer. In another embodiment, the encoder network can be trained using an L2 loss function that compares the output belief map to a ground truth belief map, where the ground truth belief map is generated using σ=2 pixels to generate peaks. In at least one embodiment, the use of stereoscopic image pairs can allow fusing the poses estimated by these images, or a point cloud can be computed and the poses determined using processes such as Procrustes analysis or iterative closest point (ICP).

In at least one embodiment, the belief maps 406 may be provided as input to a peak extraction component 408, or service, that may determine a set of coordinates in two dimensions that represent the locations of relevant robot features. As an option, key point coordinates may be calculated in each belief map as a weighted average of values around thresholded peaks, after first applying Gaussian smoothing to these belief maps to reduce noise effects. This weighted average may allow for sub-pixel accuracy. In at least one embodiment, these two-dimensional coordinates (or pixel locations) may be provided as input to a pose determination module, such as a perspective-n-point (PnP) module 414. This pose determination module may accept as input camera-specific data 410, such as calibration information for the camera, that may be used to account for image artifacts due to lens asymmetry, focal length, principal point, or other such factors.

In at least one embodiment, the pose determination module can also receive as input information about the forward kinematics 412 of this type of robot to determine possible poses. The kinematics can be used to narrow the search space where only certain feature locations are possible due to the physical configuration or limitations of this type of robot. This information can be analyzed using a PnP algorithm to output a determined camera-to-robot pose. In at least one embodiment, perspective-n-point is used to recover the camera extrinsic factors, assuming that the joint configuration of the robot manipulator is known. Since the base coordinates or other features of the robot can be precisely specified in camera space or camera coordinate system, the pose information can be used to determine the relative distance and orientation between the camera and the robot. To this end, the neural network 204 can be trained to infer the location of these features, such as by inferring a set of belief maps 406.

In at least one embodiment, the relative position and orientation information can be used to ensure that the camera coordinate space from the camera's viewpoint is aligned with the robot coordinate space of the robot, both in terms of dimensions and alignment. In at least one embodiment, these coordinates may be correct from the camera coordinate system but not the robot coordinate system, so that an inaccurate orientation or position of the camera 502 with respect to the robot 504 may give incorrect coordinates for the robot 504 to perform an action. To this end, the relative position and orientation of the camera 502 can be determined with respect to the robot 504. In at least one embodiment, while the relative position of the camera 502 may be sufficient, orientation information may be useful depending on factors such as camera specifics, where asymmetric image properties may affect accuracy if not properly accounted for. This relative position/orientation can be calibrated online (e.g., during the robot's runtime).

Figure 5 illustrates a robotic gripping system 500 controlled using a robot-to-object pose estimation system, according to one embodiment. The robotic gripping system 500 can be implemented in the context of the previously described embodiments.

As shown, the camera 502 can be used to capture images, potentially in the form of video frames, of an autonomous object such as a robot 504. The camera 502 can be positioned or externally mounted such that the robot 504 is within the field of view 510 of the camera 502 and such that the camera 502 can capture images that may include at least a partial representation of the robot 504 if not a full view representation. The captured images can be used to assist in providing instructions to the robot 504 to perform a particular task.

In at least one embodiment, the captured images can be analyzed to determine the location of the object 512 relative to the robot 504, which the robot 504 uses to interact with, such as lifting or modifying the object 512 in some way. In particular, the system 200 of FIG. 2 can be utilized to determine a robot-to-object pose for the purpose of providing precise instructions to the robot 504 or a control system for the robot 504. The robot-to-object pose can also be used for other purposes, such as to help navigate the robot 504 or provide current information about the state of the robot 504. In at least one embodiment, precise position and orientation data between the robot 504 and the object 512 allows the robot 504 to operate robustly in unstructured and dynamic environments while performing tasks such as object grasping and manipulation, human-robot interaction, and collision detection and avoidance. It is therefore desirable to determine at least one of the position or orientation of the robot 504 relative to the camera 502 and the position or orientation of the object 512 relative to the camera 502, and then determine the robot-to-object pose.

As described above, images may be captured by the camera 502 showing the current orientation of the robot 504 relative to the camera 502 and the current orientation of the object 512 relative to the camera 502. The robot 504 may have different articulated limbs 508 or components such that the robot 504 may be in different configurations or "poses" with respect to the orientation of the robot 504. In at least one embodiment, different poses of the robot 504 may result in different expressions in the images captured by the camera 502. In at least one embodiment, a single image captured by the camera 502 may be analyzed to determine features of the robot 504 that can be used to determine the pose of the robot 504. The features may correspond to joints or locations where the robot can move or adjust its position or orientation. Furthermore, because the dimensions and kinematics of the robot 504 are known, determining the pose of the robot 504 from the perspective of the camera 502 allows for accurate determination of the distance and orientation of the robot from the camera.

Machine Learning Deep Neural Networks (DNNs), which involve deep learning models developed on processors, are used in a variety of use cases, from self-driving cars to rapid drug discovery, from automatic image captioning in online image databases to smart real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, and it continuously learns and gets smarter, providing more accurate results in a faster time. Children are initially taught by adults to correctly identify and classify various shapes, and eventually become able to identify shapes without any coaching. Similarly, deep learning or neural learning systems need to be trained on object recognition and classification to smarter and more efficiently identify basic objects, occluded objects, etc., while still assigning context to the objects.

At the simplest level, neurons in the human brain look at the various inputs they receive and assign a level of importance to each of these inputs, with the output being passed on to other neurons to act upon. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is trained to recognize and classify, and each of these features is assigned a certain weight based on the importance of that feature in defining the object's shape.

Deep Neural Network (DNN) models contain multiple layers of many connected nodes (e.g., perceptrons, Boltzmann machines, radius-based functions, convolutional layers, etc.) that can be trained with vast amounts of input data to quickly solve complex problems with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer puts together the lines and looks for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate labels for the input image and identify the model of a specific car brand.

Once a DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (the process by which a DNN extracts useful information from given inputs) include identifying handwritten digits on a check deposited in an ATM machine, identifying images of friends in a photograph, providing movie recommendations to over 50 million users, identifying and classifying different types of vehicles, pedestrians, and road hazards in driverless cars, or translating human speech in real time.

During training, data is passed through the DNN in a forward propagation phase until a prediction is generated that indicates the label that corresponds to the input. If the neural network does not label the input correctly, the error between the correct label and the predicted label is analyzed, and during the backpropagation phase, the weights for each feature are adjusted until the DNN correctly labels that input and other inputs in the training data set. Training a complex neural network requires a large amount of parallel computing performance, including floating-point multiplications and additions. Inferring is a less computationally intensive, latency-sensitive process in which a trained neural network is applied to new, previously unseen inputs for image classification, speech translation, and general inference about new information.

Inference and Training Logic As mentioned above, a deep learning or neural learning system needs to be trained to generate inferences from input data. Details regarding the inference and/or training logic 615 for a deep learning or neural learning system are provided below in conjunction with Figures 6A and/or 6B.

In at least one embodiment, the inference and/or training logic 615 may include, without limitation, data storage 601 for storing forward propagation and/or output weights and/or input/output data corresponding to neurons or layers of a neural network trained and/or used to infer in one or more aspects of the embodiment. In at least one embodiment, the data storage 601 stores weight parameters and/or input/output data for each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the data storage 601 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 caches of a processor, or system memory.

In at least one embodiment, any portion of data storage 601 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, data storage 601 may be cache memory, dynamic random addressable memory ("DRAM"), static random addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether data storage 601 is internal or external to the processor, for example, or whether it is comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and/or inference functions being performed, the batch size of data used in the inference and/or training of the neural network, or any combination of these factors.

In at least one embodiment, the inference and/or training logic 615 may include, without limitation, data storage 605 for storing backpropagated and/or output weights and/or input/output data corresponding to neurons or layers of a neural network trained and/or used to infer in one or more aspects of the embodiment. In at least one embodiment, the data storage 605 stores weight parameters and/or input/output data for each layer of a neural network trained or used in conjunction with one or more embodiments while backpropagating input/output data and/or weight parameters during training and/or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the data storage 605 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 caches of a processor, or system memory. In at least one embodiment, any portion of the data storage 605 may be internal or external to one or more processors, or other hardware logic devices or circuits. In at least one embodiment, data storage 605 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether data storage 605 is internal or external to the processor, for example, or whether it is comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and/or inference functions being performed, the batch size of data used in the inference and/or training of the neural network, or any combination of these factors.

In at least one embodiment, data storage 601 and data storage 605 may be separate storage structures. In at least one embodiment, data storage 601 and data storage 605 may be the same storage structure. In at least one embodiment, data storage 601 and data storage 605 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of data storage 601 and data storage 605 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache, or system memory.

In at least one embodiment, the inference and/or training logic 615 may include, without limitation, one or more arithmetic logic units ("ALUs") 610 for performing logical and/or arithmetic operations based at least in part on or indicated by the training and/or inference code, which may result in activations (e.g., output values from layers or neurons in a neural network) that are stored in activation storage 620 and are a function of input/output and/or weight parameter data stored in data storage 601 and/or data storage 605. In at least one embodiment, the activations stored in activation storage 620 are generated according to linear algebra and/or matrix-based calculations performed by ALU 610 in response to executing instructions or other code, where the weight values stored in data storage 605 and/or data 601 are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in data storage 605 or data storage 601, or in another storage, on-chip or off-chip. In at least one embodiment, ALU 610 is contained within one or more processors or other hardware logic devices or circuits, although in other embodiments ALU 610 may be external to the processors or other hardware logic devices or circuits that use them (e.g., a co-processor). In at least one embodiment, ALU 610 may be included within an execution unit of a processor, or may be otherwise included within an ALU bank accessible by execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing unit, graphics processing unit, fixed function unit, etc.). In at least one embodiment, data storage 601, data storage 605, and activation storage 620 may be in the same processor or other hardware logic devices or circuits, and in other embodiments, they may be in different processors or other hardware logic devices or circuits, or some combination of the same processor or other hardware logic devices or circuits and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 620 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 cache of the processor, or system memory. Additionally, the inference and/or training code may be stored with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and/or processed using the processor's fetch, decode, schedule, execute, retire, and/or other logic circuitry.

In at least one embodiment, activation storage 620 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 620 may be fully or partially internal to one or more processors or other logic circuits, or may be external. In at least one embodiment, the choice of whether activation storage 620 is internal or external to a processor, for example, or whether it is comprised of DRAM, SRAM, flash, or some other type of storage, may depend on available on-chip versus off-chip storage, latency requirements of the training and/or inference functions being performed, batch sizes of data used in inference and/or training of the neural network, or any combination of these factors. In at least one embodiment, the inference and/or training logic 615 shown in FIG. 6A may be used in conjunction with an application-specific integrated circuit ("ASIC"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and/or training logic 615 shown in FIG. 6A may be used in conjunction with other hardware, such as central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or field programmable gate array ("FPGA").

6B illustrates inference and/or training logic 615 according to at least one embodiment. In at least one embodiment, the inference and/or training logic 615 may include, without limitation, hardware logic in which computational resources are dedicated to or otherwise used only in conjunction with weight values or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and/or training logic 615 illustrated in FIG. 6B may be used in conjunction with an application specific integrated circuit (ASIC), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corporation. In at least one embodiment, the inference and/or training logic 615 shown in FIG. 6B may be used in conjunction with other hardware, such as central processing unit (CPU) hardware, graphics processing unit ("GPU") hardware, or field programmable gate arrays (FPGAs). In at least one embodiment, the inference and/or training logic 615 may include, without limitation, data storage 601 and data storage 605, which may be used to store weight values and/or other information, including bias values, gradient information, momentum values, and/or other parameters or hyperparameter information. In at least one embodiment shown in FIG. 6B, each of data storage 601 and data storage 605 is associated with dedicated computational resources, such as computation hardware 602 and computation hardware 606, respectively. In at least one embodiment, each of computation hardware 606 includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in data storage 601 and data storage 605, respectively, with the results stored in activation storage 620.

In at least one embodiment, each of the data storages 601 and 605 and the corresponding computing hardware 602 and 606 correspond to a different layer of the neural network, such that activations resulting from one "storage/computation pair 601/602" of data storage 601 and computing hardware 602 are provided as inputs to the next "storage/computation pair 605/606" of data storage 605 and computing hardware 606 to reflect the conceptual organization of the neural network. In at least one embodiment, the storage/computation pairs 601/602 and 605/606 may correspond to two or more layers of the neural network. In at least one embodiment, additional storage/computation pairs (not shown) may be included in the inference and/or training logic 615 after or in parallel with the storage/computation pairs 601/602 and 605/606.

Training and Deployment of a Neural Network FIG. 7 illustrates another embodiment for training and deployment of a deep neural network. In at least one embodiment, an untrained neural network 706 is trained using a training data set 702. In at least one embodiment, the training framework 704 is the PyTorch framework, while in other embodiments, the training framework 704 is Tensorflow, Boost, Caffe, Microsoft Cognitive Toolkit/CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 704 trains the untrained neural network 706 and enables it to be trained using processing resources described herein to generate a trained neural network 708. In at least one embodiment, the weights may be selected randomly or by pre-training using a deep belief network. In at least one embodiment, training may be performed in either a supervised, semi-supervised, or unsupervised manner.

In at least one embodiment, the untrained neural network 706 is trained using supervised learning, where the training data set 702 includes inputs paired with desired outputs for the inputs, or the training data set 702 includes inputs with known outputs, and the outputs of the neural network 706 are manually scored. In at least one embodiment, the untrained neural network 706 is trained in a supervised manner, processing inputs from the training data set 702 and comparing the resulting outputs to a set of expected or desired outputs. In at least one embodiment, errors are then back-propagated through the untrained neural network 706. In at least one embodiment, the training framework 704 adjusts weights that control the untrained neural network 706. In at least one embodiment, the training framework 704 includes tools to monitor how well the untrained neural network 706 is converging toward a model, such as the trained neural network 708, that is suitable for generating the correct answer, such as in the results 714, based on known input data, such as the new data 712. In at least one embodiment, the training framework 704 iteratively trains the untrained neural network 706 while adjusting the weights to refine the output of the untrained neural network 706 using a loss function and a tuning algorithm such as stochastic gradient descent. In at least one embodiment, the training framework 704 trains the untrained neural network 706 until it reaches a desired accuracy. In at least one embodiment, the trained neural network 708 can then be deployed to implement any number of machine learning operations.

In at least one embodiment, the untrained neural network 706 is trained using unsupervised learning, where the untrained neural network 706 attempts to train itself using unlabeled data. In at least one embodiment, the training data set 702 for unsupervised learning includes input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 706 can learn groupings within the training data set 702 and determine how individual inputs relate to the untrained data set 702. In at least one embodiment, unsupervised training can be used to generate a self-organizing map, which is a type of trained neural network 708 that can perform operations useful for reducing the dimensionality of the new data 712. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows for identifying data points in the new data set 712 that deviate from the normal patterns of the new data set 712.

In at least one embodiment, semi-supervised learning may be used, which is a technique in which labeled and unlabeled data are mixed in the training data set 702. In at least one embodiment, the training framework 704 may be used to perform incremental learning, such as by transfer learning techniques. In at least one embodiment, incremental learning allows the trained neural network 708 to adapt to new data 712 without forgetting the knowledge that was instilled in the network during initial training.

8 illustrates an exemplary data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

In at least one embodiment, as shown in FIG. 8, data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources ("node C.R.") 816(1)-816(N), where "N" represents any positive integer. In at least one embodiment, node C.R. 816(1)-816(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input/output ("NW I/O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules. In at least one embodiment, one or more of the nodes C.R. 816(1)-816(N) may be a server having one or more of the computing resources described above.

In at least one embodiment, the grouped computing resources 814 may include separate groups of nodes C.R. housed in one or more racks (not shown), or multiple racks housed in a data center in various graphical locations (also not shown). The separate groups of nodes C.R. in the grouped computing resources 814 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several nodes C.R. including CPUs or processors may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

In at least one embodiment, the resource orchestrator 822 may configure or otherwise control one or more nodes C.R. 816(1)-816(N) and/or grouped computing resources 814. In at least one embodiment, the resource orchestrator 822 may include a software design infrastructure ("SDI") management entity for the data center 800. In at least one embodiment, the resource orchestrator may include hardware, software, or some combination thereof.

In at least one embodiment shown in FIG. 8, framework layer 820 includes a job scheduler 832, a configuration manager 834, a resource manager 836, and a distributed file system 838. In at least one embodiment, framework layer 820 may include a framework for supporting software 832 in software layer 830 and/or one or more applications 842 in application layer 840. In at least one embodiment, software 832 or application 842 may each include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 820 may be, but is not limited to, a type of free and open source software web application framework, such as Apache Spark® (hereinafter “Spark”), which can use distributed file system 838 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 832 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 800. In at least one embodiment, the configuration manager 834 may be capable of configuring different tiers, such as the software tier 830 and the framework tier 820, which includes Spark and a distributed file system 838 to support large scale data processing. In at least one embodiment, the resource manager 836 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support the distributed file system 838 and the job scheduler 832. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 814 in the data center infrastructure tier 810. In at least one embodiment, the resource manager 836 may work in conjunction with the resource orchestrator 812 to manage these mapped or allocated computing resources.

In at least one embodiment, the software 832 included in the software layer 830 may include software used by at least a portion of the nodes C.R. 816(1)-816(N), the grouped computing resources 814, and/or the distributed file system 838 of the framework layer 820. The one or more types of software may include, but are not limited to, Internet web page searching software, e-mail virus scanning software, database software, and streaming video content software.

In at least one embodiment, the applications 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the nodes C.R. 816(1)-816(N), the grouped computing resources 814, and/or the distributed file system 838 of the framework layer 820. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, Tensorflow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager 834, resource manager 836, and resource orchestrator 812 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may enable data center operators of data center 800 to avoid determining potentially bad configurations and eliminate underutilized and/or underperforming portions of the data center.

In at least one embodiment, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 800. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using weight parameters calculated by one or more techniques described herein.

In at least one embodiment, the data center may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and/or inference using the resources described above. Additionally, one or more of the software and/or hardware resources described above may be configured as a service to enable a user to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

Inference and/or training logic 615 is used to perform inference and/or training operations associated with one or more embodiments. In at least one embodiment, inference and/or training logic 615 may be used in the system of FIG. 8 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and/or architecture, or neural network use cases described herein.

As described herein, a method, computer readable medium, and system for estimating object-to-object pose using externally captured images of the object are disclosed. According to FIGS. 1-4, an embodiment may provide a neural network usable for performing motion inference and providing inferred data, as depicted in FIGS. 6A and 6B, the neural network is stored (partially or entirely) in one or both of data storages 601 and 605 in inference and/or training logic 615. Training and deployment of the neural network may be performed as depicted in FIG. 7 and described herein. Distribution of the neural network may be performed using one or more servers of a data center 800, as depicted in FIG. 8 and described herein.

Claims (19)

identifying an image of a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system, the image being captured by a camera not attached to the robotic gripping system and the target object, but external to the robotic gripping system and the target object;
processing images of the identified target object using a first neural network to estimate a first pose of the target object relative to the camera ;
estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the joints using a second neural network operating in parallel with the first neural network, the second neural network generating a plurality of belief maps that infer, from images of the identified robotic gripping system , positions of each of the plurality of joints that operate to grasp the target object;
and calculating a third pose of the robotic gripping system relative to the target object using the first pose and the second pose. The method of claim 1, wherein the image captured by the camera is a red-green-blue (RGB) image or a grayscale image. The method of claim 1, wherein the camera captures one of optical or non-optical wavelengths. The method of claim 1, wherein the target object is a known object to be grasped by the robotic grasping system. The method of claim 4 , further comprising causing the robotic grasping system to grasp the known object using the third pose. The method of claim 1, wherein the first pose of the target object relative to the camera includes a three-dimensional (3D) rotation and translation of the target object relative to the camera. The method of claim 1, wherein the second pose of the robotic gripping system relative to the camera includes a 3D rotation and translation of the robotic gripping system relative to the camera. The method of claim 1, wherein the second neural network performs an online calibration of the camera. The method of claim 1, wherein the third pose is a pose of the target object relative to a coordinate frame of the robotic gripping system. The method of claim 1, wherein only the first neural network is trained on the target object. The method of claim 1, wherein only the second neural network is trained for the robotic grasping system. refining the first pose;
and refining the second pose. refining the first pose comprises:
13. The method of claim 12, performed by iteratively matching images captured by the camera with a synthetic projection of a model from the first pose, and adjusting parameters of the first pose based on results of the iterative matching. refining the second pose comprises:
13. The method of claim 12, performed by iteratively matching images captured by the camera with a synthetic projection of a model at the second pose, and adjusting parameters of the second pose based on results of the iterative matching. 1. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
identifying an image of a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system, the image being captured by a camera not attached to the robotic gripping system and the target object, but external to the robotic gripping system and the target object;
processing images of the identified target object using a first neural network to estimate a first pose of the target object relative to the camera ;
estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the joints using a second neural network operating in parallel with the first neural network, the second neural network generating a plurality of belief maps that infer, from images of the identified robotic gripping system , positions of each of the plurality of joints that operate to grasp the target object;
and calculating a third pose of the robotic gripping system relative to the target object using the first pose and the second pose. a first neural network that receives as input images identifying the target object from images of the robotic gripping system and the target object captured by a camera that is not attached to a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system and that is external to the robotic gripping system and the target object, and processes the images identifying the target object to estimate a first pose of the target object relative to the camera;
a second neural network operating in parallel with the first neural network, the second neural network receiving as input an image identifying the robotic gripping system from images of the robotic gripping system and the target object, generating from the image identifying the robotic gripping system a number of belief maps that infer positions of each of the plurality of joints that operate to grasp the target object, and estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the plurality of joints ;
and a processor that uses the first pose and the second pose to calculate a third pose of the robotic gripping system relative to the target object. The system of claim 16, wherein the camera is external to the system. The system of claim 16, wherein the target object is a known object. The system of claim 18, wherein the processor causes the robotic grasping system to grasp the target object using the third pose.