JP7845261B2 - Rotation state derivation device and robot system - Google Patents

Description

This disclosure relates to a rotational state derivation device and a robotic system.

Generally, when deriving the rotational state of a sphere, as disclosed in Patent Document 1, marks on the sphere are extracted as feature parts from each of the multiple image data of the sphere detected by a sensor, and the rotational state of the sphere is derived based on the time-series displacement of these feature parts.

Japanese Patent Publication No. 2018-4638

The applicant has identified the following problem: When a sphere approaches a sensor, the area of the sphere that can be detected by the sensor narrows. For example, a characteristic feature of the sphere may fall outside the sensor's detection range, making it difficult to accurately determine the sphere's rotational state.

This disclosure was made in view of the above problems, and aims to realize a rotation state derivation device and robot system capable of accurately deriving the rotation state of a sphere.

A rotation state derivation device according to one aspect of the present disclosure is a rotation state derivation device that derives the rotation state of a sphere based on image data detected by a sensor on the sphere,
If the distance between the sensor fixed to the moving body and the sphere is greater than or equal to a preset first threshold, the feature portion of the sphere is extracted, and the rotational state of the sphere is derived based on the time-series displacement of the extracted feature portion.
If the distance between the sensor and the sphere is less than the first threshold, the orientation of the sphere at the current time is estimated based on data indicating the rotation state of the sphere derived in the past, and the rotation state of the sphere is derived.

According to this disclosure, a rotation state derivation device and robot system capable of accurately determining the rotation state of a sphere can be realized.

This is a block diagram showing the configuration of the robot system according to Embodiment 1. This is a diagram illustrating the detection range of the sensor in the robot system of Embodiment 1. This is a flowchart showing the processing flow of the rotation state derivation device in the robot system of Embodiment 1. (a) is a diagram illustrating the detection range of the sensor when the fingertips of the robot's hand are touching the basketball, (b) is a diagram showing image data detected by the sensor when the fingertips of the robot's hand are touching the basketball, (c) is a diagram illustrating the detection range of the sensor when the robot's hand is grasping the basketball, and (d) is a diagram showing image data detected by the sensor when the robot's hand is grasping the basketball. This diagram illustrates the normal process for deriving the rotation state. This is a flowchart illustrating the control flow when a robot dribbles a basketball.

The following describes specific embodiments applying this disclosure, with reference to the drawings. However, this disclosure is not limited to the following embodiments. Furthermore, the following description and drawings have been simplified as appropriate.

<Embodiment 1>
First, the configuration of the robot system of this embodiment will be described. Figure 1 is a block diagram showing the configuration of the robot system of this embodiment. As shown in Figure 1, the robot system 1 of this embodiment includes a robot 2, a sensor 3, a rotation state derivation device 4, and a database (DB) 5. These robot 2, sensor 3, rotation state derivation device 4, and DB 5 are connected via a network 6. Here, the network 6 is a wired or wireless communication line, for example, the internet.

Robot 2 can be composed of, for example, a general humanoid robot. That is, although detailed illustrations are omitted, robot 2 comprises a head, torso, arms, hands, and legs. Here, the arms and hands of robot 2 form a robotic arm, and the hands of robot 2 form the hand portion of the robotic arm.

Robot 2 is configured to perform predetermined actions by controlling actuators 2a, located at the joints of its head, torso, arms, hands, and legs, via a control unit 2b. For example, Robot 2 may be configured to perform actions that mimic those of a basketball player (such as grasping a basketball, dribbling, and free throws).

Sensor 3 can be configured, for example, as a three-dimensional distance measuring sensor. Figure 2 is a diagram illustrating the detection range (i.e., field of view) of the sensor in the robot system of this embodiment. Here, in Figure 2, the detection range R of sensor 3 is indicated by a dashed line.

The sensor 3 is fixed, for example, to the base of the hand portion 2d of the robot arm 2c, that is, to the carpal portion of the hand of the robot 2, as shown in Figure 2, and has a detection range R on the palm side.

The rotation state derivation device 4, as detailed later, derives the rotation state of the basketball 10 (for example, the amount and direction of rotation of the basketball 10) based on image data detected by the sensor 3, as shown in Figure 2. DB 5 stores the image data and data indicating the rotation state of the basketball 10 that has been previously derived.

Next, the processing flow of the rotation state derivation device 4 in the robot system 1 of this embodiment will be explained. Figure 3 is a flowchart showing the processing flow of the rotation state derivation device in the robot system of this embodiment. In this embodiment, the rotation state of the basketball 10 is derived.

First, the rotation state derivation device 4 acquires image data of the basketball 10 detected by the sensor 3 (S1). Then, based on this image data, the rotation state derivation device 4 calculates the distance to the basketball 10 (i.e., the distance between the sensor 3 and the basketball 10) and acquires the amount of rotation of the basketball 10 from the DB 5, which was previously derived from the data indicating the rotation state of the basketball 10.

Next, the rotation state derivation device 4 determines whether the distance to the basketball 10 is greater than or equal to the first threshold, and whether the amount of rotation of the basketball 10 is less than the second threshold (S2). Here, Figure 4(a) is a diagram illustrating the sensor's detection range when the fingertips of the robot's hand are touching the basketball; Figure 4(b) shows image data detected by the sensor when the fingertips of the robot's hand are touching the basketball; Figure 4(c) is a diagram illustrating the sensor's detection range when the robot's hand is grasping the basketball; and Figure 4(d) shows image data detected by the sensor when the robot's hand is grasping the basketball.

As shown in Figure 2, when the distance to the basketball 10 is greater than or equal to the first threshold, and the basketball 10 is substantially within the detection range R of the sensor 3, the sensor 3 can detect substantially the entire basketball 10. Therefore, the displacement of the basketball's pattern (i.e., its characteristic features) can be detected effectively.

On the other hand, as shown in Figures 4(a) and 4(c), when the distance to the basketball 10 is less than the first threshold, meaning that substantially the entire basketball 10 is not within the detection range R of the sensor 3, the sensor 3 cannot detect substantially the entire basketball 10, as shown in Figures 4(b) and 4(d). Therefore, it may not be possible to accurately detect the displacement of the pattern on the basketball 10.

Furthermore, for example, if the pattern on a basketball 10 has a regularity such as symmetry, it may be difficult to clearly recognize the displacement between the pattern of the basketball 10 shown in the image data previously detected by sensor 3 and the pattern of the basketball 10 shown in the image data currently detected by sensor 3. If the rotation amount of the basketball 10 exceeds the second threshold, it may be difficult to accurately detect the displacement between the patterns.

Therefore, if the distance to the basketball 10 is greater than or equal to the first threshold, and the amount of rotation of the basketball 10 is less than the second threshold (YES in S2), the rotation state derivation device 4 performs the normal rotation state derivation process (S3). Figure 5 is a diagram illustrating the flow of the normal rotation state derivation process.

In detail, first, the rotation state derivation device 4 acquires three-dimensional point cloud data based on image data detected by the sensor 3, as shown in Figure 5(a). Then, the rotation state derivation device 4 uses a method such as the RANSAC (Random Sample Consensus) algorithm to acquire point cloud data indicating the spherical shape of the basketball 10 and its position (i.e., coordinates) within the environment, as shown in Figure 5(b).

Next, the rotation state derivation device 4 acquires a two-dimensional image as shown in Figure 5(c) based on the image data detected by the sensor 3. Then, the rotation state derivation device 4 performs feature extraction from the two-dimensional image using techniques such as SIFT (Scale-Invariant Feature Transform) or FAST (Features from Accelerated Segment Test) to acquire the pattern of the basketball 10 as shown in Figure 5(d).

Next, the rotation state derivation device 4 extracts point cloud data corresponding to the pattern on the basketball 10, and derives the rotation state of the basketball 10 based on the displacement of the point cloud data corresponding to the pattern on the basketball 10 acquired this time (as shown in Figure 5(f)) relative to the point cloud data corresponding to the pattern on the basketball 10 acquired previously (as shown in Figure 5(e)).

On the other hand, if the distance to the basketball 10 is greater than or equal to the first threshold, and the amount of rotation of the basketball 10 is not less than the second threshold (NO in S2), that is, if the distance to the basketball 10 is less than the first threshold, or the amount of rotation of the basketball 10 is greater than or equal to the second threshold, the rotation state derivation device 4 does not perform the normal rotation state derivation process, but instead estimates the current posture of the basketball 10 based on data indicating the rotation state of the basketball 10 derived in the past, and derives the rotation state of the basketball 10 (S4).

In detail, for example, the rotation state derivation device 4 acquires data indicating the rotation state of the basketball 10, which was derived from DB5 in the previous and the time before that. Based on this data, it estimates the current orientation of the basketball 10 and generates point cloud data for that orientation.

The rotation state derivation device 4 then extracts point cloud data (second point cloud data) corresponding to the region of point cloud data (first point cloud data) based on image data (first point cloud data) as shown in Figure 4(b) or Figure 4(d) where the basketball 10 was detected by sensor 3, from the estimated point cloud data of the basketball 10's orientation at the current time. By matching the second point cloud data with the first point cloud data, the rotation state of the basketball 10 is derived.

In this manner, if the distance to basketball 10 is less than the first threshold, or if the amount of rotation of basketball 10 is greater than or equal to the second threshold, the current orientation of basketball 10 is estimated based on previously derived data indicating the rotation state of basketball 10, and the rotation state of basketball 10 is derived.

In other words, this method derives the rotational state of the basketball 10 at the current time, without relying solely on image data detected by the sensor 3. Therefore, it is possible to accurately derive the rotational state of the basketball 10. Here, matching can be performed using methods such as ICP (Iterative Closest Point).

Next, based on the data indicating the rotation state of the basketball 10 derived as described above, we will explain the control flow when the robot 2 dribbles the basketball 10. Figure 6 is a flowchart showing the control flow when the robot dribbles the basketball.

First, the control unit 2b of robot 2 acquires data from DB5 indicating the current rotation state of basketball 10 (S11). Then, the control unit 2b of robot 2 determines whether the amount of topspin of basketball 10 included in the data indicating the current rotation state of basketball 10 is within a preset rotation range (S12).

If the amount of topspin in the basketball 10 is within the specified rotation range (YES in S12), the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 to perform a normal dribble without adding topspin or backspin to the rotation of the basketball 10 (S13). In other words, the control unit 2b of the robot 2 provides basic control of each actuator 2a.

On the other hand, if the amount of topspin of the basketball 10 is not within the rotation range, that is, if the amount of topspin of the basketball 10 is outside the rotation range (NO in S12), the control unit 2b of the robot 2 determines whether the amount of topspin of the basketball 10 is greater than the rotation range (S14).

If the amount of topspin on the basketball 10 exceeds the specified rotation range (YES in S14), the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 so that backspin is added to the rotation of the basketball 10 (S15).

On the other hand, if the amount of topspin of the basketball 10 is not greater than the rotation range (NO in S14), that is, if the amount of topspin of the basketball 10 is less than the rotation range, the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 so that topspin is added to the rotation of the basketball 10 (S16).

In this embodiment, when the robot 2 dribbles the basketball 10, the basketball 10 approaches the sensor 3, and the distance to the basketball 10 falls below the first threshold. However, based on the data indicating the current rotation state of the basketball 10, which has been accurately derived as described above, the robot 2 can be made to dribble the basketball 10. Therefore, the robot 2 can dribble the basketball 10 with high accuracy.

Thus, in this embodiment, the rotation state derivation device 4 estimates the current orientation of the basketball 10 based on previously derived data indicating the rotation state of the basketball 10, and derives the rotation state of the basketball 10, when the distance to the basketball 10 is less than the first threshold, or the amount of rotation of the basketball 10 is greater than or equal to the second threshold.

In other words, this method derives the rotational state of the basketball 10 at the current time, without relying solely on the image data detected by sensor 3. Therefore, it is possible to accurately derive the rotational state of the basketball 10.

Furthermore, the robot system 1 of this embodiment causes the robot 2 to dribble the basketball 10 based on the data indicating the rotational state of the basketball 10 at the current time, as derived as described above. Therefore, the robot 2 can dribble the basketball 10 with high precision.

Here, it is advisable to supervise the robot 2's operation of each actuator 2a when it makes a free throw of the basketball 10, based on the rotational state of the basketball 10 when it is placed on the hand part 2d for a free throw (i.e., the arrangement of the pattern when the basketball 10 is stationary).

This allows the actuators 2a of the robot 2 to be operated according to the rotational state of the basketball 10 when it is placed on the hand unit 2d, enabling the robot 2 to throw free throws with high precision towards the hoop using the basketball 10.

<Embodiment 2>
In Embodiment 1, when the amount of rotation of the basketball 10 is greater than or equal to a second threshold, the current posture of the basketball 10 is estimated based on data indicating the rotation state of the basketball 10 derived in the past, and the rotation state of the basketball 10 is derived. However, for example, the rotation state of the basketball 10 may be derived as follows.

For example, similar to the normal rotation state derivation process, the rotation state derivation device 4 matches the point cloud data corresponding to the pattern of the basketball 10 acquired in the previous step with the point cloud data corresponding to the pattern of the basketball 10 acquired in the current step. Then, by removing point clouds that successfully matched, it eliminates point cloud data corresponding to patterns with regularity.

In other words, the rotation state derivation device 4 extracts point cloud data corresponding to unique patterns, such as characters, from the point cloud data of the basketball 10's pattern acquired in the previous and current sessions. This data is obtained based on the success rate of matching point cloud data corresponding to regular patterns.

The rotation state derivation device 4 then derives the rotation state of the basketball 10 based on the displacement of the point cloud data corresponding to the unique patterns, such as letters, on the basketball 10, which was acquired in the previous session, and the point cloud data corresponding to the unique patterns, such as letters, on the basketball 10, which was acquired in the current session.

In this way, by excluding the point cloud data corresponding to the regular pattern of the basketball 10 and using the point cloud data corresponding to unique patterns such as letters, the rotation state of the basketball 10 can be derived accurately even if a regular pattern like the basketball 10 exists.

Here, while methods such as ICP are used for point cloud data matching, the computational load on the rotation state derivation device 4 becomes high when the amount of data is large. Therefore, it is advisable to downsample during matching. In this case, it is good practice to vary the downsampling amount based on the matching process.

For example, when extracting point cloud data of a basketball 10 based on image data detected by sensor 3, and matching the point cloud data based on previously detected image data with the point cloud data based on the currently detected image data, the downsampling amount should be set to the first reduction amount.

Furthermore, when matching point cloud data corresponding to a pattern with regularity, the downsampling amount should be set to a second reduction amount, which is smaller than the first reduction amount. When matching point cloud data corresponding to a unique pattern, the downsampling amount should be set to a third reduction amount, which is smaller than the second reduction amount (for example, no downsampling). This reduces the computational burden on the rotation state derivation device 4.

<Other Embodiments>
In the embodiments described above, the disclosure was explained as a hardware configuration, but the disclosure is not limited thereto. The disclosure can also be implemented by having a CPU (Central Processing Unit) execute a program for each process.

Here, a program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions. A program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs), or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray® discs, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. A program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable mediums or communication media that include electrically, optically, acoustically, or otherwise propagating signals.

This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from the spirit of the invention.
For example, in the above embodiment, since the sensor 3 is fixed to the hand portion 2d of the robot 2, the position and orientation of the sensor 3 can be derived based on the rotation angles of each joint of the robot 2. However, if the position and orientation of the sensor 3 cannot be derived, the position and orientation of the sensor 3 can be derived based on the point cloud data of the environmental portion other than the basketball 10 shown in the image data detected by the sensor 3.
For example, in the above embodiment, the rotational state of the basketball 10 was derived, but any sphere would suffice. Also, in the above embodiment, the sensor 3 is fixed to the robot 2, but the sensor 3 can be fixed to any moving object.
For example, in the above embodiment, if the amount of rotation of the basketball 10 is greater than or equal to the second threshold, the rotation state derivation device 4 estimates the current orientation of the basketball 10 based on data indicating the rotation state of the basketball 10 derived in the past and derives the rotation state of the basketball 10, or derives the rotation state of the basketball 10 based on the displacement of point cloud data of parts corresponding to unique patterns such as letters on the basketball 10. However, at least if the distance to the basketball 10 is greater than or equal to the first threshold, it is sufficient to estimate the current orientation of the basketball 10 based on data indicating the rotation state of the basketball 10 derived in the past and derive the rotation state of the basketball 10.

1 Robot system 2 Robot, 2a Actuator, 2b Control unit, 2c Robot arm, 2d Hand unit 3 Sensor 4 Rotation state extraction device 5 Database 6 Network 10 Basketball® Sensor detection range

Claims (5)

A rotation state derivation device that derives the rotation state of a sphere based on image data detected by a sensor on the sphere,
If the distance between the sensor fixed to the moving body and the sphere is greater than or equal to a preset first threshold, the feature portion of the sphere is extracted, and the rotational state of the sphere is derived based on the time-series displacement of the extracted feature portion.
A rotation state derivation device that, when the distance between the sensor and the sphere is less than the first threshold, estimates the orientation of the sphere at the current time based on data indicating the rotation state of the sphere derived in the past, and derives the rotation state of the sphere. The rotation state derivation device according to claim 1, wherein, if the amount of rotation of the sphere included in the previously derived data indicating the rotation state of the sphere is greater than or equal to a preset second threshold, the device estimates the orientation of the sphere at the current time based on the previously derived data indicating the rotation state of the sphere and derives the rotation state of the sphere. The rotation state derivation device according to claim 1, wherein if the amount of rotation of the sphere included in the previously derived data indicating the rotation state of the sphere is greater than or equal to a preset second threshold, the device extracts the characteristic parts of the sphere, excludes the characteristic parts having regularity from the extracted characteristic parts, and derives the rotation state of the sphere based on the time-series displacement of the remaining characteristic parts. The aforementioned sensor is a three-dimensional distance measuring sensor.
A rotation state derivation device according to any one of claims 1 to 3, wherein, when the distance between the sensor and the sphere is less than the first threshold, a second point cloud data of the region corresponding to the first point cloud data from the estimated point cloud data of the sphere's orientation at the current time is matched with a first point cloud data based on image data of the sphere detected by the sensor, and the rotation state of the sphere is derived. A rotation state derivation device according to claim 1 or 2,
A robot with a robotic arm,
A sensor fixed to the base of the hand portion of the robot arm,
A control unit that controls the robot arm based on the rotation state of the sphere derived from image data of the sphere detected by the sensor,
Equipped with,
If the amount of top spin of the sphere included in the data indicating the rotation state of the sphere is greater than a preset rotation range, the control unit controls the robot arm so that back spin is added to the sphere.
A robot system in which, if the amount of top spin of the sphere included in the data indicating the rotation state of the sphere is less than the rotation range, the control unit controls the robot arm so that top spin is added to the sphere.