JP7606221B2 - Harvesting robot, control method and control program for harvesting robot, and harvesting system - Google Patents

Description

The embodiments disclosed herein relate to a harvesting robot, a control method and control program for the harvesting robot, and a harvesting system for harvesting crops that grow on deciduous trees.

In recent years, harvesting robots that harvest crops such as fruit that grows on trees have become known. For example, Patent Documents 1 and 2 describe a control technology in which a harvesting robot analyzes captured images to estimate the three-dimensional positions of the crops to be harvested and any obstacles that may impede harvesting, and harvests the crops while avoiding contact between the harvesting robot's end effector and the obstacles.

JP 2020-195335 A JP 2010-207118 A

However, the technologies described in Patent Documents 1 and 2 are methods for estimating the position of obstacles based on images captured during harvesting, and because obstacles such as tree branches hidden by leaves are not visible in the captured images, the positions of the obstacles could not be accurately estimated. This has led to a problem in that the harvesting robot's harvesting operations result in components of the harvesting robot, such as the end effector and manipulator, colliding with obstacles whose positions could not be estimated.

The harvesting robot, harvesting robot control method and control program, and harvesting system disclosed in this specification aim to improve the accuracy of estimating the position of obstacles that impede harvesting and to prevent collisions between the harvesting robot and the obstacles.

The disclosed harvesting robot is a harvesting robot that harvests crops that grow on deciduous trees, and includes a memory unit that stores three-dimensional position data during the deciduous season that includes the position of at least a portion of the tree that has lost its leaves, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data related to the position of the crop, and a processing unit that detects the crop to be harvested based on the position data acquired by the sensor unit when the crop is harvested, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the deciduous season, and controls the manipulator when the crop to be harvested is detected so that the picking unit moves to a harvestable position for the detected crop without colliding with at least a portion of the harvesting robot and with the estimated obstacle.

In addition, in the disclosed harvesting robot, when a crop to be harvested is detected, the processing unit preferably acquires three-dimensional position data at the time of harvesting, including the position of at least a portion of an obstacle, based on the position data acquired by the sensor unit, and estimates at least the position of the obstacle that is impeding the movement of the picking unit, based on the three-dimensional position data during the defoliation stage and the three-dimensional position data at the time of harvesting.

In addition, in the disclosed harvesting robot, the sensor unit preferably has a photographing unit that photographs a subject and outputs photographed image data of the subject, and a measuring unit that measures the distance to surrounding objects and outputs distance data indicating the measured distance, and the processing unit preferably calculates three-dimensional position data corresponding to one or more points in the photographed image data based on the distance data corresponding to the output one or more points, and obtains the calculated three-dimensional position data as three-dimensional position data at the time of harvesting.

Furthermore, in the disclosed harvesting robot, it is preferable that the three-dimensional position data during the defoliation period is three-dimensional position data corresponding to one or more points, calculated based on image data captured by the image capture unit during the defoliation period immediately prior to the harvesting of the crop, and distance data corresponding to one or more points in the captured image data output by the measurement unit at the time of capturing the image.

Furthermore, in the disclosed harvesting robot, it is preferable that the processing unit estimates, from the captured image data, an image area showing the tree branches and an image area showing the members supporting the tree branches as obstacle image areas, and obtains, from the calculated three-dimensional position data, three-dimensional position data within the obstacle image areas as three-dimensional position data at the time of harvesting.

The disclosed harvesting robot also preferably includes a cart that moves within a field in which trees are planted, and the processing unit preferably controls the movement of the cart when a crop to be harvested is detected, so that the picking unit can move to a harvestable position for the detected crop without colliding with an estimated obstacle and at least a part of the harvesting robot.

In addition, in the disclosed harvesting robot, it is preferable that the processing unit estimates the position and shape of the obstacle based on the three-dimensional position data during the leaf-shedding period.

The disclosed control method is a control method for a harvesting robot that harvests crops that grow on deciduous trees, and includes a memory unit that stores three-dimensional position data during the deciduous season that includes the position of at least a portion of the tree that has lost its leaves, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data on the position of the crop, and a processing unit, in which the processing unit detects the crop to be harvested based on the position data acquired by the sensor unit when harvesting the crop, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the deciduous season, and controls the manipulator when the crop to be harvested is detected so that the picking unit moves to a harvestable position of the detected crop without colliding with at least a portion of the harvesting robot with the estimated obstacle.

The disclosed control program is for a harvesting robot that harvests crops that grow on deciduous trees, and includes a memory unit that stores three-dimensional position data during the defoliation period that includes the position of at least a portion of the tree that has lost its leaves, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data on the position of the crop, and a processing unit, and causes the processing unit to detect the crop to be harvested based on the position data acquired by the sensor unit when the crop is harvested, estimate at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the defoliation period, and, when the crop to be harvested is detected, control the manipulator so that the picking unit moves to a harvestable position for the detected crop without collision between the estimated obstacle and at least a portion of the harvesting robot.

The disclosed harvesting system includes a harvesting robot that harvests crops that grow on deciduous trees and a server device that communicates with the harvesting robot. The server device includes a memory unit that stores three-dimensional position data during the deciduous season, including the position of at least a portion of the tree that has lost its leaves, and a transmission unit that transmits the three-dimensional position data during the deciduous season to the harvesting robot. The harvesting robot includes a receiving unit that receives the three-dimensional position data during the deciduous season transmitted from the server device, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data on the position of the crop, and a processing unit that detects the crop to be harvested based on the position data acquired by the sensor unit when harvesting the crop, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the received three-dimensional position data during the deciduous season, and controls the manipulator to move the picking unit to a harvestable position for the detected crop when the crop to be harvested is detected, without collision between the estimated obstacle and at least a portion of the harvesting robot.

The disclosed harvesting robot, harvesting robot control method and control program, and harvesting system make it possible to improve the accuracy of estimating the position of obstacles that impede harvesting, and prevent collisions between the harvesting robot and the obstacles.

FIG. 1 is a diagram illustrating an example of a schematic configuration of a harvesting system. FIG. 2A is a perspective view showing an example of the appearance of a harvesting robot, and FIG. 2B is a perspective view showing an example of the appearance of an end effector system. FIG. 1 is a diagram illustrating an example of a schematic configuration of a harvesting robot. FIG. 2 illustrates an example of a schematic configuration of a server device. FIG. 11 is a diagram illustrating an example of an operational flow of a process for extracting obstacle data during the deciduous season. 1A is a diagram showing an example of photographed image data taken in a field where deciduous trees are planted, and FIG. 1B is a schematic diagram showing an example of depth data measured when the photographed image data was taken. 10A is a diagram showing an example of estimated image data indicating an obstacle image region, and FIG. 10B is a schematic diagram showing an example of obstacle data during the leaf-shedding season. FIG. 11 is a diagram showing an example of an operational flow of a harvesting process. FIG. 13 is a diagram illustrating an example of an operation flow of a harvesting obstacle data extraction process. (a) is a diagram showing an example of captured image data of a crop to be harvested at harvest time, (b) is a schematic diagram showing an example of depth data measured when the captured image data was captured, (c) is a diagram showing an example of estimated image data showing an obstacle image area, and (d) is a schematic diagram showing an example of harvest-time obstacle data. FIG. 11 is a diagram showing an example of an operation flow of a harvest control process.

Various embodiments of the present invention will be described below with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions described in the claims and their equivalents.

(Overview of Harvesting System 1)
1 is a diagram showing an example of a schematic configuration of a harvesting system 1. The harvesting system 1 of the present embodiment includes a harvesting robot 2 and a server device 3.

The harvesting robot 2 has various functions for harvesting crops that grow on deciduous trees. Deciduous trees are grown in a specified field. Crops that grow on deciduous trees include cherries, peaches, plums, akebia japonica, blueberries, raspberries, pears, apples, persimmons, grapes, plums, and apricots. Hereinafter, "deciduous trees" may be simply referred to as "trees." The specified field is a field suitable for growing trees, such as an open field, a vinyl greenhouse field, or a pipe greenhouse field.

The server device 3 has a function for managing three-dimensional position data of trees with shed leaves. The three-dimensional position data is data indicating three-dimensional coordinates corresponding to points on the surface of an object such as a tree according to a predetermined position coordinate system. Note that the objects indicated by the three-dimensional position data may include not only "trees" but also objects that obstruct the movement of the harvesting robot 2, the end effector system 22, and the manipulator system 23 (for example, "various members that support the branches (posts (pipes, stakes, etc.), wires, tapes, and/or strings, etc.)").

The predetermined position coordinate system is a geographic coordinate system such as the Japanese Geodetic System 2011, which is indicated by latitude, longitude, and altitude. The predetermined position coordinate system may be a plane rectangular coordinate system or a projected coordinate system of the UTM (Universal Transverse Mercator) coordinate system, which corresponds to a predetermined area including the field in which the trees are planted. The predetermined position coordinate system may also be a coordinate system that has an origin at a predetermined point in the field in which the trees are planted, and has an x-axis parallel to a predetermined direction on a predetermined horizontal plane, a y-axis on the predetermined horizontal plane that is perpendicular to the x-axis, and a z-axis in the vertical direction. The predetermined position coordinate system may also be a world coordinate system that defines the entire three-dimensional virtual space corresponding to the field in which the trees are planted.

The three-dimensional coordinate points indicated by the three-dimensional position data may be points on a part of the surface of a tree. The three-dimensional coordinate points indicated by the three-dimensional position data may be points on the surface of multiple trees, or may be points on a part of the surface of each of the multiple trees. By including multiple three-dimensional coordinate points in the three-dimensional position data, the shape (volume) of the corresponding tree is indicated, and the shape of an obstacle such as a tree can be estimated by generating a surface model described later. Hereinafter, the "three-dimensional position data of a tree that has lost its leaves" may be referred to as "three-dimensional position data in the deciduous period." When a tree has multiple periods in which it loses its leaves, the "deciduous period" of the "three-dimensional position data in the deciduous period" is the period immediately before the harvest of the crops that have borne fruit on the tree, among the multiple periods in which the tree loses its leaves. For example, a tree that has a deciduous period from December to March of the following year goes through the deciduous period every year since planting. In such a case, if the harvesting time by the harvesting system 1 of this embodiment is June, the "deciduous period" of the "deciduous period three-dimensional position data" is the deciduous period from December immediately before June to March (in other words, the "deciduous period three-dimensional position data" of the "deciduous period" one year or more before the harvesting time is not used at the harvesting time). The "deciduous period" of the "deciduous period three-dimensional position data" may be a period when all trees planted in the field have not lost their leaves. For example, the deciduous period may be a period when at least some of the leaves of one or more trees planted in the field have lost their leaves. In addition, the "defoliation" of a tree means "natural defoliation in response to various factors such as temperature and/or hours of sunlight" or "leaves being removed from the branches and trunks of the tree by humans, and/or branches with leaves being removed from the branches and trunks of the tree by humans."

In FIG. 1, one server device 3 is illustrated as a component of the harvesting system 1, but the server device 3 may be a collection of multiple computers, each of which is a physically separate entity. In this case, each of the multiple computers may have the same functions, or may have the functions of the single server device 3 in a distributed manner. In the harvesting system 1, an information processing device such as a personal computer (PC), notebook PC, and/or tablet terminal, which has the same functions as the server device 3, may be used in place of or together with the server device 3.

(Harvesting robot 2)
Figure 2(a) is an oblique view showing an example of the appearance of the harvesting robot 2, Figure 2(b) is an oblique view showing an example of the appearance of the end effector system 22 equipped on the harvesting robot 2, and Figure 3 is a diagram showing an example of the general configuration of the harvesting robot 2.

As shown in Figures 2(a) and 3, the harvesting robot 2 includes a carriage 21, an end effector system 22, a manipulator system 23, wheels 24, and a support member 25.

The cart 21 has a control function for controlling the end effector system 22 and the manipulator system 23, and a movement function for moving inside and outside the field. As shown in FIG. 2(a) and FIG. 3, the cart 21 has a control device 211 and a movement device 212, thereby realizing the control function and the movement function.

The control device 211 executes processing to make the harvesting robot 2 realize various functions, and outputs control signals to the moving device 212, as well as the sensor device 221, the driving device 222, the irradiation device 224, and the driving device 251, which will be described later. The control device 211 uses the three-dimensional position data during the deciduous season managed by the server device 3 to control the manipulator system 23 so that at least a part of the harvesting robot 2 does not collide with the trees. In this way, the three-dimensional position data during the deciduous season is used by the control device 211 as obstacle data indicating obstacles to the harvesting robot 2. Note that although an "obstacle" is a "tree" (not including "leaves"), an object that impedes the movement of the harvesting robot 2, the end effector system 22, and the manipulator system 23 (for example, "various members that support the branches (posts (pipes, stakes, etc.), wires, tapes, and/or strings, etc.)") may be included in the "obstacle". Hereinafter, all or a part of the three-dimensional position data during the deciduous season may be referred to as deciduous season obstacle data. The control device 211 has a communication unit 211a, a memory unit 211b, and a processing unit 211c.

The communication unit 211a includes a communication interface circuit that performs wireless communication based on a wireless communication method of a predetermined communication standard with an access point of a wireless LAN (Local Area Network) (not shown). The predetermined communication standard is, for example, the IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 802.11 standard. The communication unit 211a receives various data and the like transmitted from the server device 3 via wireless communication. The communication unit 211a may transmit various data to the server device 3 via wireless communication.

The communication unit 211a may also establish a wireless communication link with a base station (not shown) via a channel assigned by the base station, and communicate with the base station. The wireless communication link is, for example, a wireless communication link conforming to the LTE (Long Term Evolution) standard, a wireless communication link of a fourth generation mobile communication system (LTE-Advanced, 4G), or a wireless communication link of a mobile communication system such as the fifth generation mobile communication system (5G) standard.

The communication unit 211a may also have an interface circuit for short-distance wireless communication according to a communication method such as Bluetooth (registered trademark), receive radio waves corresponding to various data from the server device 3, and transmit radio waves corresponding to the various data to the server device 3. The communication unit 211a may also have a transmission circuit for transmitting various signals corresponding to the various data by infrared communication or the like, and a reception circuit for receiving various signals corresponding to the various data by infrared communication or the like. The communication unit 211a may also have a communication interface circuit for a wired LAN.

In this way, by being equipped with the communication unit 211a, the harvesting robot 2 is able to transmit various data to the server device 3 and receive various data from the server device 3 via the communication unit 211a.

In place of or in addition to the communication unit 211a, the harvesting robot 2 may be provided with an input/output device that detachably holds a portable storage medium. When a portable storage medium that stores data managed by the server device 3 is attached to the input/output device provided in the harvesting robot 2, the input/output device acquires the data stored in the portable storage medium and passes it to the processing unit 211c described below. This allows the harvesting robot 2 to receive data from the server device 3 via the portable storage medium. The input/output device also stores data in the portable storage medium in response to instructions from the processing unit 211c. This allows the harvesting robot 2 to provide data to the server device 3 via the portable storage medium.

The storage unit 211b is a semiconductor memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 211b stores an operating system program, a driver program, an application program, and data used in the processing in the processing unit 211c. The driver programs stored in the storage unit 211b are a communication device driver program that controls the communication unit 211a, a sensor device driver program that controls the sensor device 221, and a lighting device driver program that controls the irradiation device 224. The application programs stored in the storage unit 211b are a harvest obstacle data extraction process that extracts harvest obstacle data (described later) based on sensor data from the sensor device 221, a defoliation obstacle data extraction process that extracts defoliation obstacle data based on sensor data from the sensor device 221, and a harvest control process that controls the end effector system 22 and the manipulator system 23 and controls the movement of the moving device 212 during harvesting, and the like, and the like, for causing the processing unit 211c to execute such processes. The storage unit 211b may also temporarily store data related to a specific process.

The processing unit 211c is a processing device that loads the operating system program, driver program, and application program stored in the storage unit 211b into memory and executes the instructions contained in the loaded programs. The processing unit 211c is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), or a combination of various electronic circuits. In FIG. 3, the processing unit 211c is illustrated as a single component, but the processing unit 211c may be a collection of multiple physically separate processors.

The processing unit 211c executes various commands contained in the control program to realize various functions. An example of the functions realized by the processing unit 211c will be described in detail later.

The moving device 212 includes a drive motor that rotates the wheels 24 via a drive transmission mechanism (not shown), and a steering mechanism including a motor that changes the steering angle of the wheels 24. The moving device 212 controls the wheels 24 according to a movement control signal from the processing unit 211c so that the carriage 21 moves along a specified movement path to a specified position inside or outside the field.

The end effector system 22 has a harvesting function for harvesting one or more types of crops. As shown in FIG. 2(b) and FIG. 3, the end effector system 22 has a sensor device 221, a drive device 222, a picking device 223, and an irradiation device 224, thereby realizing the harvesting function.

The sensor device 221 has a function of acquiring data on the position of the crop, and includes, for example, an image capturing unit 221a for capturing an image of a subject and outputting captured image data of the subject, and a measurement unit 221b for measuring the distance to objects around the harvesting robot 2. For example, an RGB-Depth sensor such as the "Intel RealSense Depth Camera D435i ("Intel RealSense" is a registered trademark)" may be used as the sensor device 221. The sensor device 221 is an example of a sensor unit.

The photographing unit 221a has an optical lens and an image sensor. The optical lens focuses a light beam from a subject on the image sensor's image plane. The image sensor is a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and outputs an image of the subject image focused on the image sensor's image plane. The photographing unit 221a creates still image data in a specified file format from the image generated by the image sensor as photographed image data and outputs it. The photographing unit 221a may output the photographed image data in association with the acquisition time of the photographed image data. The photographing unit 221a may also create and output moving image data in a specified file format as photographed image data at specified intervals from images continuously generated by the image sensor.

The measurement unit 221b is, for example, a pair of infrared sensors. In the example shown in FIG. 2B, a pair of a first lens 221b1 and a second lens 221b2 is provided in the sensor device 221. Each of the first lens 221b1 and the second lens 221b2 forms an image of infrared rays from an object in approximately the same space as the shooting target space corresponding to the shooting image data acquired by the shooting unit 221a on the light receiving element. The measurement unit 221b outputs infrared images corresponding to each of the first lens 221b1 and the second lens 221b2 based on an electrical signal from the light receiving element, measures distances corresponding to multiple points on the surface of the object from the output infrared images using a known active stereo method, and outputs depth data (distance data) corresponding to each of the multiple points. The measurement unit 221b may output the depth data in association with the acquisition time of the depth data. In addition, the measurement unit 221b may create and output time-series depth data in a predetermined file format from the continuously output infrared images at predetermined intervals. In addition, the infrared (IR) projector 221c has the function of projecting infrared rays to improve the sensitivity of the measurement unit 221b to receiving infrared rays from an object.

The sensor device 221 supplies sensor data including the captured image data output from the image capture unit 221a and the depth data output from the measurement unit 221b to the processing unit 211c. The sensor data (captured image data and depth data) is an example of data related to the position of the crop. Note that only one of the captured image data and the depth data may be used as data related to the position of the crop.

The sensor device 221 is not limited to the above examples, and may be, for example, any of a variety of sensor devices that output distance data corresponding to multiple points on the surface of an object (such as a pulse-echo ultrasonic distance sensor, or a laser scanner such as LiDAR (Light Detection and Ranging) or a laser range finder).

The drive device 222 is a drive motor or the like for operating the harvesting device 223, which will be described later. The harvesting device 223 has a harvesting function of gripping the crop to be harvested and plucking the gripped crop from the branch. The harvesting device 223 shown in FIG. 2(b) has a gripping unit that grasps the stalk when the crop to be harvested is a crop (such as cherries) that is attached to the branch via the stalk. The harvesting device 223 is not limited to the example shown in FIG. 2(b), and may have a configuration suitable for harvesting each crop to be harvested. The harvesting device 223 performs a harvesting operation suitable for each crop to be harvested (an operation of gripping the crop to be harvested and plucking the gripped crop from the branch) to harvest the crop. The harvesting device 223 is an example of a picking unit.

The irradiation device 224 is, for example, an LED lighting device, and emits light in response to operation instructions from the operator operating the harvesting robot 2 or control instructions from the processing unit 211c.

The manipulator system 23 has a function of moving the picking device 223 of the end effector system 22 to a position where the crop can be harvested. The manipulator system 23 shown in FIG. 2(a) is, for example, a vertical articulated manipulator having joints.

The support member 25 is installed on the upper surface of the cart 21 and supports the manipulator system 23. The support member 25 may be configured to be movable relative to the cart 21. The support member 25 includes a drive unit 251. The drive unit 251 is a drive motor or the like for rotating the manipulator system 23 relative to the support member 25. For example, the drive unit 251 is rotatably connected to the tip of the manipulator system 23 and rotates the manipulator system 23 about two predetermined axes (for example, an axis in the y direction in FIG. 2(a) and an axis perpendicular to the y direction axis). This allows for movements similar to "shoulder rotation" and "arm raising" to be realized when the drive unit 251 and the manipulator system 23 are considered to be a human "shoulder" and "arm".

The manipulator system 23 may include a drive motor for rotating the end effector system 22 relative to the manipulator system 23. For example, the drive motor is rotatably connected to the tip of the manipulator system 23 and rotates the end effector system 22 about a predetermined axis of the harvesting robot 2 (for example, an axis in the extension direction of the manipulator system 23 and an axis perpendicular to said axis). This allows movements similar to "wrist twisting" and "wrist bending" to be realized when the end effector system 22 and the manipulator system 23 are considered to be a human "hand" and "arm".

The moving device 212 moves the carriage 21 in accordance with a movement control signal from the processing unit 211c so that the carriage 21 is positioned within a range where the crop can be harvested by the end effector system 22. The driving device 251 is then controlled in accordance with the movement control signal from the processing unit 211c so that the end effector system 22 moves to a position where the crop can be harvested.

(Server device 3)
FIG. 4 is a diagram showing an example of a schematic configuration of the server device 3. As shown in FIG.

The server device 3 has a data management function for storing and managing defoliation obstacle data, and a transmission function for transmitting defoliation obstacle data to the harvesting robot 2. The server device 3 also has a receiving function for receiving defoliation obstacle data transmitted from an external device. The external device may be the harvesting robot 2. To this end, the server device 3 includes a server communication unit 31, a server storage unit 32, and a server processing unit 33.

The server communication unit 31 is implemented as hardware, firmware, or communication software such as a TCP/IP driver or a PPP driver, or a combination of these. The server device 3 can transmit various data, such as obstacle data during the defoliation season, to the harvesting robot 2 via the server communication unit 31. The server device 3 can receive various data from an external device via the server communication unit 31. The server communication unit 31 may also have an interface circuit for short-range wireless communication according to a communication method such as Bluetooth (registered trademark), and may transmit radio waves to the harvesting robot 2 and receive radio waves from the harvesting robot 2. The server communication unit 31 may also have a transmission circuit for transmitting various signals corresponding to various data by infrared communication or the like, and may also have a receiving circuit for receiving various signals corresponding to various data by infrared communication or the like. The server communication unit 31 may also have a communication interface circuit for a wired LAN. This allows the server device 3 to transmit and receive various data to and from the harvesting robot 2 via the server communication unit 31. The server device 3 may be provided with an input/output device that detachably holds a portable storage medium instead of or in addition to the server communication unit 31. In this case, the input/output device acquires various data, such as the defoliation obstacle data, stored in the portable storage medium, and supplies the acquired various data to the storage processing unit 332. This allows the server device 3 to acquire various data, such as the defoliation obstacle data, from an external device via the portable storage medium.

The server storage unit 32 is, for example, a semiconductor memory device such as a ROM or a RAM. The server storage unit 32 may be, for example, a magnetic disk, an optical disk, or any other storage device capable of storing data. The server storage unit 32 stores an operating system program, a driver program, an application program, data, and the like used in the processing in the server processing unit 33. The computer programs stored in the server storage unit 32 may be installed in the server storage unit 32 from a computer-readable portable storage medium such as a CD-ROM or a DVD-ROM using a known setup program, and the like. The data stored in the server storage unit 32 includes various data such as defoliation obstacle data acquired from an external device. The server storage unit 32 may temporarily store data related to a specified process.

The server processing unit 33 is a processing device that loads the operating system program, driver program, and application program stored in the server storage unit 32 into memory and executes the instructions contained in the loaded programs. The server processing unit 33 is, for example, an electronic circuit such as a CPU, MPU, or DSP, or a combination of various electronic circuits. In FIG. 4, the server processing unit 33 is illustrated as a single component, but the server processing unit 33 may be a collection of multiple physically separate processors.

The server processing unit 33 functions as a server receiving unit 331, a memory processing unit 332, and a server transmitting unit 333 by executing various commands contained in the application program (control program) stored in the server memory unit 32.

The server receiving unit 331 receives various data, such as defoliation obstacle data, transmitted from an external device via the server communication unit 31. The storage processing unit 332 stores the various data received by the server receiving unit 331 in the server memory unit 32. The server transmitting unit 333 transmits various data, such as defoliation obstacle data, stored in the server memory unit 32 to the harvesting robot 2 via the server communication unit 31 in response to a request from the harvesting robot 2 or in response to an operation by the operator.

(Deciduous season obstacle data extraction processing)
An example of the deciduous season obstacle data extraction process will be described below with reference to Figures 5 to 7. The deciduous season obstacle data extraction process is executed using sensor data acquired by the sensor device 221 of the harvest robot 2. For example, when the harvest robot 2 is moving through a field in which trees of a crop to be harvested are planted during the season in which the trees have lost their leaves, the sensor device 221 of the harvest robot 2 acquires sensor data at a predetermined timing. The deciduous season obstacle data extraction process may be executed by an external device different from the harvest robot 2. For example, the external device may have the same functions as the sensor device 221 and the processing unit 211c of the harvest robot 2, and the deciduous season obstacle data extraction process may be executed by the external device during or after the operator moves using sensor data acquired while moving with the external device.

Figure 5 shows an example of the operation flow of the deciduous season obstacle data extraction process executed by the processing unit 211c.

First, in step S101, the processing unit 211c acquires sensor data output from the sensor device 221. The sensor device 221 may output sensor data at predetermined time intervals (e.g., every second), in which case the processing unit 211c acquires the sensor data output from the sensor device 221 every time the sensor data is output from the sensor device 221.

The sensor data includes the photographed image data output from the photographing unit 221a and the depth data output from the measuring unit 221b. FIG. 6(a) is a diagram showing an example of photographed image data taken in a field where defoliated trees are planted. The photographed image data is, for example, data for displaying a rectangular image composed of a plurality of pixels, and RGB data for expressing color is associated with each pixel. FIG. 6(b) is a schematic diagram showing an example of depth data (distance data) output when the photographed image data shown in FIG. 6(a) is photographed. The depth data (distance data) is, for example, a set of numerical data indicating the distance from the sensor device 221 corresponding to each of a plurality of pixels that constitute a rectangular shape substantially the same as the photographed image data photographed at the same time. The resolution of the depth data may be the same as that of the photographed image data or may be different from that of the photographed image data. Hereinafter, the depth data measured when the trees are defoliated may be referred to as defoliation stage depth data.

Next, in step S102, the processing unit 211c acquires the captured image data contained in the sensor data (acquired in step S101).

Next, in step S103, the processing unit 211c estimates an obstacle image area in the captured image data (obtained in step S102). Below, an example of the obstacle image area estimation process executed by the processing unit 211c is described.

The processing unit 211c performs the estimation process of the obstacle image area using a learning model generated by deep learning based on, for example, U-Net (Convolutional Networks for Biomedical Image Segmentation).

U-Net is a method for recognizing objects for each pixel that constitutes the captured image data, and the processing unit 211c estimates an image area that indicates a "branch" in the captured image data as an obstacle image area. The processing unit 211c may estimate, in addition to the image area that indicates the "branch," image areas that indicate various components that support the branch in the captured image data (supports (pipes, stakes, etc.), wires, tapes, and/or strings, etc.) as obstacle image areas. In this way, the processing unit 211c estimates, as obstacle image areas, image areas that indicate obstacles that impede the movement of the end effector system 22 and/or manipulator system 23 during harvesting.

Figure 7 (a) is a diagram showing an example of estimated image data showing an obstacle image area. The estimated image data shown in Figure 7 (a) is output from the output layer of the trained neural network when the captured image data input to the input layer of the trained neural network is the captured image data shown in Figure 6 (a). The estimated image data is image data in which pixels showing obstacles and pixels showing non-obstacles are displayed in different ways.

The process of generating the learning model is executed by the processing unit 211c of the harvesting robot 2, the server processing unit 33 of the server device 3, or a processing unit of an external device, etc.

For example, multiple pieces of training data are created by an operator or manager of the harvesting robot 2. Then, deep learning is performed using the multiple pieces of training data to generate or update a learning model in which the weights of each neuron in the neural network are learned.

Next, the process returns to step S104, where the processing unit 211c determines whether an obstacle image area was estimated in step S103 (whether an obstacle image area exists in the estimated image data output in step S103).

If an obstacle image area is not estimated in step S103 (step S104-No), the processing unit 211c advances the deciduous season obstacle data extraction process to step S108.

If an obstacle image area is estimated in step S103 (step S104-Yes), in step S105, the processing unit 211c acquires defoliation depth data included in the sensor data (acquired in step S101).

Next, in step S106, the processing unit 211c calculates deciduous season three-dimensional point cloud data based on the deciduous season depth data (acquired in step S105). The deciduous season three-dimensional point cloud data is a collection of three-dimensional position data corresponding to each pixel of the deciduous season depth data.

The calculation process of the deciduous season three-dimensional point cloud data is a process of calculating the deciduous season three-dimensional point cloud data by simultaneously converting the position and depth data of each pixel of the deciduous season depth data into three-dimensional position data according to a specified position coordinate system (for example, a geographic coordinate system such as the Japanese Geodetic System 2011, which is indicated by latitude, longitude, and altitude).

As a calculation process for the deciduous season three-dimensional point cloud data, for example, the processing unit 211c may execute a process for calculating the deciduous season three-dimensional point cloud data based on a known self-location estimation and environmental map creation method such as SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion).

When the calculation process of the three-dimensional point cloud data in the defoliation season based on the self-location estimation and environmental map creation technique is executed, the depth data included in the sensor data may not be used. In this case, the processing unit 211c may calculate the relative positions in a virtual three-dimensional space for each of the shooting positions of the two pieces of captured image data and the positions corresponding to the multiple feature points according to SLAM or SfM, etc., using only multiple captured image data from different shooting positions, and calculate the three-dimensional point cloud data in the defoliation season. When the processing unit 211c executes the calculation process of the three-dimensional point cloud data in the defoliation season based on the self-location estimation and environmental map creation technique, it may execute a known calibration process by passing again a point that has been passed since starting to acquire sensor data (for details of the calibration process, see Japanese Patent No. 6525148, etc.).

Next, returning to step S107 of the defoliation obstacle data extraction process shown in FIG. 5, the processing unit 211c extracts defoliation obstacle data from the calculated defoliation obstacle three-dimensional point cloud data. For example, the processing unit 211c superimposes the defoliation obstacle data and the estimated image data so that the measurement target space of the defoliation obstacle depth data (acquired in step S105) and the shooting target space of the estimated image data (estimated in step S103) are substantially the same. Next, the processing unit 211c extracts pixels that overlap with the obstacle image area in the estimated image data from among the pixels of the defoliation obstacle depth data. The processing unit 211c extracts the defoliation obstacle three-dimensional point cloud data corresponding to the extracted pixels as defoliation obstacle data. FIG. 7(b) is a schematic diagram showing an example of defoliation obstacle data extracted based on the defoliation obstacle depth data shown in FIG. 6(b).

Next, in step S108, the processing unit 211c determines whether or not acquisition of sensor data from the sensor device 221 has been completed. If it is determined that new sensor data has been acquired from the sensor device 221 (step S108-No), the processing unit 211c returns the defoliation season obstacle data extraction process to step S101. Furthermore, if it is determined that acquisition of sensor data from the sensor device 221 has been completed (step S108-Yes), the processing unit 211c ends the defoliation season obstacle data extraction process.

When the defoliation season obstacle data extraction process is completed, the processing unit 211c transmits the defoliation season obstacle data to the server device 3 via the communication unit 211a. The server receiving unit 331 of the server device 3 receives the defoliation season obstacle data via the server communication unit 31, and the storage processing unit 332 stores the received defoliation season obstacle data in the server storage unit 32. Note that the processing unit 211c may store the defoliation season obstacle data in the storage unit 211b without transmitting the defoliation season obstacle data to the server device 3.

(Harvesting process)
When the crops are ready to be harvested, the harvest robot 2 harvests them. An example of the harvesting process executed by the processing unit 211c of the harvest robot 2 will be described below with reference to FIG. 8. Note that before the harvesting process is executed, the server transmission unit 333 of the server device 3 transmits the defoliation season obstacle data stored in the server storage unit 32 to the harvest robot 2 via the server communication unit 31. The processing unit 211c of the harvest robot 2 receives the defoliation season obstacle data via the communication unit 211a and stores the received defoliation season obstacle data in the storage unit 211b. Note that when the defoliation season obstacle data is stored in the storage unit 211b, the processing unit 211c does not need to receive the defoliation season obstacle data from the server device 3.

First, in step S201, the processing unit 211c acquires sensor data output from the sensor device 221. The sensor device 221 may output sensor data at predetermined time intervals (e.g., every second), in which case the processing unit 211c acquires the sensor data output from the sensor device 221 every time the sensor data is output from the sensor device 221.

Next, in step S202, the processing unit 211c acquires the captured image data contained in the sensor data (acquired in step S201).

Next, in step S203, the processing unit 211c detects the presence or absence of crops by determining whether the captured image data (obtained in step S202) includes crops to be harvested. An example of the crop detection process executed by the processing unit 211c will be described below. Note that the example will be described in which the crop to be detected is "cherries."

First, the processing unit 211c instructs the sensor device 221 to acquire sensor data. Note that the processing unit 211c may instruct the sensor device 221 to acquire sensor data while the illumination device 224 is irradiating light.

Next, the processing unit 211c estimates a candidate image area corresponding to a cherry fruit based on the captured image data included in the sensor data acquired from the sensor device 221. For example, the processing unit 211c executes a process of determining whether or not the candidate image area corresponds to a cherry fruit, using a learning model generated by machine learning based on a known SVM (Support Vector Machine).

If the processing unit 211c estimates that the candidate image area corresponding to the cherry fruit is not present in the image data (step S203-No), the processing unit 211c advances the harvesting process to step S209. If the processing unit 211c estimates that the candidate image area corresponding to the cherry fruit is present in the image data (step S203-Yes), the processing unit 211c further estimates an image area corresponding to the stalk connected to the cherry fruit in the candidate image area using a predetermined learning model, and calculates the position of the crop (step S204).

For example, in step S204, the processing unit 211c outputs three-dimensional position data of the stalk estimated from the candidate image area (three-dimensional position data in the camera coordinate system) as the position of the crop. For example, the processing unit 211c calculates three-dimensional coordinates (x, y, z) in the camera coordinate system corresponding to one or more pixels corresponding to the stalk, based on one or more pixels in the image area corresponding to the stalk and the depth value indicated in the depth data. The processing unit 211c then outputs three-dimensional position data indicating the calculated three-dimensional coordinates (x, y, z) as the position of the crop. In this way, the processing unit 211c makes it possible to detect the crop to be harvested based on the sensor data.

In addition, to determine the crop to be harvested, a learning model generated by machine learning based on a CNN (Convolutional Neural Network) such as SSD (Single Shot Multibox Detector) or YOLO (You Only Look Once) may be used to determine the crop to be harvested. For object recognition methods using SSD, see W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed, "SSD: Single Shot Multibox Detector", December 29, 2016, [online], <https://arxiv.org./pdf/1512.02325.pdf>, etc.

Next, in step S205, the processing unit 211c executes a harvesting obstacle data extraction process. The details of the harvesting obstacle data extraction process will be described later.

Next, in step S206, the processing unit 211c acquires the defoliation obstacle data stored in the memory unit 211b.

Next, in step S207, the processing unit 211c converts the three-dimensional coordinates (X, Y, Z) in a predetermined position coordinate system indicated by the acquired defoliation obstacle data into three-dimensional coordinates (x, y, z) in a camera coordinate system, for example, using the camera parameters of the measurement unit 221b, thereby aligning the position of the defoliation obstacle data (three-dimensional coordinates (X, Y, Z) in the predetermined position coordinate system) with the shooting target space (three-dimensional space in the camera coordinate system) within a predetermined viewing angle in the current shooting direction of the sensor device 221.

When defoliation season obstacle data and shooting time obstacle data are generated based on the self-position estimation and environmental map creation techniques, a three-dimensional model (e.g., a known surface model) based on each obstacle data may be generated, and the position of the defoliation season obstacle data may be aligned with the shooting target space (three-dimensional space in the camera coordinate system) within a specified field of view in the current shooting direction of the sensor device 221 by rotating and/or moving the defoliation season obstacle data so that the two three-dimensional models approximately match.

Next, in step S208, the processing unit 211c executes a harvest control process that controls the end effector system 22, the manipulator system 23, and the moving device 212 based on the defoliation season obstacle data after the alignment process and the harvest obstacle data extracted in step S205. The harvest control process will be described in detail later.

Next, in step S209, the processing unit 211c determines whether or not to harvest other crops. For example, the processing unit 211c may determine to further harvest other crops in response to a harvest instruction from the operator. In this case, the processing unit 211c may determine to end the harvest of other crops in response to an end instruction from the operator.

If it is determined that other crops are to be harvested (step S209-Yes), the processing unit 211c returns the harvesting process to step S201, and if it is determined that the harvesting of other crops is to be ended (step S209-No), the processing unit 211c ends the harvesting process.

(Harvesting obstacle data extraction process)
An example of the harvest obstacle data extraction process will be described below with reference to Figures 9 and 10. The harvest obstacle data extraction process is executed in step S205 of the harvest process using sensor data acquired by the sensor device 221 of the harvest robot 2. For example, during harvesting, when the harvest robot 2 moves through a farm field where trees bearing crops to be harvested are planted, the sensor device 221 of the harvest robot 2 acquires sensor data at a predetermined timing.

Figure 9 shows an example of the operation flow of the harvesting obstacle data extraction process executed by the processing unit 211c.

First, in step S301, the processing unit 211c acquires the captured image data included in the sensor data (acquired in step S201). Note that the processing in step S301 is similar to the processing in step S102. FIG. 10(a) is a diagram showing an example of captured image data of a crop to be harvested at harvest time. FIG. 10(b) is a schematic diagram showing an example of harvest depth data measured when the captured image data shown in FIG. 10(a) was captured.

Next, in step S302, the processing unit 211c estimates an obstacle image area in the captured image data (obtained in step S301). Note that the processing in step S302 is similar to the processing in step S103. FIG. 10(c) is a diagram showing an example of estimated image data showing an obstacle image area estimated from the captured image data shown in FIG. 10(a).

Next, in step S303, the processing unit 211c determines whether or not an obstacle image area was estimated in step S302. Note that the processing in step S303 is similar to the processing in step S104.

If an obstacle image area is not estimated in step S302 (step S303-No), the processing unit 211c ends the harvesting obstacle data extraction process.

If an obstacle image area is estimated in step S302 (step S303-Yes), in step S304, the processing unit 211c acquires harvest depth data included in the sensor data (acquired in step S201). Note that the processing in step S304 is similar to the processing in step S105.

Next, in step S305, the processing unit 211c calculates three-dimensional point cloud data at harvest based on the harvest depth data (acquired in step S304). For example, the three-dimensional point cloud data at harvest is three-dimensional position data indicating the three-dimensional coordinates (x, y, z) in the camera coordinate system corresponding to each pixel, calculated from each pixel of the harvest depth data and the depth data of each pixel.

Then, in step S306, the processing unit 211c extracts harvest obstacle data from the calculated harvest 3D point cloud data, and ends the harvest obstacle data extraction process. The process of step S306 is the same as the process of step S107. FIG. 10(d) is a schematic diagram showing an example of harvest obstacle data extracted based on the harvest depth data shown in FIG. 10(b). The harvest obstacle data is an example of harvest 3D position data. The harvest obstacle data extracted by the harvest obstacle data extraction process is harvest 3D point cloud data corresponding to trees that do not include "leaves". The "leaves" of trees are objects that do not impede the movement of the harvest robot 2, the end effector system 22, and the manipulator system 23, and the processing unit 211c makes it possible to generate harvest obstacle data that does not include 3D point cloud data corresponding to "leaves".

(Harvest Control Treatment)
An example of the harvest control process will be described below with reference to Fig. 11. The harvest control process is executed by the processing unit 211c in step S208 of the harvest process.

First, in step S401, the processing unit 211c determines whether or not at least a part of the harvesting robot 2 will collide with an obstacle when the end effector system 22 is moved from its current position to a position where a picking operation can be performed on the crop to be harvested. The crop to be harvested is the crop corresponding to the position output in step S204. At least a part of the harvesting robot 2 is, for example, a part of the end effector system 22 and the manipulator system 23.

First, the processing unit 211c estimates at least one of the position of the obstacle and the shape of the obstacle based on at least one of the defoliation obstacle data and the harvest obstacle data. For example, the processing unit 211c generates a known surface model based on the three-dimensional position data of the defoliation obstacle data and the harvest obstacle data. The surface model may be represented by a predetermined three-dimensional object (e.g., a cylindrical object) data, or may be represented by polygon data in which the surface is represented by polygons. In this way, the surface model is generated, and the position and shape of the obstacle are estimated.

Next, the processing unit 211c calculates the movement paths of the end effector system 22 and the manipulator system 23 accompanying the movement of the end effector system 22. Then, the processing unit 211c determines whether the generated surface model intersects with the calculated movement path. If the processing unit 211c determines that the generated surface model intersects with the calculated movement path, it determines that at least a part of the harvesting robot 2 will collide with an obstacle. Furthermore, if the processing unit 211c determines that the generated surface model does not intersect with the calculated movement path, it determines that at least a part of the harvesting robot 2 will not collide with an obstacle. Note that the processing unit 211c may determine whether the calculated movement path intersects with at least one of the deciduous season obstacle data and the harvesting obstacle data. In this case, if the processing unit 211c determines that at least one of the defoliation obstacle data and the harvest obstacle data intersects with the calculated movement path, it determines that at least a part of the harvest robot 2 will collide with an obstacle, and if it determines that at least one of the defoliation obstacle data and the harvest obstacle data does not intersect with the calculated movement path, it determines that at least a part of the harvest robot 2 will not collide with an obstacle.

If it is determined that at least a part of the harvesting robot 2 will collide with an obstacle (step S401-Yes), in step S402, the processing unit 211c sends an avoidance control instruction to at least one of the moving device 212, the driving device 251, the manipulator system 23, and the driving device 222, and returns the process to step S401.

For example, when the processing unit 211c determines that there is an operation that does not cause the harvesting robot 2 to collide with an obstacle among the multiple operations (movement paths) of the end effector system 22 and the manipulator system 23 for moving the manipulator system 23, the processing unit 211c controls at least one of the driving device 251, the manipulator system 23, and the driving device 222 according to the control information of the operation. Also, if at least a part of the harvesting robot 2 is sure to collide with an obstacle when the end effector system 22 is moved from the current position of the harvesting robot 2 to a position where a picking operation is possible, the processing unit 211c notifies the moving device 212 of a movement instruction and changes the position of the harvesting robot 2 by moving the cart 21.

If it is determined that at least a part of the harvesting robot 2 will not collide with an obstacle (step S401-No), in step S403, the processing unit 211c notifies at least one of the driving device 251, the manipulator system 23, and the driving device 222 of a harvesting control instruction, and ends the harvesting control process. The harvesting control process enables the manipulator system 23 to move to a position where it can pick the crop, and the picking device 223 to perform a predetermined picking operation.

The harvest control process may be performed using only the defoliation obstacle data, without using the harvest obstacle data.

As described above in detail, the harvesting system 1 of this embodiment uses sensor data acquired when the leaves of the trees of the crop to be harvested have fallen to determine whether or not there is an obstacle that will impede harvesting. This makes it possible for the harvesting system 1 of this embodiment to prevent accidents in which the obstacle is hidden by the leaves of the trees and goes undetected, resulting in the harvesting robot 2 colliding with the obstacle.

(Variation 1)
The defoliation stage obstacle data extraction process (FIG. 5) may include a data addition step of adding flower bud position data indicating the position of the flower bud to the harvesting 3D point cloud data. For example, the data addition step is executed when it is determined that the acquisition of sensor data from the sensor device 221 has ended (step S108-Yes). The data addition step is, for example, a process in which a learning model (for example, a model based on a known SVM) generated by machine learning determines whether or not the position of the flower bud exists in the captured image data, and when it is determined that the position of the flower bud exists, a label indicating the position of the flower bud is added to the harvesting 3D point cloud data corresponding to the position. The data addition step may be an operation process in which an operator adds flower bud position data regarding the position of the flower bud to the harvesting 3D point cloud data.

When flower bud position data is added to the harvest time 3D point cloud data, for example, in step S203 of the harvest process, it may be determined whether or not the captured image data contains a crop to be harvested, depending on whether or not the captured image data contains flower bud position data. Also, in step S204 of the harvest process, the flower bud position data may be output as the position of the crop.

In this way, by adding flower bud position data to the 3D point cloud data at the time of harvest, it becomes possible to estimate the position of the crop even in cases where it is not possible to determine whether or not the crop is present in the captured image data.

(Variation 2)
The "defoliation period" in the "defoliation period three-dimensional position data" used in the harvesting system 1 of this embodiment may be the "defoliation period" one year or more before the harvest time.

REFERENCE SIGNS LIST 1 Harvesting system 2 Harvesting robot 21 Cart 211 Control device 211a Communication unit 211b Memory unit 211c Processing unit 212 Moving device 22 End effector system 221 Sensor device 221a Photography unit 221b Measurement unit 222 Driving device 223 Picking device 224 Irradiation device 23 Manipulator system 24 Wheel 25 Support member 251 Driving device 3 Server device 31 Server communication unit 32 Server memory unit 33 Server processing unit 331 Server receiving unit 332 Storage processing unit 333 Server processing unit

Claims (10)

A harvesting robot that harvests crops that grow on deciduous trees,
a storage unit that stores three-dimensional position data during the leaf-shedding season, the data including the position of at least a portion of the tree from which leaves have fallen;
A harvesting unit for harvesting the crop;
A manipulator that moves the picking unit;
A sensor unit for acquiring data relating to the position of the crop;
Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop;
estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data;
a processing unit that controls the manipulator when the harvest target crop is detected so that the picking unit moves to a harvestable position of the detected crop without causing a collision between the estimated obstacle and at least a part of the harvesting robot;
A harvesting robot comprising: The processing unit includes:
When the harvest target crop is detected, harvest time three-dimensional position data including a position of at least a part of the obstacle is acquired based on the position data acquired by the sensor unit;
The harvesting robot according to claim 1 , wherein at least a position of an obstacle that impedes movement of the picking unit is estimated based on the three-dimensional position data during leaf fall and the three-dimensional position data during harvest. the sensor unit has an image capturing unit that captures an image of a subject and outputs captured image data of the subject, and a measurement unit that measures distances to surrounding objects and outputs distance data indicating the measured distances;
The processing unit includes:
calculating three-dimensional position data corresponding to one or more points in the captured image data based on the distance data corresponding to the one or more points output;
The harvesting robot according to claim 2 , wherein the calculated three-dimensional position data is acquired as the three-dimensional position data at the time of harvesting. The harvesting robot according to claim 3, wherein the three-dimensional position data during the defoliation period is three-dimensional position data corresponding to one or more points, calculated based on the image data captured by the image capture unit during the defoliation period immediately prior to the harvesting of the crop, and distance data corresponding to one or more points in the captured image data output by the measurement unit at the time of capture. The processing unit includes:
From the captured image data, an image area showing a branch of the tree and an image area showing a member supporting the branch of the tree are estimated as obstacle image areas;
The harvesting robot according to claim 3 or 4, wherein three-dimensional position data within the obstacle image area among the calculated three-dimensional position data is acquired as the harvesting time three-dimensional position data. A cart that moves within a field in which the trees are planted is provided,
The harvesting robot according to any one of claims 1 to 5, wherein, when the crop to be harvested is detected, the processing unit controls the movement of the cart so that the picking unit can move to a harvestable position of the detected crop without colliding with the estimated obstacle and at least a part of the harvesting robot. The harvesting robot according to any one of claims 1 to 6, wherein the processing unit estimates the position and shape of the obstacle based on the three-dimensional position data during the leaf-shedding period. A method for controlling a harvesting robot that harvests crops that grow on deciduous trees, the harvesting robot comprising: a memory unit that stores three-dimensional position data during a deciduous season that includes a position of at least a portion of the tree from which leaves have fallen; a picking unit that harvests the crops; a manipulator that moves the picking unit; a sensor unit that acquires data on the position of the crops; and a processing unit,
The processing unit includes:
Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop;
estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data;
When the crop to be harvested is detected, the control method includes controlling the manipulator so that the picking unit moves to a harvestable position of the detected crop without colliding with the estimated obstacle and at least a part of the harvesting robot. A control program for a harvesting robot that harvests crops that grow on deciduous trees, the harvesting robot comprising: a memory unit that stores three-dimensional position data during the deciduous season, the data including the position of at least a portion of the tree from which leaves have fallen; a picking unit that harvests the crops; a manipulator that moves the picking unit; a sensor unit that acquires data regarding the position of the crops; and a processing unit,
The processing unit includes:
Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop;
estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data;
A control program for executing the following: when a crop to be harvested is detected, controlling the manipulator so that the picking unit moves to a harvestable position of the detected crop without causing a collision between the estimated obstacle and at least a part of the harvesting robot. A harvesting system having a harvesting robot that harvests crops that grow on deciduous trees and a server device that communicates with the harvesting robot,
The server device includes:
a storage unit that stores three-dimensional position data during the leaf-shedding season, the data including the position of at least a portion of the tree from which leaves have fallen;
A transmitter that transmits the three-dimensional position data during the leaf-shedding period to the harvesting robot;
Equipped with
The harvesting robot is
a receiving unit for receiving the leaf-shedding season three-dimensional position data transmitted from the server device;
A harvesting unit for harvesting the crop;
A manipulator that moves the picking unit;
A sensor unit for acquiring data relating to the position of the crop;
Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop;
estimating at least a position of an obstacle that impedes movement of the picking unit based on the received three-dimensional position data during the leaf fall period;
a processing unit that controls the manipulator when the harvest target crop is detected so that the picking unit moves to a harvestable position of the detected crop without causing a collision between the estimated obstacle and at least a part of the harvesting robot;
A harvesting system comprising: