AU2019340235B2

AU2019340235B2 - Automated vehicle

Info

Publication number: AU2019340235B2
Application number: AU2019340235A
Authority: AU
Inventors: Timothy Cassell; Rhys Andrew McKercher; Jeffrey James Sterling
Original assignee: Universal Field Robots Pty Ltd
Current assignee: Universal Field Robots Pty Ltd
Priority date: 2018-05-16
Filing date: 2019-05-16
Publication date: 2024-08-01
Anticipated expiration: 2039-05-16
Also published as: WO2020051620A1; AU2019340235A1; AU2025208474A1; AU2024205011A1

Abstract

The present invention relates to an automated vehicle comprising a vision system comprising at least one vision sensor to detect a terrain; a manipulator comprising an effector configured for performing an outdoor activity in the terrain; and an arm configured to position the effector in the vicinity of a feature of the terrain and at least one control unit which utilises scanned terrain data from the vision system to improve movement of the automated vehicle and the manipulator during use.

Description

AEROSPACE CONFERENCE, 3 March 2018 (2018-03-03), Big Sky, MT, USA, pages 1 - 11, XP033365464, DOI: 10.1109/AERO.2018.8396726 SERON, J. ET AL.: "Automation of the Arm-Aided Climbing Maneuver for Tracked Mobile Manipulators", IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, vol. 61, no. 7, July 2014 (2014-07-01), pages 3638-3647.

(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) (19) World Intellectual Property (1) Organization11111111111111111111111I1111111111111ii111liiili International Bureau (10) International Publication Number (43) International Publication Date W O 2020/051620 Al 19 March 2020 (19.03.2020) W IPO I PCT

(51) International Patent Classification: (74) Agent: HOPGOODGANIM; Level 8, Waterfront Place, 1 G05D 1/02 (2006.01) B25J5/00 (2006.01) Eagle Street, Brisbane, QLD 4000 (AU). G05B 19/42 (2006.01) B25J 9/16 (2006.0 1) (81) Designated States (unless otherwise indicated, for every G06K 9/52 (2006.0 1) A 01D 46/30 (2006.01) kind of nationalprotection available): AE, AG, AL, AM, (21) International Application Number: AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW, BY, BZ, PCT/AU2019/000057 CA, CH, CL, CN, CO, CR, CU, CZ, DE, DJ, DK, DM, DO, (22)InternationalFilingDate: DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, 16 May 2019 (16.05.2019) HR, HU, ID, IL, IN, IR, IS, JO, JP, KE, KG, KH, KN, KP, KR, KW, KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, ME, (25) Filing Language: English MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, (26)PublicationLanguage: English SC, SD, SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN, (30) Priority Data: TR, TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW. 2018901710 16 May 2018 (16.05.2018) AU (84) Designated States (unless otherwise indicated, for every (71) Applicant: UNIVERSAL FIELD ROBOTICS PTY kind of regionalprotection available): ARIPO (BW, GH, LTD [AU/AU]; c/HopgoodGanim, Level 8, Waterfront GM, KE, LR, LS, MW, MZ, NA, RW, SD, SL, ST, SZ, TZ, Place, 1 Eagle Street, Brisbane, QLD 4000 (AU). UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ, TM), European (AL, AT, BE, BG, CH, CY, CZ, DE, DK, (72) Inventors: STERLING, Jeffrey James; c/ HopgoodGan- EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU, LV, im, Level 8, Waterfront Place, 1 Eagle Street, Brisbane, MC, MK, MT, NL, NO, PL, PT, RO, RS, SE, SI, SK, SM, QLD 4000 (AU). MCKERCHER, Rhys Andrew; c/ Hop- TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, goodGanim, Level 8, Waterfront Place, 1 Eagle Street, KM, ML, MR, NE, SN, TD, TG). Brisbane, QLD 4000 (AU). CASSELL, Timothy; c/ Hop goodGanim, Level 8, Waterfront Place, 1 Eagle Street, Bris- Published: bane, QLD 4000 (AU). - with internationalsearch report (Art. 21(3))

(54) Title: AUTOMATED VEHICLE

100

10

= 12 14

Figure 1

(57)Abstract: The present invention relates to an automated vehicle comprising a vision system comprising at least one vision sensor to CA detect a terrain; a manipulator comprising an effector configured for performing an outdoor activity in the terrain; and an arm configured to position the effector in the vicinity of a feature of the terrain and at least one control unit whichutilises scanned terrain data from the vision system to improve movement of the automated vehicle and the manipulator during use.

AUTOMATED VEHICLE TECHNICAL FIELD

[0001] The present invention relates to an automated vehicle. Preferably, the present invention relates to an automated vehicle for automating outdoor activities such as in the horticultural, agricultural, construction, mining and forestry industries.

BACKGROUND ART

[0002] Traditionally, outdoor activities such as fruit harvesting, mulching, mowing, digging trenches or clearing unused land are labour intensive. Consequently these tasks can be expensive to the farmer with low efficiency.

[0003] Known machinery which is used for automating these working tasks such as straddle-type bulk apple harvester's still require the use of human operators (which can be represent a significant proportion of the total cost of a crops cost) and limits the time period the machinery can be utilised (unless multiple human operators in shift work are used which increases labour cost). In addition, being a bulk harvester, the proportion of the harvested crop which is damaged or not at the correct stage of ripeness can be large.

[0004] More recently, robots are increasingly being used to automate labour intensive work to vastly increase machine utilisation and improve efficiency. In addition, robots can improve the consistency of the work performed which improves quality. Further, they can operate in situations which would be hazardous to humans e.g. at height such as in tree tops. All these factors increase efficiency and increase profits.

[0005] Dedicated robotic machines while offering the advantages as described above can have the disadvantage that the cost to set up and maintain them can be high and the performance of them such as harvesting rates are not significantly higher compared to manually operated machines.

OBJECT OF THE INVENTION

[0006] The present invention is directed to an automated vehicle which may at least partially overcome at least one of the abovementioned disadvantages or provide the consumer with a useful or commercial choice.

SUMMARY OF INVENTION

[0007] The present invention relates to automated vehicles which provide for automation of outdoor activities which traditionally requires manual labour in industries such as horticulture, agriculture, construction, mining, forestry, to improve productivity and therefore decrease costs.

[0008] With the foregoing in view, the present invention in one form, resides broadly in an automated vehicle comprising:

a vision system comprising at least one vision sensor to detect a terrain;

a manipulator comprising

an effector configured for performing an outdoor activity in the terrain; and

at least one arm configured to position the effector in the vicinity of a feature of the terrain

wherein the automated vehicle also comprises at least one control unit which utilises scanned terrain data from the vision system to improve movement of the automated vehicle and the manipulator during use.

[0009] The automated vehicle may be any suitable vehicle. Preferably, the automated vehicle may be any suitable powered vehicle. For instance, a suitable powered vehicle may include a tractor, a harvester, a loader, a backhoe, a bobcat or an excavator. In a particular embodiment of the invention the automated vehicle is an excavator. Preferably, the automated vehicle comprises a boom and a suitable attachment portion for removably attaching the manipulator to the boom.

[0010] In an embodiment of the invention, the vision system comprises at least one camera to detect a terrain. In a preferred embodiment of the invention, the vision system comprises at least one stereo camera to detect a terrain. It is envisaged, that in use, the camera may be used to detect a terrain or identify a features thereof, such as an object, a person, a land formation or a hazard in the terrain. In an embodiment of the invention, at least one camera may be positioned on the automated vehicle. For instance, the at least one camera may be positioned on top of the automated vehicle so as to have an unimpeded view of the surrounding terrain. In a further embodiment, at least one camera may be positioned on the manipulator so as locate or identify the position of a feature of the terrain. Preferably, the automated vehicle and the manipulator each comprise a vision system comprising at least one vision sensor to detect a terrain and/or obstacles.

[0011] In an embodiment of the invention, the vision system may comprise at least one camera to sense a feature of the terrain. The feature of the terrain may be any suitable type, such as an object, a person, a land formation, a hazard, or any suitable combinations thereof. The feature of the terrain may be natural or man made. For example, in the construction industry, the feature of the terrain may be a H-beam pile, the location where a piling hole must be drilled or a storage location in a warehouse logistics and distribution centre. For example, in the mining industry, the feature of the terrain may be a mine face to be scaled, downhole piping to be inserted into a wellbore, or a drill hole into which a wireline logging tool will be lowered. For example, in the logging industry, the features of the terrain may be a felled log to be loaded onto a truck or a truck to be refuelled.

[0012] In a further embodiment of the invention, the vision system may comprise at least one camera to sense an object to be harvested. The objects to be harvested may be of any suitable form, such as fruits, vegetables, nuts, flowers, herbs or the like, or any suitable combination thereof. In this embodiment, the vision system may comprise at least one stereo camera to sense objects to be harvested. The camera may be used to identify and/or locate and then harvest objects from amongst non-harvested objects, such as leaves and branches in the case of fruit. In an embodiment of the invention, the objects to be harvested are fruit such as avocadoes; however other types of fruit or vegetables which could be harvested by the automated vehicle include oranges, apples, melons, mangoes, and tomatoes. In one embodiment, the at least one camera may be positioned in an upper region of the automated vehicle. For instance, the camera may be positioned on top of the automated vehicle so as to have an unimpeded view of the surrounding terrain. In a further embodiment, a camera may be positioned on the manipulator so as to identify and/or locate the position of an object to be harvested. Preferably, the automated vehicle and the manipulator each comprise a vision system comprising at least one vision sensor to sense objects to be harvested in the terrain.

[0013] In an embodiment of the invention, the automated vehicle may be operated autonomously. In a further embodiment of the invention, the automated vehicle may operate semi-autonomously. In this embodiment, it is envisaged that the automated vehicle may be adapted to be controlled remotely by an operator. The operator may control the automated vehicle using any suitable technique. For instance, the operator may be provided with a controller that is connected by one or more wires to the automated vehicle. More preferably, the operator may control the automated vehicle remotely. In this embodiment of the invention, it is envisaged that a wireless connection may be provided between the operator and the automated vehicle. Thus, the operator may be located remotely to the automated vehicle. In some embodiments, a remote operator may be provided with an interface (such as a screen or other display device) that allows the operator to view and monitor the operation of the automated vehicle. In these embodiments, the remote operator may be capable of intervening in the operation of the automated vehicle at any time if, for instance, the automated vehicle is in danger of colliding with an object, is not operating optimally, is needed for a different task, has completed its task and so on. It is envisaged that the remote operator may be able to switch the automated machine between being operator controlled and being operated autonomously.

[0014] In some embodiments of the invention, the automated vehicle may be operated entirely autonomously. In this embodiment, the control unit may be provided with one or more predetermined rules relating to the terrain, features of the terrain, the effector being used and the task to be performed or the like, or any suitable combination thereof. Thus, it is envisaged that the automated vehicle may be operated solely by the control unit (and the rules contained therein). In this embodiment, it is envisaged that no human operator may be required.

[0015] In some embodiments, the at least one control unit may be adapted for connection via the internet to a control device operated by the operator. In addition, it is envisaged that data generated by the vision system during use may be adapted to be read in real-time by a remote operator. The use of a Global Positioning System (GPS) may enable the location of the automated vehicle to be known by the remote operator.

[0016] It is envisaged that the control unit may include software. The software may be of any suitable form, although in some embodiments of the invention the software is embedded software. In an embodiment of the invention, the software may include state machine software, a behaviour tree, or the like. It is envisaged that, in use, the software may provide the logic for each element of a work task, control the sequence of the tasks to be completed, control functions of the automated machine and the manipulator, and the like. Preferably, the software may be suitable for use with a machine learning system or an artificial intelligence. It is envisaged that in use, the scanned terrain data that has been collected during operation of the automated vehicle may be used in the machine learning system to improve the operation of the automated vehicle. By this it is meant that the operation of the automated vehicle may be improved by, for example, increasing the speed at which the vehicle completes a task or by improving the accuracy of the positioning of the automated vehicle relative to an object to be collected/harvested/moved etc. In addition, improved operation of the automated vehicle may include improved recognition of objects (such as fruit), improved control over the pressure applied by the effector (in order to minimise or eliminate damage to objects being handled) and/or improved placement of an object once picked up by the effector. Further, improved operation of the automated vehicle may include improved identification of and response to hazards, changes in operating conditions etc.

[0017] In a preferred embodiment, the machine learning system may be used to identify a feature of the terrain and react accordingly. For example, the machine learning system may learn to identify a person in the vicinity of the automated vehicle and to stop its operation, or move to a different area and continue operating. In another example, the machine learning system may learn to identify a fruit in a tree and learn to distinguish the fruit from other objects in the vicinity of the effector, such as leaves or branches. In a further example, the machine learning system may learn to identify a hazard in the terrain, such as dislodged rocks and learn to navigate around the hazard.

[0018] In an embodiment of the invention, data from the vision system may be used by the control unit (and embedded software where present) to improve the movement of the automated vehicle and the manipulator. In particular, data from the vision system may be used by the control unit (and embedded software where present) to improve the accuracy of position calculation of the automated vehicle and the manipulator. Data from the vision system may include scanned terrain data, sensed object data and combinations thereof. It is envisaged that data generated by the vision system may be used by the control unit (and embedded software where present) to improve movement of the automated vehicle and the manipulator during use. In a preferred embodiment of the invention, data from the vision system may be used by the control unit (and embedded software where present) to prevent movement of the automated vehicle and the manipulator during use. It is envisaged that data generated by the vision system may be used by the control unit (and embedded software where present) to avoid collision hazards in the terrain, such as people in vicinity of the automated vehicle and the manipulator during use.

[0019] It is envisaged that an operator may transfer and receive data from the control unit. Suitable data may include GPS coordinates of features of the terrain, such as objects, land formations and hazards in the terrain, training data to aid in machine learning for new tasks, such as fruit recognition, or updated instructions or rules.

[0020] In an embodiment of the invention, the automated vehicle comprises a manipulator. The manipulator may be adapted for mounting to a boom of the automated vehicle. The manipulator may be of any suitable form. For instance, the manipulator may be a mulcher, a brush cutter, a flail mower, a tilt bucket, a sieve bucket, a crusher bucket, a rock saw, a compaction plate or wheel, a ripper, tree shears, a pruning saw, a stick rake, a grapple, a gripper, a spray boom, a winch, a fuel nozzle, a drill, a pile driver, a post hole borer, a hydraulic breaker, a drilling rig, telescoping arms, a vacuum lifting system, robotic picking arm, an auger driver, or an excavator bucket or the like.

[0021] In an embodiment of the invention, the manipulator may comprise an effector configured for performing an outdoor activity in the terrain and at least one arm configured to position the effector in the vicinity of feature of the terrain. In a further embodiment of the invention, the manipulator may comprise an effector configured for harvesting the objects in the terrain and at least one arm configured to position the effector in the vicinity of an object to be harvested. By replacing the effector type it is possible to perform a different outdoor activity with the automated vehicle of the present invention. A different effector attachment does not therefore depart from the scope of the present invention.

[0022] The at least one arm may be of any suitable form. Preferably, the arm comprises an attachment portion for exchangeably fastening an effector to the arm. The attachment portion may be of any suitable form, such as a coupling or the like. It is envisaged that the attachment portion may be conventional, and no further discussion of this is required.

[0023] In an embodiment of the invention, the at least one arm may be an excavator arm. In a further embodiment of the invention, the arm may be a robotic arm. Preferably, however, the at least one arm comprises a robotic arm. Any suitable robotic arm may be provided, although in an embodiment of the invention, the robotic arm may be provided with at least 5 axes. Thus, the movement of the at least one robotic arm may have at least degrees of freedom (5 DOF). In a preferred embodiment of the invention, the robotic arm may be provided with at least 6 axes, thus the movement of the at least one robotic arm may have at least 6 degrees of freedom (6 DOF). In a further embodiment of the invention, the robotic arm may be provided with at least 7 axes, thus the movement of the at least one robotic arm may have at least 7 degrees of freedom (7 DOF). By replacing the degrees of freedom it is possible to alter its mode of operation to suit different outdoor activities.

[0024] The effector may be of any suitable form. The form of the effector may vary depending on the type of outdoor activity. For instance, in the horticulture industry suitable effectors might include: a gripper configured for harvesting fruit, a brush cutter or flail mower for mowing vegetation, a mulcher, a tree branch pruning saw, a spray boom, an auger, a stick rake and an excavator bucket. More preferably, the effector is a gripper configured for harvesting fruit and comprising at least two grip fingers for gripping the fruit to be harvested. The process of harvesting an object by the gripper comprises approaching the fruit at a respective angle of approach, gripping the object, and cutting if required with a cutter free from an attachment point on a tree. The gripper places the harvested object in a storage container (which can be positioned either on the automated vehicle or separately).

[0025] In the mining industry, suitable effectors might include: an excavator bucket, a crusher bucket, a winch, a stick rake, a grapple, a gripper, an auger, a post hole borer, a drilling rig, or telescoping arms.

[0026] In the construction industry, suitable effectors might include: an excavator bucket, a stick rake, an auger, a pile driver, a post hole borer, a drilling rig, a grapple, a gripper, or telescoping arms,

[0027] In the logging industry, suitable effectors might include: a grapple, a grapple saw, tree shears, a mulcher, or a brushcutter,

[0028] Preferably, the effector may comprise at least one sensor. In an embodiment of the invention, the sensor may be a pressure sensor configured to sense the pressure on the surface of an object when the effector contacts the object. In this manner, the effector may grip the object without applying either excessive pressure to the object (which could result in damage to the object) or insufficient pressure to the object (which could result in the object being dropped). In a further embodiment of the invention, the sensor may be a laser distance sensor to determine the position of an object accurately.

[0029] The method of performing an outdoor activity in a terrain using an automated vehicle, the method comprising: sensing the location of a feature of the terrain using the vision system; approaching the vicinity of the feature of the terrain via movement of at least one arm of a manipulator; and actuating an effector to make contact with the feature. The method may further comprise the step of using the data generated by the vision system by the control unit (and embedded software where present) to move the automated vehicle into the vicinity of the feature.

[0030] In an embodiment of the invention, the automated vehicle may also comprise at least one Light Detection And Ranging (LIDAR) laser scanner configured to survey the terrain wherein the scanned terrain data is used by the control unit (and embedded software where present) to control the movement of the automated vehicle and the manipulator. The automated vehicle has at least one LIDAR laser scanner system configured to survey the terrain and measure the distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor. Differences in laser return times and wavelengths would be used to make digital 3-D representations of the terrain. It is envisaged that data generated by the LIDAR laser scanner may be used by the control unit (and embedded software where present) in addition to the data generated by the vision system (i.e. making use of at least one stereo camera) to avoid collision hazards in the terrain, such as people in vicinity of the automated vehicle and the manipulator during use. Thus, it is envisaged that the scanned terrain may be obtained using at least one stereo camera, at least one LIDAR laser scanner, or a combination thereof.

[0031] The automated vehicle may further comprise at least one sensor. In an embodiment of the invention, the automated vehicle may comprise at least one vehicle position sensor configured to sense the position of the automated vehicle in the terrain, wherein the sensed vehicle position is used by the control unit to establish a change in position in the terrain over time to autonomously control the steering and acceleration of the automated vehicle to a particular location in the terrain. In an embodiment of the invention, the automated vehicle may comprise at least one position sensor located on the boom, wherein the position sensor is configured to sense the position of the boom in the terrain, wherein the sensed boom position is used by the control unit to autonomously control the reach and angle of the boom.

[0032] In an alternative embodiment of the invention, the automated vehicle may comprise at least one track encoder, wherein the track encoder is configured to establish the distance the machine has been driven relative to a start position via the wheel diameter or track length and the speed of the machine.

[0033] In an embodiment of the invention, the automated vehicle may comprise at least one collision sensor, wherein the collision sensor is configured to sense collision hazards in the terrain, such as people in its vicinity.

[0034] In an embodiment of the invention, the automated vehicle may comprise at least one manipulator arm position sensor, wherein the manipulator arm position sensor is configured to sense the position of the manipulator arm in the terrain, wherein the sensed manipulator arm position is used by the control unit to autonomously control the reach and angle of the manipulator arm.

[0035] The present invention provides numerous advantages over the prior art. For instance, the combination of an excavator boom to control the position of a highly dexterous, smaller robotic arm allows for greater reachability and hence yield of fruit and removes people from the risks of working at height. Additionally, the application of machine learning with vision systems for sensing the fruit and path planning for flexible control to provide for picking in various picking constraints and with reach and dexterity for a large variety of horticultural products allows the machine to pick with higher reliability and productivity. In addition, the present invention provides an alternative solution to undertaking repetitive, dangerous or manual work improving safety and improving accuracy.

[0036] In another aspect, the invention resides broadly in an automated vehicle comprising:

a vision system comprising at least one vision sensor to detect a terrain;

a manipulator comprising

an effector configured for performing an outdoor activity in the terrain; and at least one arm configured to position the effector in the vicinity of the an object in the terrain wherein the automated vehicle also comprises at least one control unit comprising embedded software in the form of at least one optimisation algorithm which utilises sensed object data from the vision system to improve the accuracy of position calculation to improve movement of the automated vehicle and the manipulator during use.

[0037] In yet another aspect, the invention resides broadly in an automated vehicle comprising:

a vision system comprising at least one vision sensor to sense objects to be harvested in the terrain;

a manipulator comprising

an effector configured for harvesting the objects in the terrain

at least one arm configured to position the effector in the vicinity of the objects to be harvested

wherein the automated vehicle also comprises at least one control unit comprising embedded software in the form of at least one optimisation algorithm which utilises sensed object data from the vision system to improve the accuracy of position calculation to improve the movement of the automated vehicle and the manipulator during use.

[0038] Any of the features described herein can be combined in any combination with any one or more of the other features described herein within the scope of the invention.

[0039] The reference to any prior art in this specification is not, and should not be taken as an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

BRIEF DESCRIPTION OF DRAWINGS

[0040] Preferred features, embodiments and variations of the invention may be discerned from the following Detailed Description which provides sufficient information for those skilled in the art to perform the invention. The Detailed Description is not to be regarded as limiting the scope of the preceding Summary of the Invention in any way. The Detailed Description will make reference to a number of drawings as follows:

[0041] Figure 1 shows a side schematic view of a an automated vehicle suitable for harvesting fruits, vegetables, nuts, flowers or herbs according to an embodiment of the present invention;

[0042] Figure 2 shows a side photographic view of an automated vehicle suitable for mowing vegetation according to an embodiment of the present invention;

[0043] Figure 3 shows a main program flowchart of harvesting fruit in an orchard;

[0044] Figure 4 shows a program flowchart for harvesting a specific fruit on a particular tree;

[0045] Figure 5 shows a front photographic view of an automated vehicle suitable for moving felled logs according to an embodiment of the present invention;

[0046] Figure 6 shows a front photographic view of an automated vehicle suitable for scraping a rock face according to an embodiment of the present invention;

[0047] Figure 7 shows a side photographic view of an automated vehicle suitable for picking up and placing a H-beam pile according to an embodiment of the present invention. The inset is an overlay of sensor data and scanned terrain data; and

[0048] Figure 8 shows the scanned terrain data of an orchard obtained by an automated vehicle according to an embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

[0049] In a first preferred form of the invention, an automated vehicle suitable for harvesting fruits, vegetables, nuts, flowers or herbs is generally indicated by arrow

100. The automated vehicle 100 comprises a mobile base machine 10 in the form of a commercially available 3 ton Caterpillar excavator.

[0050] The automated vehicle 100 also comprises at least control unit (1) which utilises scanned terrain data from the vision system to improve movement of the automated vehicle 100 during use.

[0051] The automated vehicle 100 also comprises a vision system comprising at least one vision sensor (2) to sense objects to be harvested in the terrain. The vision sensors (2) are positioned on top of the mobile base machine (10) in order for their vision to not be impaired and positioned on the manipulator controller (6) and effector (7) so as to locate the position of an object to be harvested.

[0052] The mobile base machine (10) also comprises a boom (12) and an excavator arm (14) to which a number of position sensors (4) are located, and a manipulator arm (5) to position an end effector (7) in the vicinity of an object to be harvested. It is envisaged, that for harvesting fruit, vegetables, nuts, flowers or herbs, an effector in the form of a gripper comprising at least two fingers may be used. The manipulator arm (5) is controlled by a manipulator controller (6) and control unit (1).

[0053] The automated vehicle 100 also comprises at least one light detection and ranging (LIDAR) laser scanner (3) configured to survey the terrain wherein the scanned terrain data is used by the control unit (and embedded software where present) to control the movement of the automated vehicle and the manipulator.

[0054] Another example of an effector which can be used with the present invention is shown in Figure 2 in the form of a vegetation mower is generally indicated by arrow 200. The mower effector is in the form of a flail mower 210 for vegetative management (such as clearing scrub from non-used land or pruning trees).

[0055] Referring to Figure 3, a main program flowchart for harvesting fruit in an orchard by the automated vehicle 100 is shown generally by arrow 300. The start of the embedded software program (step 31) is to map the orchard terrain initially with the LIDAR laser scanner (3) in step 32. The automated vehicle (100) is then moved to a start location in the orchard (step 33). Additional visual input from the vision sensor (2) stereo cameras allows for more accurate identification and selection of target fruit and enables the automated vehicle (100) to be moved near a tree to be picked (step 34) and then the manipulator arm (5) and effector (7) gripper to be moved to pick the fruit on the tree (step 35; see more detailed description of this process in Figure 4). After the fruit is picked, an assessment of whether all the trees have been picked is made (step 36) by the control unit (1). If no, then the process of steps (34) and (35) is repeated until all the trees in the vicinity have been picked which stops the harvesting procedure 300 (step 37).

[0056] Referring to Figure 4, a program flowchart for harvesting a specific fruit on a particular tree is shown generally by arrow 400. The start of the program (step 41) is to position the boom (12) and excavator arm (14) into an unexplored (unharvested) area of the tree (step 42). Fruit to be harvested in the particular vicinity is found in frame (step 43) mainly via use of the visual input from the vision sensor (2) stereo cameras. A decision step to determine whether the fruit in the vicinity has been found is made step (44). If no fruit has been found, then a decision step to assess whether the tree has been completely explored is made at step (45). If no, then the process is begun again by looping back to step 42 above. If yes, then the harvesting procedure 400 is stopped at step (46).

[0057] If at step (44) fruit is found, then harvesting is achieved by determining the exact position of the target fruit in a three-dimensional cylindrical coordinate system (X,Y,Z,R.P,Y) at step (47). The lowest valid hanging fruit is selected at those coordinates at step (48) which determines the grasp pose of the fingers of the gripper end effector (7) at step (49) to grasp the fruit. The fruit is then picked at step (50) by either moving the grasped fruit away from its vicinity to pull the fruit from its vegetative attachment point on the tree or by manoeuvring a cutter to cut the fruit to be harvested from a tree. The harvested fruit is then placed in a storage container on the automated vehicle 100 or alternatively, in an external storage container.

[0058] An example of an automated machine according to an embodiment of the present invention is shown in Figure 5. The automated machine 500 comprises an effector 510 in the form of a bucket and thumb for moving felled logs.

[0059] An example of an automated machine according to an embodiment of the present invention is shown in Figure 6. The automated machine 600 comprises an effector 610 in the form of an excavator bucket suitable for excavating soil to load on a truck or scraping a rock face.

[0060] An example of an automated machine according to an embodiment of the present invention is shown in Figure 7. The automated machine 700 comprises an effector 710 in the form of an articulated side grip suitable for picking up and placing a H-beam pile. Inset 720 is an overlay of sensor data and scanned terrain data used by the control unit (not shown) to improve the movement of the automated vehicle and manipulator and align the effector with the H-beam pile.

[0061] In Figure 8, the scanned terrain data 820 of an orchard 810 obtained by the vision system (not shown) of an automated vehicle 800 is illustrated. It is envisaged that in use, data from the vision system, including scanned terrain data, sensed object data and combinations thereof, may be used by the control unit (and embedded software where present) to improve the movement of the automated vehicle and the manipulator. The control unit may use the scanned terrain data to map the most efficient path between and to features in the terrain, such as objects, land formations and hazards in the terrain.

[0062] In the present specification and claims (if any), the word 'comprising' and its derivatives including 'comprises' and 'comprise' include each of the stated integers but does not exclude the inclusion of one or more further integers.

[0063] Reference throughout this specification to 'one embodiment' or'an embodiment' means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearance of the phrases 'in one embodiment' or'in an embodiment' in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more combinations.

[0064] In compliance with the statute, the invention has been described in language more or less specific to structural or methodical features. It is to be understood that the invention is not limited to specific features shown or described since the means herein described comprises preferred forms of putting the invention into effect. The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims (if any) appropriately interpreted by those skilled in the art.

Claims

1. An automated vehicle comprising: a vision system comprising at least one vision sensor to detect a terrain; a boom mounted on the automated vehicle and movable relative thereto; a manipulator located on the boom and movable separately to the boom, the manipulator having at least 2 degrees of freedom; and a replaceable effector attachable to the manipulator and positionable by the manipulator, the effector configured for performing an outdoor activity in the terrain; wherein the automated vehicle also comprises at least one control unit which utilises scanned terrain data from the vision system and which is configured to control movement of the automated vehicle, the boom and the manipulator during use.

2. The automated vehicle as claimed in claim 1, wherein the vision system comprises at least one stereo camera.

3. The automated vehicle as claimed in claim 1 or 2, wherein the automated vehicle further comprises at least one LIDAR laser scanner configured to survey the terrain.

4. The automated vehicle as claimed in claim 3, wherein the scanned terrain data generated by the at least one LIDAR laser scanner is used by the control unit to control the movement of the automated vehicle, the boom and/or the manipulator.

5. The automated vehicle as claimed in any one of the preceding claims, wherein the automated vehicle further comprises at least one sensor.

6. The automated vehicle as claimed in claim 5, wherein the at least one sensor may be a position sensor located on the automated vehicle, the boom, or on the effector arm of the manipulator, and combinations thereof.

7. The automated vehicle as claimed in claim 5, wherein the at least one sensor is a collision sensor located on the automated vehicle, the boom, or on the effector arm of the manipulator, and combinations thereof.

8. The automated vehicle as claimed in any one of claims 5 to 7, wherein the control unit uses data generated by the at least one sensor to improve accuracy of a position calculation of the automated vehicle and/or the manipulator during use.

9. The automated vehicle as claimed in any one of claims 5 to 7, wherein the control unit uses data generated by the at least one sensor to control the movement of the automated vehicle, the boom and/or the manipulator during use.

10. The automated vehicle as claimed in any one of claims 5 to 7, wherein the control unit uses data generated by the at last one sensor to control the reach and angle of the boom of the automated vehicle and/or the effector arm of the manipulator.

11. The automated vehicle as claimed in any one of claims 1 to 10, wherein data collected while performing an outdoor activity in a terrain using an automated vehicle is used in a machine learning system to improve the operation of the automated vehicle.

12. The automated vehicle as claimed in any one of the preceding claims, wherein the feature of the terrain is an object to be effected.

13. The automated vehicle of claim 12 wherein the object to be effected comprises fruits, vegetables, nuts, flowers or herbs.

14. The automated vehicle as claimed in any one of claims 1 to 11, wherein the feature of the terrain is a mine face, a wellbore, a drill hole or a combination.

15. The automated vehicle as claim in claim 14, wherein the mining vehicle is a tractor, a loader, a backhoe, a bobcat, or an excavator

16. A method of performing an outdoor activity in a terrain using an automated vehicle according to any one of the preceding claims, the method comprising: a. detecting a terrain using at least one vision system; b. sensing the location of a feature of the terrain using the vision system; c. approaching the vicinity of the feature of the terrain via movement of the manipulator; and d. actuating the effector to make contact with the feature.

17. The method as claimed in claim 16, further comprising using a control unit to process the detected terrain data and to move the automated vehicle into the vicinity of the feature of the terrain.

18. The method as claimed in either claim 16or 17, further comprising using a control unit to process data generated by a LIDAR laser scanner and to move the automated vehicle into the vicinity of the feature of the terrain.

19. The method as claimed in any one of claims 16 to 18, further comprising using a control unit to process data obtained from at least one sensor located on the automated vehicle, the boom on the automated vehicle, or on the effector arm of the manipulator, and combinations thereof, to improve the accuracy of a position calculation of the automated vehicle and/or the manipulator during use.

20. The method as claimed in any one of claims 16 to 19 further comprising using data collected while performing an outdoor activity in a terrain using the automated vehicle in a machine learning system to improve the operation of the automated vehicle.

I

12 14 :

2 7

4

5 3 6

4

Figure 1

Figure 2

Start 300

32 Map Orchard()

33 Move Robot To Start Location

34 Move Robot Near Tree To Be Picked

35 Pick Fruit On Tree()

36 Have All Trees No Been Picked?

Yes

37 Stop

Figure 3

41 Start

400 42 Position Main Arm Into Unexplored ROI Of Tree

43 Find Fruit In Frame

44 Yes No Found Fruit?

45 X.Y.Z. R. P. Y Coordinates

46 Select Lowest Valid Hanging Fruit

47 Determine Grasp Pose

48 Pick Fruit No Has Tree Been Completely Explored?

49 Place Fruit 50

Yes

51 End

Figure 4

Figure 5 510

600

610

Figure 6

Figure 7 720

810

800 820 Figure 8