JP7786348B2 - Harvesting vehicle - Google Patents

Description

This invention relates to the analysis of field crops and field maps using harvesting vehicles equipped with an imaging device and a satellite positioning system, enabling autonomous driving.

There are harvesting vehicles that are equipped with an imaging device and have the function of detecting whether crops have fallen over.
(Patent Document 1).

Japanese Patent Application Laid-Open No. 2022-2475

Conventional technology uses the height of the crop as a reference point and detects the state of the crop in a lodging state by measuring the width of the area that extends at the same height.

However, when an imaging device captures an area exceeding 20 m from a work vehicle, it is possible to detect crop height over a wide range, but due to the imaging angle, it may not be able to detect partially fallen areas, or it may be misdetected due to noise from weeds, etc.

This invention improves the accuracy of determining lodging by analyzing color shades from image data and using artificial intelligence to compare the color sequence with reference data. The objective is to estimate the degree of lodging during perimeter harvesting at the beginning of work and determine the work process, estimate areas that cannot be photographed, and measure the condition of the field more accurately by incorporating additional data as work progresses.

The first invention is solved by the following technical means.

The weight and moisture of the crop are measured when the periphery of the field is harvested, the direction change position of the work vehicle is set to the corner of the field, and the estimated weight and moisture of the crop in the unharvested area and the unharvested area are added to the registered field map data ;
an imaging device mounted on a work vehicle photographs the field at predetermined intervals, converts the arrangement data of green and brown areas in the photographed images into a line image to determine the state of lodging of the crops, and adds the areas that will fall from the lodging state to the field map data;
adding a route from the lodging area to farm field map data;
The farm field map data is used as a reference data for automatic driving .

According to the present invention , by determining the work schedule by estimating the degree of lodging during the perimeter picking at the beginning of the work, it becomes easier to create work schedules and work routes for separating and harvesting high-quality crops.

FIG. 1 is an external view of a harvesting vehicle according to an embodiment of the present invention. A field map and information screen diagram of the harvesting vehicle of the present invention Work flow diagram of the harvesting machine of the present invention using a field map A block diagram of the image data state change of the imaging device of the present invention. A block diagram showing a state change due to the combination of image data of the imaging device of the present invention. Image of the lodged crop of the present invention Image of a fallen crop analyzed using the RGB color model of the present invention and converted into a dot image Image of the dot image of the fallen crop obtained by averaging and linear processing Image of the present invention's unlodged crop Image of unlodged crop analyzed using the RGB color model of the present invention and converted into a dot image Image of unlodged crops averaged and linearly processed using the dot images of the present invention A diagram showing how the state of data from an imaging device is changed based on the judgment of artificial intelligence using a neural network according to the present invention. Schematic diagram of how the unimaged region of the present invention is created from the captured image. Schematic diagram of how the unimaged area of the present invention is created from past image data The field map of the present invention shows the state of lodging of crops in polygonal form.

The present invention will now be described based on the embodiments shown in the drawings.

The harvesting vehicle shown in Figures 1 to 15 shows an example of this embodiment.

This article describes the harvesting vehicle of the present invention.

In FIG. 1, a combine harvester 1, as an example of a harvesting work vehicle according to an embodiment of the present invention, has a vehicle body 2. A pair of left and right traveling devices 11 are provided below the vehicle body 2. The traveling devices 11 in this embodiment are, as an example, configured as endless tracks, so-called crawlers. A riding section 12 on which an operator can ride is provided on the front right of the vehicle body. A harvesting device 13 that harvests crops in the field is located at the front of the vehicle body. A transporting device 14 that transports harvested grain is located behind the harvesting device 13. A thresher 16 that threshers the grain transported by the transporting device 14 is located behind the transporting device 14. A grain tank 17 that stores grain processed by the thresher 16 is located to the right of the thresher 16. A discharge device 18 that discharges grain from the grain tank 17 to a container on a truck outside the field is connected to the rear of the grain tank 17. A straw discharge device 19 that discharges straw is located at the rear of the vehicle body.

The combine harvester 1 of this embodiment is equipped with a satellite positioning system as an example of a current position measurement unit. The satellite positioning system receiver 41 is installed on the top surface of the riding section 12. In Japan, the Global Navigation Satellite System version of the satellite positioning system is called GNSS, so hereafter it will be referred to as GNSS.

The receiver 41 can receive signals from GNSS satellites and measure the current position of the combine harvester 1. Therefore, the combine harvester 1 of this embodiment can use GNSS to travel autonomously (automatic travel, unmanned travel), or it can be operated by an operator on the riding section 12 (manual travel, manned travel). During autonomous travel, the operator operating a terminal capable of wireless communication with the combine harvester 1 can be outside the combine harvester 1 (outside or inside the field), or can be on board the riding section 12 while carrying the terminal.

The combine harvester 1 of this embodiment is equipped with an inertial measurement unit (IMU). This IMU 22 uses a three-axis gyro and a three-directional accelerometer to calculate three-dimensional angular velocity and acceleration, enabling the vehicle's heading and tilt to be determined. This can be used to correct the satellite positioning system mentioned above, allowing for more accurate measurement of the vehicle's position.

In the combine harvester 1 of this embodiment, obstacle sensors 31, 32, and 33, which serve as an example of crop detection means, are installed at the front and on both the left and right sides of the vehicle body 2. The obstacle sensors 31 to 33 are capable of detecting crops and obstacles in front of and on both the left and right sides of the vehicle body 2. Note that while obstacle sensors have been used as an example of crop detection means, this is not limiting, and imaging devices such as cameras can also be used.

The combine harvester 1 of this embodiment is equipped with an imaging device 21, capable of capturing 360-degree images in all directions. The imaging device 21 is equipped with LIDAR. LIDAR is a type of optical remote sensing technology that measures scattered light in response to pulsed laser irradiation and can analyze the shape and properties of an object. It also enables measurement of the distance to an object, and can measure distances from long to short distances and the object's movement speed. This makes it possible to distinguish between people, other animals, and fixed objects. The imaging device 21 can also use millimeter-wave radar. This has the advantage of being able to identify objects even during early morning work in conditions with morning fog, sudden weather changes, backlight, or when the combine harvester is in the shadow. Furthermore, a device such as a CCD camera that converts light from the subject into an electrical signal using optical methods such as a lens may also be used. The imaging device 21 is equipped with any of these sensors, or a combination of these sensors, and can measure the shape, color, movement, and distance of objects.

Explain the background of this invention.

When harvesting vehicles are manned, the degree of lodging of crops in the field is estimated based on human visual inspection and experience. This may require changes to the work process.

For example, if there is a large amount of lodging in the center of a field, you need to determine the lodged area from the overall field area, estimate the amount that can be harvested, and decide how best to plan the harvesting process when dividing the field based mainly on the lodged area. This involves deciding whether to proceed with the usual perimeter mowing, or to make a middle cut near the lodged area and mow the lodged area all at once.

However, unmanned robotic harvesting vehicles cannot make this judgment, so they work in fields without distinguishing between the same type of crop. This results in crops of poor quality being mixed in due to lodging.

With this invention, by calculating the lodging state of crops in the field early on during edge cutting, the harvesting process can be planned taking into account subsequent transport vehicles and dryer classification at the drying and processing facility. Furthermore, by gradually increasing accuracy as the work progresses, it becomes possible to select and harvest crops of higher quality, and even with unmanned operation, harvesting work can be carried out while taking human judgment into account.

This section explains the first invention, which adds unharvested areas and estimated crop weight and moisture data for unharvested areas to field map data.

A map of the corresponding field is pre-registered on the harvesting vehicle. This field map data may be stored on the vehicle's terminal or in the cloud. The field map data also registers the field's perimeter position in latitude and longitude, and the location of the work vehicle on the map can be displayed based on GNSS and IMU data.

A typical harvesting vehicle begins by mowing the perimeter of the field. The work vehicle of this invention is a robotic work vehicle that allows for unmanned operation, and begins its work process with a similar process. To enable automatic operation, this work is performed by remotely or mannedly driving the mowing work on the perimeter of a field where there are many obstacles, determining a safe basic route, and storing and registering turning positions and turning areas to determine the route for unmanned operation. Another effect is that the mowing process on the perimeter clearly indicates the unharvested area. This area is determined by enclosing it on the perimeter. Normally, the area inside the enclosure is considered the unharvested area, but it is also possible to switch between inside and outside.

The harvesting machine's grain tank is equipped with a weight scale and moisture meter, which can detect the weight of the grain when it is harvested in the outer periphery. The work vehicle's traveling speed and distance are measured, and the harvested area can be calculated from the work vehicle's harvest width. This makes it possible to convert the weight per unit area of the harvested grain in the outer periphery. In this way, the weight of the crop in the outer periphery, the weight of the unit area, the moisture and average moisture at each point in the outer periphery can be calculated from the harvested grain in the outer periphery, and it is also possible to estimate the estimated crop weight in the unharvested area, as well as the moisture and moisture deviation at each point in the unharvested area.

By measuring the crop weight and moisture content when harvesting the outer perimeter, and registering the weight and moisture content of the unharvested areas as field map data once the outer perimeter is harvested, the data can also be uploaded to the cloud. At drying and processing facilities, the weight and moisture content of the crop in the unharvested areas can be used to select the dryer to load after harvesting in advance. For example, if the weight of the unharvested area is estimated to be 3 tons, and there are areas with moisture contents of 23% or more and 20% or less, estimated at 2 tons and 1 ton, respectively, rather than loading the crop all at once into a 30-koku dryer, if there is one vacant 20-koku dryer and one 10-koku dryer, harvesting can be done so that the crop is distributed in this ratio, reducing fuel costs for the dryers and allowing quality to be differentiated by classification.

In addition, by registering the work vehicle's direction change position as a corner, it becomes the reference point for turning during automatic driving, and by turning inside that point, it can prevent the ridge from being damaged by the driving part, avoid obstacles at the edge of the ridge, and can also be used to determine the turning direction and moisture measurement points for the entire field. In particular, if you want to register the outer perimeter for center cutting from the start, registering the direction change position as a corner is an effective setting.

Figure 2 shows a field map and information screen of the harvesting vehicle. The field to be worked on is selected. 42 shows an illustration of operating a mobile device, but similar operations can be performed using the display screen equipped on the work vehicle. Since this invention is based on unmanned vehicles, explanations will be given using a mobile device.

Reference number 42 is an image of the field at the time when harvesting of the outer perimeter has been completed. Image acquisition by the imaging device and calculation of images and information data of unharvested areas will be explained later, but the system has all image data and information data of the field, including estimation calculations, at the time when harvesting of the outer perimeter has been completed.

Image 42, as shown in Figure 15, displays the fallen areas of the entire field. These fallen areas are displayed as polygonal dots, with the color changing depending on the degree of lodging. For example, when the user touches the location of a fallen area with their finger, the information corresponding to that area is read out. In other words, when the outer periphery is harvested, various pieces of information are registered in conjunction with the location information. Image 43 is an image of the fallen area when it is actually photographed. By looking at this image, the user can confirm the condition of the field. Because work is done at high speed, some areas may not be fully visible, so the image data is useful information. Work vehicles take sequential images with their imaging devices and analyze their traveling position and image data to create a map of the field as seen from above.

44 is the content calculated from field information. It is possible to calculate the degree of lodging, and for example, the lodging rate is calculated and displayed from the area of the entire field on a three-level scale: weak, medium, and strong. This information is linked to the corresponding area, and selecting the area number will display detailed data for the corresponding area.

45 displays overall field information. It displays the weight per unit area based on information from when the outer perimeter was harvested. It also calculates and displays the average moisture content. It estimates the unharvested area and calculates the estimated weight of the unharvested portion from the information from the outer perimeter harvest. Because this information is available at the time the outer perimeter was harvested, it is useful for later transporting the crop and preparing post-harvest equipment.

46 is the settings screen for the work that will follow. Since the present invention is about dealing with fallen areas using an imaging device, the settings related to this will be explained, but there are many associated settings. You can set how to deal with fallen areas, change the harvesting process, perform intermediate harvesting, and create images of unharvested areas.

Figure 3 explains the series of steps involved in adding unharvested areas and estimated crop weight and moisture data for unharvested areas to field map data.

This content explains the system of the present invention that determines the lodging state of crops, adds information to field map data, and maps the lodging state within unharvested areas.

For harvesting vehicles, it is desirable to start harvesting crops in a field by performing edge cutting (S3-1). By addressing the corners of the field as described above, the field map is divided into areas, making it possible to identify unharvested areas. This can be set by the user, but since the center of the field is often registered on the field map, the process of S3-2 can be determined automatically. If S3-3 is determined, this is a difficult state for autonomous driving, and a decision must be made by the user. Conversely, comparing the center position of the field map with registered data makes it possible to determine where the unharvested area is located relative to the current position of the implement, allowing for immediate autonomous driving.

Reading the field corners (S3-4) enables the calculation of the unharvested area (S3-7). The weight of the perimeter cut (S3-5), the moisture content of the perimeter cut, and the aforementioned running performance of the combine are used to calculate the harvest yield per unit area (S3-8). This allows the weight of the unharvested area to be calculated (S3-9), enabling post-harvest support based on the estimated weight.

Unharvested areas include areas where the crop has fallen, and if you select harvesting (S3-10) to distinguish these areas as fallen, the appropriate storage tank classification calculation (S3-11) is performed as a corresponding operation. This operation involves harvesting the unharvested areas separately, but this serves as the basic data for calculating the driving route (S3-11) based on the estimated volume of the fallen areas and the size of the storage tank.

Meanwhile, there is a process of creating field map data from this unharvested area. A field map basically contains the latitude and longitude of the outer perimeter and the center of the field. Field map data is constructed by adding data to this.

When the imaging of the perimeter mowing begins (S3-17), the imaging data is registered and associated with the location where the image was taken on the field map. It is compared with the satellite positioning system and registered as polygon data on the field data map as the first mapping.

Figure 15 shows an example of a screen displaying areas detected as lodged in the unharvested area. If you want to know the condition of the field, just touch the desired location on the screen, and the data associated with each area on the image will be read out. At the time the data was acquired, you can read out images of the crops, moisture, air temperature, soil temperature, weather, the degree of lodging as judged by the image, and the state of harvest of the rice ears. In addition, an overall image is displayed showing the percentage of lodged areas in the entire field, the moisture distribution of the entire field, and moisture deviation.

The degree of lodging in this field can be determined using image processing, as described below. A simple classification divides the degree of lodging into three levels: light lodging, moderate lodging, and severe lodging. This is done by calculating the angle of the lodged stalks and the height of the rice ears from the field.

Returning to the explanation of Figure 3, the location and area of the fallen area can be estimated by creating data in the detection of fallen areas S3-18. While the inner area may not be clear when only the outer perimeter is mowed, this can be estimated using the image processing capabilities of the present invention, which will be described later, and the data is added to the field map data as the first map and growth registration S3-19.

In addition, the storage tank's appropriate classification calculation (S3-11) adds data to a second map showing the route that would be taken if harvested using normal procedures, and to the yield registration (S3-12). Data such as the estimated weight of the entire field, the degree of lodging, and the area and ratio of lodging are added to the field map.

Furthermore, the planned driving route for actual harvesting is calculated (S3-13) and registered as a third map (driving route registration (S3-14)). The field map is then added with the information data from the first, second, and third maps to create a fourth map (category registration (S3-20)), and this field map data is sent to the cloud 226 and provided to the user.

The user checks this field map data and confirms that there are no problems with the driving route or harvesting area, and then confirms the registration details (S3-15). Once each map has been corrected (S3-16), a field map is created and registered (S3-21), and the edge harvesting work is complete. From this point on, automated driving will begin based on the registered field map data. The registered field map overlays the harvested crop quality and field area, and remains as image data even after harvesting is complete, and can be used as basic data for the next season's cultivation. By doing this for each field, a user database will be built.

Figure 4 is a block diagram showing the use of images from an imaging device.

The imaging device performs color analysis 50, shape analysis 60, and height (distance) analysis 70 of the crops. However, because images contain a lot of mixed information, much of the data is unnecessary. Masking is inevitably required, and each analysis calculation determines the amount of masking to remove anything outside the standard. In this invention, this corresponds to crop distance analysis 90. A standard for color tone is also set for greens, and a determination is made based on saturation and density to determine whether it is crop or weed. Weeds are masked as noise. The same judgment is made for browns as for greens, and they are determined to be damaged rice or rice malt, which is then masked. Masking is based on length, as shown in 91 and 92.

The black area at position 93 is used to mask the image of the field. By assuming that the black area at the bottom is capturing an image of the field and masking it from the image, it becomes possible to analyze the crop image more accurately. When determining whether the crop has fallen, one method is to remove noise by using magnification and a wide angle. In this invention, the masking process is a method in which unknown information is not taken into account in the calculation, as will be explained in a later section.

Since image data is difficult to handle, image processing is performed to replace it with an area. This area can be represented by a collection of small rectangles, and the entire area is displayed as a polygon. The outer periphery is not a smooth line type, but the level of analysis is within a range of resolution that is acceptable for field analysis.

The color analysis 50 determines green polygon areas to identify stalks and brown polygon areas to identify rice ears. The imaging data is from an imaging system that can display green polygon areas as dots, and it is also possible to display linear color analysis in gradually different shades. The dot display of brown polygon areas has a similar function.

Even when capturing images from the same position, the imaging device compares both the enlarged and wide-angle images to remove noise caused by lighting conditions and clearly capture the arrangement of greens and browns. This corresponds to the controls at 52 and 53 in Figure 4. A basic mapping of the fallen state is registered, and whether or not the plant has fallen is determined based on the degree of match with this mapping data.

Shape analysis 60 analyzes the shape of the crop. Color analysis 50 primarily analyzes the arrangement of brown and green tones, which allows for a general assessment of the lodging state of the crop, while analyzing the relative position of the color in the field. The imaging device can measure distance, making it possible to determine the position of the rice ear and stalk. Figure 4 shows two imaging devices, 21 and 23 in Figure 1. The difference in these positions is used to determine the three-dimensional shape of the object. Rice Ear Position Confirmation 1 and Rice Ear Position Confirmation 2 confirm the shape of the rice ear and analyze its position from the field. At the same time, plant height 1 and plant height 2 determine the shape of the stalk and the shape of the weed, allowing the height of the stalk from the field to be measured. It is also possible to determine the direction the crop is facing from the position of the stalk and rice ear. By installing multiple imaging devices in this way, the depth of the object can be determined from differences in the imaging angle, and noise caused by lighting can be prevented, making the above measurements easier.

The field data 80 includes crop moisture at the time of harvest, the air temperature around the field, and the soil temperature of the field. A comprehensive analysis of the color analysis 50 and shape analysis 60 data makes it possible to estimate and calculate the degree of lodging 95, lodging area 96, and grain quality 97.

The method for changing the image state for the lodging analysis in the captured image is explained in Figures 6 to 11, and the analysis method using artificial intelligence (AI) with a neural network, which is a method for changing the input and output state of each sensor to change the state, is explained in Figure 12.

This article describes a system that compares data on the arrangement of green and brown areas of the present invention, determines the lodging state of crops, adds information to field map data, and maps the lodging state within unharvested areas.

When harvesting the periphery of a field, an imaging device mounted on a work vehicle takes images of the field at predetermined intervals, and the image data is displayed as color-coded dots to determine the condition of the imaged crop.

Figure 6 shows the state of rice plants lying down as captured by the imaging device 21. The rice stalks 101, 102, and 103 are tilted horizontally relative to the ground, while the rice ears 111, 112, and 113 are oriented vertically relative to the ground. Similarly, weeds 104 and 105 in the surrounding area often fall sideways as they are pulled along by the fallen rice plants.

This image data was analyzed using an RGB color model and converted into a dot image, as shown in Figure 7. As the drawing cannot be displayed in color, we will explain it in words. The rice stalks 101A, 102A, and 103A are green dots and are tilted horizontally relative to the ground, while the rice ears 111A, 112A, and 113A are brown dots and are oriented vertically relative to the ground. Weeds 104A and 105A are green dots and are tilted horizontally relative to the ground.

The wavelengths of light that indicate stalks and weeds are green and those that indicate rice ears are brown, and they are easy to distinguish. They also differ from the black of the ground in the field, and by analyzing several images, misrecognitions and inability to analyze are greatly reduced.

Figure 8 shows the dot image data after averaging and linear processing. This processing makes the relationship between the stalks and ears of rice clear. Rice stalks 101B, 102B, and 103B are green lines that are tilted horizontally relative to the ground, while ears of rice 111B, 112B, and 113B are brown lines that are oriented vertically relative to the ground. Weeds 104B and 105B are green lines that are tilted horizontally relative to the ground. From this linear image, the intersection angles α1 and α2 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

Figure 9 shows rice plants captured by the imaging device 21 in an unlodged state. The rice stalks 101C and 103C extend vertically relative to the ground, and the rice ears 111C and 113C also face vertically relative to the ground. Similarly, the surrounding weeds 104C also extend vertically.

This image data was analyzed using an RGB color model and converted into a dot image, as shown in Figure 10. Rice stalks 101D and 103D are green dots that extend vertically relative to the ground, while rice ears 111D and 113D are brown dots that also extend vertically relative to the ground. Weeds 104D are green dots that extend vertically relative to the ground.

Figure 11 shows the dot image data after averaging and linear processing. This processing makes the relationship between the stalks and ears of rice clear. Rice stalks 101E and 103E are green lines that extend vertically relative to the ground, while ears of rice 111E and 113E are brown lines that extend vertically relative to the ground. Weeds 104E are green lines that extend vertically relative to the ground. From this linear image, the intersection angles α3 and α4 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

As explained in the overview of image state changes, it is possible to convert the arrangement data of green and brown areas into a linear image, detect the intersection angle of the green and brown lines, and calculate the degree of lodging based on the intersection angle. In this way, by breaking down and analyzing image data into individual elements and comparing the relationships between each piece of analyzed information with existing data information, it is possible to perform calculations that establish different information judgments, thereby making it possible to make judgments that are closer to human empirical judgments. As an example of how the present invention determines the lodging state of crops, a judgment using artificial intelligence using the neural network in Figure 12, and a decision to change the state of the image data, is an effective strategy.

Figure 12 explains the details of a second invention in which the arrangement of green and brown areas in an image captured by the imaging device of the present invention is calculated using a state change system, and the arrangement of green and brown areas is compared with pre-registered data on the arrangement of green and brown areas to determine the lodging state of the crop.

In this embodiment, three imaging devices are installed. Each imaging device has similar functions, but by utilizing differences in installation position, the accuracy of the three-dimensional shape and color analysis of the image can be further improved. The main analysis data is differentiated, and by using structural and data processing differences in the noise removal method, the accuracy of the image analysis can be further improved.

The data from the first imaging device 21 and obstacle sensors 31, 32, and 33 is mainly processed by the crop size detection unit 201. Element-specific data from the imaging device and element-specific data from each obstacle sensor are combined and judged in a new way to control the status change to information that can determine the size of an object. As a first status change, the information content status changes to the overall crop height 203, the overall crop width 204, the stalk height 205, and the position of the crop part (fruit), which in this example corresponds to the ear of rice, 206.

The second imaging device 23 mainly consists of a crop color detection unit 202. By combining and judging new elemental data from the imaging device, it controls the state change to information that can determine the color of an object. As a first state change, the information content changes state to green direction 207, brown direction 208, masking amount determination 209, and crop density 210.

As a result of these state changes, the image state of the image changes from Figure 6 to Figure 8, and from Figure 9 to Figure 11, and is handled by the decision unit 211.

As described above, the reference data for comparison is registered in the data storage unit 216, and image matching is performed by comparing 211 and 216 and matching similarities. The first state change units 201 and 202 then recalculate whether the degree of similarity is within the range of image state changes, resulting in a determination of whether the data has fallen 202, a determination of the area that has fallen 213, a determination of the degree of falling 214, and an estimate of quality 215. This second state change is a combination of state change, matching, and judgment, in which a new combination of element data is compared with the registered reference data, and is the result of so-called artificial intelligence functions.

This function allows the work route change and determination unit 217 to select an ideal route, which is then reflected in the automatic driving function as the work is carried out. However, for unmanned operation, the work route must be determined by comparing it with various setting data.

External data includes how to revise the pre-set route setting 228. Field map data resulting from the harvesting of the outer perimeter is also used as the environmental map 227. This information is exchanged with the user via the communication unit 229 and cloud 226. The third imaging device 24 also analyzes the condition of the field in front of the work vehicle and the condition of the harvesting unit, and data from the satellite positioning device 22 is also added to determine whether to change the work route, and the determination unit 217 makes the decision.

The artificial intelligence in this embodiment is responsible for controlling the traveling unit 218 and working unit 221. By determining the work route 217, the traveling speed 219 and traveling direction 220 are determined and travel begins. Furthermore, by determining the working height 222, working unit inclination 223, working unit rotation speed 224, and harvesting unit control 225 as a work machine, unmanned operation using artificial intelligence becomes possible.

This section describes a third invention, which is a method for creating image data of areas that cannot be captured because they are beyond the imaging range of the imaging device.

Figure 13 shows the relationship between the harvesting vehicle 250, the imaging device's imaging range 302, and the field 240. The harvesting vehicle proceeds with circumferential mowing in a counterclockwise direction, moving in the following order: 250 ⇒ 251 ⇒ 252 ⇒ 253 ⇒ 254 ⇒ 255 ⇒ 256 ⇒ 257.

The white arc is the area that the imaging device can reliably capture. While it is possible to capture much of the travel route by increasing the number of captures, this invention utilizes a method of stretching and utilizing existing captured data. For example, once imaging of area 301, the forward area that has been captured, is completed, an image gradation is created, viewed from above within 301. Using this arrangement, the data for 301A, 301B, and 301C is expanded at the same ratio to create area 302. As the harvesting vehicle travels from 251 to 252, it can capture actual images of area 302, allowing this actual image to be compared with image 302. This data comparison calculates the difference. If there is a difference in the ratio of the degree of lodging, the ratio is calculated. The moving average of these ratios is calculated to derive a coefficient for correcting the estimated image, such as 302. The neural network function shown in Figure 12 is also used to calculate this correction coefficient. The image state change and determination unit 211 corresponds to the estimated image 302, and the data storage unit 216 corresponds to the actual captured image. This second state change calculation is used to determine the degree of falling 212, the fallen area 213, the degree of falling 214, and the quality estimation 215, while also correcting the image from above 302.

This captured image is corrected to create an estimated image of the uncaptured area using a technique for creating an estimated image of the uncaptured area. As explained in Figure 13, a correction coefficient is acquired from the image data of 302, and the image gradation above 303A, 303B, and 303C is used in the captured data of area 303 to create image 304. However, the correction coefficient is also used to correct not only the image, but also the data for determining the degree of falling 212, determining the fallen area 213, determining the degree of falling 214, and estimating quality 215.

This function creates estimated image data for unimaged areas from image data captured based on this correction coefficient, and accuracy can be improved by sequentially overlaying the estimated data. It also has a learning function that repeatedly calculates, corrects, and registers changes made to the image or fallen data, weighting them more frequently.

By using the first and second state changes shown in Figure 12, the data stored in the imaging device, obstacle sensor, and data storage unit 216 can be compared, collated, removed, analyzed, and modified to determine the state of crops in the field, something that was previously determined by human vision and experience.

Furthermore, as an expanded function, the traveling speed 219 and traveling direction 220 of the traveling unit 218 are controlled. The working unit control 221 controls the working unit height 222, working unit inclination 223, working unit rotation speed 224, and the output of the harvesting unit control 225.

In conventional technology, the height of the crop, for example, is determined from distance information and image data from each imaging device and information from obstacle sensors, and the control directly outputs the driving unit control 218 and the working unit control 221. However, in this invention, a first state change and a second state change are performed before the driving unit control 218 and the working unit control 221 are output, which not only improves control accuracy but also makes it possible to change the working path while working by obtaining detailed information about the state of the crop, and the output control is changed based on these changes.

Therefore, the first and second state changes can only be analyzed through judgment by artificial intelligence, and we believe that the travel unit control 218 and working unit control 221, as well as other unique controls, utilize judgment control by artificial intelligence using neural networks.

The fifth invention explains the control leading up to the aforementioned output, detecting the size of the crop from the imaging device 21, detecting the color of the crop from the imaging device 23, performing RGB element analysis of the image, and determining the lodging area of the crop 213, the degree of lodging 214, and quality estimation 215 based on the arrangement of greens and browns in the image, and comparing the results with the data registration unit 216 to control the traveling unit 218 and working unit 221.

In the fourth invention, which involves overlaying an image on previous data, weighting the dot data of the overlapping areas toward the side with higher image accuracy, and combining them to estimate the degree of lodging, weight per unit area, and moisture content of each unharvested area, in addition to the explanation in the previous section, the correspondence of areas where image data overlaps is explained in Figure 13.

In Figure 13, this corresponds to area 314. Although the image was taken from different directions, it is the same area, and both captured images can be referenced. However, the polygonal image viewed from above must be synthesized. In this case, both images are added together and weighted to the image with higher image accuracy. In Figure 13, image 314 is an image of the work vehicle 257 in front of the imaging device, and as it has higher accuracy, synthesis processing is performed with weighting given to the image in front of the work vehicle 257.

This section explains how to handle past data when the imaging device's shooting range is exceeded.

The area that has actually been photographed is used to create estimated image data. However, when the area far exceeds the range that can be estimated, reliability becomes low, so the approach shown in Figure 14 is taken. This method involves reusing past data. Last year's field data is registered in the data storage unit, and past image data is used to provisionally display areas that have not yet been photographed. This past image data includes data for estimating the degree of lodging, weight per unit area, and moisture content.

By touching an uncaptured area on the image, users can choose between estimated image data or past image data. Based on the image data they select, the system has a function that displays the degree of lodging, weight per unit area, and moisture content. Users can then use this estimated data to change the route or adjust harvest classifications.

If no previous image data exists, the data will be displayed blank, or data close to the relevant area will be used, with a message indicating that the data accuracy is low.

Image data from these imaging devices cannot capture the entire field in one image; the image data must be combined. Figure 5 is a block diagram showing a standard image combination. Fields are often rectangular, with four sides and a square. Therefore, by using the data for the square and the center data for the four sides, it is possible to capture a minimum amount of field data. A simple technique for combining these images is to change the color to that seen from above, and by combining the images while performing RGB color analysis between the square and the four sides in order, it is possible to create an aerial image of the field using imaging devices attached to a work machine on the ground.

Figure 5 outlines the steps for combining each image and the final process for creating a map of the lodging area and degree of lodging in the field.

1 Combine 21 First Imaging Device 22 Inertial Measurement Unit (IMU)
23 Second imaging device 24 Third imaging device 31, 32, 33 Obstacle sensor 41 Satellite positioning system receiver (GNSS receiver)
S3-12 Second map (yield)
S3-14 Third map (driving route)
S3-19 First Map (Growth)
S3-20 Fourth Map (Division)
201 Crop size detection unit 202 Crop color detection unit 211 Image state change and determination unit

Claims (1)

The weight and moisture of the crop are measured when the periphery of the field is harvested, the direction change position of the work vehicle is set to the corner of the field, and the estimated weight and moisture of the crop in the unharvested area and the unharvested area are added to the registered field map data ;
an imaging device mounted on a work vehicle photographs the field at predetermined intervals, converts the arrangement data of green and brown areas in the photographed images into a line image to determine the state of lodging of the crops, and adds the areas that will fall from the lodging state to the field map data;
adding a route from the lodging area to farm field map data;
A harvesting vehicle that uses the field map data as a data standard for automatic driving .