JP7525844B2 - Moisture content estimation method, cone index estimation method, cone index estimation system, vehicle and unmanned aerial vehicle - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies https://www.tc-iaip.org/dia/2020/php/reg_login/index.php Published on March 9, 2020 https://edas.info/index.php?c=25900 Published on October 15, 2019 GCCE2019 Senri Life Science Center (1-4-2 Shinsenri Higashimachi, Toyonaka City, Osaka Prefecture) Date held October 18, 2019 https://confit.atlas. jp/guide/event/ssiii2019/subject/IS1-14/detail Posted on June 12, 2019 SSII2019 Pacifico Yokohama Annex Hall (1-1-1 Minatomirai, Nishi-ku, Yokohama, Kanagawa Prefecture) Date held June 12, 2019

The present disclosure relates to a moisture content estimation method, a cone index estimation method, a cone index estimation system, a vehicle, and an unmanned aerial vehicle.

When vehicles or construction machinery are dispatched to an area where a disaster has occurred, it is necessary to obtain information about the ground in that area in advance. For example, Patent Document 1 describes an example of a soil quality determination device.

International Publication No. 2016/136231

However, the soil quality determination device described in Patent Document 1 requires the sensor to be in contact with the target soil.

The present disclosure has been made in consideration of the above problems, and aims to provide a moisture content estimation method, a cone index estimation method, a cone index estimation system, a vehicle, and an unmanned aerial vehicle that can estimate the moisture content of an object in a non-contact manner with high accuracy.

To achieve the above object, a water content estimation method according to one aspect of the present disclosure includes an imaging step of capturing an image of an object using a multispectral camera, a spectral reflectance acquisition step of acquiring a first spectral reflectance, which is the spectral reflectance of the object in an infrared wavelength band, and a second spectral reflectance, which is the spectral reflectance of the object in another wavelength band, based on the image, and a water content estimation step of estimating the water content of the object based on the difference between the first spectral reflectance and the second spectral reflectance, where the other wavelength band is a wavelength band of light having a shorter wavelength than the infrared wavelength band.

A preferred embodiment of the water content estimation method further includes a derivation step of deriving a vector of a spectral reflectance spectrum from an image of the object captured by a multispectral camera, and an object estimation step of estimating the type of the object based on the vector and a pre-stored spectral reflectance spectrum for each type of object, and in the water content estimation step, the water content of the object is estimated based on pre-stored data indicating the correspondence between the difference between the first spectral reflectance and the second spectral reflectance for each type of object, and the water content.

In a preferred embodiment of the water content estimation method, the object estimation step estimates the type of object using a trained neural network based on images of each type of object captured in advance by a multispectral camera.

As a preferred embodiment of the water content estimation method, among the vectors, when the average spectral reflectance in the wavelength band from 350 [nm] to 450 [nm] is α and the average spectral reflectance in the wavelength band from 700 [nm] to 750 [nm] is β, the vector for which α×10>β holds is not used in the object estimation step.

A preferred embodiment of the moisture content estimation method includes a determination step of determining whether the brightness of the image is appropriate based on a calibration sheet showing known spectral reflectance, and does not estimate the moisture content of the object based on an image whose brightness is inappropriate.

In order to achieve the above object, the cone index estimation method of one aspect of the present disclosure includes an imaging step of capturing an image of the ground including soil using a multispectral camera, a spectral reflectance acquisition step of acquiring a first spectral reflectance, which is the spectral reflectance of the ground in an infrared wavelength band, and a second spectral reflectance, which is the spectral reflectance of the ground in another wavelength band, based on the image, a derivation step of deriving a vector of a spectral reflectance spectrum from the image of the ground captured by the multispectral camera, an object estimation step of estimating the type of soil based on the vector and a pre-stored spectral reflectance spectrum for each type of soil, a moisture content estimation step of estimating the moisture content of the ground based on data indicating a correspondence between the difference between the first spectral reflectance and the second spectral reflectance for each type of soil and the moisture content, and a cone index estimation step of estimating the cone index of the ground based on the type of soil estimated in the object estimation step and the moisture content estimated in the moisture content estimation step, the other wavelength band being a wavelength band of light shorter than the infrared wavelength band.

In a preferred embodiment of the moisture content estimation method, the cone index estimation step estimates the cone index of the soil based on pre-stored data showing the correspondence between the moisture content and the cone index for each type of soil.

To achieve the above object, a cone index estimation system according to one embodiment of the present disclosure includes a multispectral camera that captures an image of the soil, and an information processing device that estimates the cone index of the soil based on the image. The information processing device acquires a first spectral reflectance, which is the spectral reflectance of the soil in an infrared wavelength band, and a second spectral reflectance, which is the spectral reflectance of the soil in another wavelength band, based on the image, estimates the water content of the soil based on the difference between the first spectral reflectance and the second spectral reflectance, and includes a calculation unit that uses information indicating the estimated water content of the soil as information relating to the estimation of the cone index of the soil, and the other wavelength band is a wavelength band of light with a shorter wavelength than the infrared wavelength band.

To achieve the above object, a vehicle according to one embodiment of the present disclosure is used in the water content estimation system described above and is equipped with the multispectral camera.

To achieve the above objective, an unmanned aerial vehicle according to one embodiment of the present disclosure is used in the water content estimation system described above and is equipped with the multispectral camera.

According to this disclosure, it is possible to estimate the water content of an object in a non-contact and highly accurate manner.

FIG. 1 is a block diagram showing an example of the main configuration of an estimation system. FIG. 2 is a diagram showing an example of a backhoe equipped with a first imaging device and a second imaging device. FIG. 3 is a schematic diagram showing a comparative example between an RGB image corresponding to visible light and a multispectral image. FIG. 4 is a functional block diagram of the estimation system. FIG. 5 shows examples of visible light images of ten types of soil. FIG. 6 is a graph showing the spectral reflectance spectra in the wavelength band from 500 nm to 900 nm for three of the ten types of soil shown in FIG. FIG. 7 is a graph showing an example of the relationship between the number of spectra of a specified captured image and the accuracy of identifying the type of soil. FIG. 8 is a schematic diagram showing an example of the relationship between a multispectral image and input/output of a neural network. FIG. 9 is a schematic diagram showing an example of a verification experiment environment related to estimation of soil type. FIG. 10 is a confusion matrix showing the accuracy of soil type estimation when the neural network is trained with a batch size of 128 and 12 epochs. FIG. 11 is an explanatory diagram of the water content of soil. FIG. 12 is a schematic graph showing an example of the relationship between the water content of soil and the spectral reflectance spectrum. FIG. 13 is a graph showing an example of the relationship between the first spectral reflectance, the second spectral reflectance, the difference between the first spectral reflectance and the second spectral reflectance, and the water content of soil. FIG. 14 is a graph with plots showing the difference between the first and second spectral reflectances shown in FIG. FIG. 15 is a graph showing a fitting curve generated based on the plot in FIG. FIG. 16 is a schematic diagram showing an example of a verification experiment environment related to estimation of soil moisture content. FIG. 17 is a schematic diagram showing an example of the configuration of the image pickup target in FIG. FIG. 18 is a graph showing examples of fitting curves for each of types A, B, D, F, and G. FIG. 19 is a diagram showing the relationship between a multispectral image including a first partial region to which the first pre-processing is applied and a second partial region to which the first pre-processing is not applied, and a graph showing an example of a comparison of the spectral reflectance spectra of these partial regions. FIG. 20 is a diagram showing examples of multispectral images that are adopted in the process related to estimating the water content of soil when the second preprocessing is performed, and examples of multispectral images that are not adopted. FIG. 21 is a diagram showing an example of a graph of the coefficient of variation of the brightness value of a partial image in which a calibration sheet is captured in a multispectral image, and the relationship between the suitability for use in processing by a moisture content estimation unit for estimating the moisture content of soil. FIG. 22 is a graph showing an example of the relationship between the water content of soil and the Cone Index. FIG. 23 is a diagram showing a specific example of the interpolation process. FIG. 24 is a flowchart showing an example of a process flow relating to estimation of a cone index performed by an information processing device in the embodiment. FIG. 25 is a flowchart showing an example of the flow of the soil-type estimation process shown in FIG. 24 . FIG. 26 is a flowchart showing an example of the flow of the soil moisture content estimation process shown in FIG. 24 . FIG. 27 is a flowchart showing an example of the flow of the cone index estimation process shown in FIG. 24 . FIG. 28 is a schematic diagram showing the relationship between the backhoe and the imaging range in the verification experiment. FIG. 29 is a graph showing the relationship between the moisture content and the cone index of the soil excavated and backfilled in the verification experiment described with reference to FIG. 28. FIG. 30 is a table showing the relationship between the measured values and the estimated values of the moisture content and the cone index of the soil for three patterns. FIG. 31 is a graph showing the relationship between the actual measured values and the estimated values of the cone index for three patterns. FIG. 32 is a table showing an example of the results of judging the travelling performance of a backhoe based on the cone index estimated in the three patterns shown in FIG. FIG. 33 is a diagram showing a configuration example of a backhoe to which a modified example of the embodiment is applied.

The following describes in detail an embodiment of the estimation system according to the present disclosure with reference to the drawings. Note that the present disclosure is not limited to the description of the following embodiment. Furthermore, the components in the following embodiment include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same. Furthermore, the components in the embodiments described below may be omitted, replaced, or modified in various ways without departing from the gist of the present disclosure. In the following embodiment, the components necessary to illustrate the embodiment of the estimation system according to the present disclosure are described, and other components are omitted.

FIG. 1 is a block diagram showing an example of the main configuration of an estimation system 1. The estimation system 1 includes a multispectral camera 10 and an information processing device 20. The multispectral camera 10 includes, for example, a first imaging device 11 and a second imaging device 12.

Figure 2 is a diagram showing an example of a backhoe 100 equipped with a first imaging device 11 and a second imaging device 12. The backhoe 100 is an example of a vehicle used in the estimation system 1. The first imaging device 11 and the second imaging device 12 are installed, for example, on the upper side of the roof portion 101 of the backhoe 100. The first imaging device 11 and the second imaging device 12 are installed, for example, so that the ground surface GR on the front side from which the arm 102 extends in the backhoe 100 is included within the imaging range SUB. In FIG. 2, the first imaging device 11 and the second imaging device 12 are installed so as to be supported by a support member 13 erected on the roof portion 101, and the second imaging device 12 is positioned above the first imaging device 11, but this is merely an example of an installation example and is not limited to this, and can be changed as appropriate.

The backhoe 100 in FIG. 2 is merely a specific example of a configuration that can mount the multispectral camera 10 and is mobile, and is not limited to this. The multispectral camera 10 may be provided in other configurations, such as an aircraft, which will be described later.

Unlike general imaging devices that are limited to the function of detecting light in a spectrum corresponding to human visible light and generating a single captured image including the entire wavelength band, the multispectral camera 10 is an imaging device that has the function of detecting light in a specific spectrum that is specified in advance and generating a captured image. Here, multiple types of specific spectra can be specified. Furthermore, the multispectral camera 10 of the embodiment can detect light in a spectrum that includes light invisible to humans and generate a captured image.

Hereinafter, when we say "spectrum of captured image", it refers to the spectrum of light that the multispectral camera 10 is specified to detect when generating the captured image. Furthermore, when we say "captured image of α [nm]", it refers to a captured image generated when the multispectral camera 10 is specified to detect α [nm] as the "spectrum of captured image". α is a number that represents the wavelength of light or the wavelength band of light. Furthermore, when we say "multispectral image", it encompasses all captured images captured by the multispectral camera 10.

The first imaging device 11 and the second imaging device 12 have different detectable light spectra. In the embodiment, the first imaging device 11 is a multispectral camera 10 that can specify the spectrum of the captured image in 5 nm increments within the range from 350 nm to 1100 mm. The second imaging device 12 is a multispectral camera 10 that can specify a total of seven types of spectrum, including six wavelengths of 490 nm, 570 nm, 671 nm, 800 nm, 900 nm, and 950 nm, and a wavelength band from 900 nm to 1700 mm. The spectrum of light that can be detected by the first imaging device 11 and the second imaging device 12 illustrated in the embodiment is merely an example and is not limited thereto, and does not indicate that these spectra are essential to achieve the same effect as the present embodiment.

Figure 3 is a schematic diagram showing a comparison between an RGB image corresponding to visible light and a multispectral image. The x-axis and y-axis directions in Figure 3 indicate the imaging plane, and the z-axis direction indicates the spectrum of light detected during imaging. The upper side in Figure 3 represents shorter wavelengths, and the lower side represents longer wavelengths.

An RGB image is an image in which records showing the light intensity of three wavelength bands, red (R), green (G), and blue (B), are combined into one image. In contrast, a multispectral image can specify many types of spectrum as the spectrum of the captured image, not limited to the three wavelength bands of red (R), green (G), and blue (B). In FIG. 3, each imaging surface stacked in the z-axis direction in the image shown as a "multispectral image" shows an image (spectral image) corresponding to each spectrum of light specified as the spectrum of the captured image. A multispectral image includes multiple such spectral images. Each imaging surface can be considered as image data in which multiple pixels are arranged in two-dimensional directions along the x-axis and y-axis directions. More specifically, the image data is two-dimensional image data in which, for example, 640 pixels are arranged in the x-axis direction and 480 pixels are arranged in the y-axis direction, but is not limited to this, and the specific number of pixels can be changed as appropriate. The gradation value of each pixel of the image data is a value corresponding to the spectral reflectance of light detected at the position of the pixel by the imaging element of the multispectral camera 10 that captured the imaging range SUB. That is, each imaging surface is a spectral image of the same imaging range SUB, and can be treated as image data in which the spectral reflectance in each different spectrum is recorded as a gradation value. Furthermore, since each imaging surface is specialized for the spectral reflectance of the corresponding light spectrum, each pixel of each imaging surface represents the light intensity of the corresponding light spectrum as a gradation value. Therefore, each imaging surface becomes like so-called grayscale image data in which the imaging range SUB is imaged with the corresponding light spectrum. Furthermore, a multispectral image functions as an image that can obtain a greater amount of information in the wavelength range of light than an RGB image. Furthermore, the wavelength range of light that can be used in a multispectral image includes both the wavelength ranges of visible light and invisible light.

Such a multispectral image can be used to estimate the properties of the ground GR captured by the multispectral camera 10. The properties of the ground GR that can be estimated based on the multispectral image include the type of soil covering the ground GR and the moisture content of the soil. In the embodiment, the cone index is estimated based on the type of soil and the moisture content of the soil estimated based on the multispectral image.

The cone index is one of the soil parameters and is the penetration resistance when the cone of a cone penetrometer is pushed into the soil. The soil parameter refers to an index that indicates the degree of the soil's properties. A specific example of the configuration of a cone penetrometer is one in which a cone-shaped cone is attached to one end of a shaft with a length of 91.4 cm and a diameter of 0.95 cm, and a gauge and handle for measuring the resistance of the soil are attached to the other end. The cone has, for example, a tip angle of 30° and a base area of 1.61 cm. The base side of the cone is attached to one end of the shaft. As an example of how to use such a cone penetrometer, a method is known in which a user holds the shaft vertically with one end facing the ground, places the tip of the cone on the ground, and applies force to push the cone into the soil, and then reads the resistance acting on the cone from the soil with a gauge on the top. The cone index is calculated by dividing the resistance value by the base area of the cone and multiplying it by a certain coefficient. Note that the cone penetrometer specifications given here are merely those of a cone penetrometer developed by the U.S. Army Corps of Engineers in 1948, and do not indicate that the calculation of the cone index is limited to those using cone penetrometers with these specifications.

The cone index can be used as an indicator of the drivability of a mobile construction machine. Generally, construction machines equipped with caterpillar tracks 103 as shown in FIG. 2 are manufactured to be able to run on ground surfaces with a cone index of 200 or more. Therefore, when determining the travel route of such construction machines, there is a demand to measure in advance the cone index of ground surfaces that are candidates for the travel route.

On the other hand, when using a cone penetrometer to find the cone index of ground, the user of the cone penetrometer must go to the ground in question. For this reason, in areas where it is dangerous to enter due to a disaster, for example, it may be difficult for the user to bring the cone penetrometer into the area, making it difficult to calculate the cone index. In addition, the cone index of ground found using a cone penetrometer is a point-wise cone index at the position where the cone penetrates. For this reason, if one tries to find the cone index of a wider area of ground, it is necessary to penetrate the cone in multiple places.

In the embodiment, as described above, the cone index is estimated based on the type of soil estimated based on the multispectral image and the water content of the soil. This makes it possible to estimate the cone index even in areas where it is difficult to bring in a cone penetrometer. In addition, by fitting the ground surface GR within the imaging range SUB using the multispectral image, the cone index can be estimated for the entire ground surface GR included in the imaging range SUB. The multispectral camera 10 of the embodiment captures a multispectral image with the ground surface GR as the imaging target. The information processing device 20 performs various processes related to the estimation of the cone index based on the multispectral image captured by the multispectral camera 10.

Specifically, a multispectral image reflects the spectral reflectance spectrum of the captured soil. A spectral reflectance spectrum is the ratio of the energy of light reflected by a material to the energy of light incident on that material, arranged by wavelength.

The spectral reflectance spectrum can be expressed as the following formula (1). In formula (1), ρ(λ) represents the spectral reflectance of a material. Furthermore, L _s (λ) represents the intensity of light reflected by the material. Furthermore, L _i (λ) represents the intensity of light incident on the material. Furthermore, λ represents the wavelength.

The degree of reflection, scattering, absorption, and emission of each wavelength of light varies depending on the molecular and atomic structure of the material, or the size and shape of the particles that make up the material, and the unevenness of its surface. The different degrees of reflection, scattering, absorption, and emission of each wavelength of light means that the spectral reflectance spectrum is different.

Soil parameters that affect the spectral reflectance spectrum include the mineral composition of soil particles, the organic matter content of soil, the particle size distribution of soil, the sphericity of the particles that make up the soil, and the water content of soil. Among these, the combination of the mineral composition of soil particles, the organic matter content of soil, the particle size distribution of soil, and the sphericity of the particles that make up the soil is reflected in the material, diameter distribution, shape, surface roughness, etc. of the particles of each type of soil, and functions as a soil parameter that is specific to each type of soil. The effect on the spectral reflectance spectrum differs depending on the combination according to the type of soil covering the ground GR captured by the multispectral camera 10. More specifically, each imaging surface stacked in the z-axis direction in the image illustrated as a "multispectral image" in FIG. 3 reflects the spectral reflectance of a different light spectrum. Here, the "multispectral image" in which each imaging surface is arranged in spectral order along the z-axis direction reflects the spectral reflectance spectrum of the soil covering the ground GR captured by the multispectral camera 10. Therefore, based on the multispectral image, it is possible to estimate the type of soil covering the ground surface GR captured by the multispectral camera 10.

Among the soil parameters, the moisture content of the soil functions as a soil parameter independent of the type of soil. As will be described in detail later, even if the soil is of the same type, if the moisture content of the soil is different, the spectral reflectance spectrum will be different. In this way, the moisture content of the soil affects the spectral reflectance spectrum. That is, the moisture content of the soil covering the ground GR captured by the multispectral camera 10 affects the spectral reflectance spectrum obtained from the multispectral image differently. Therefore, the moisture content of the soil covering the ground GR captured by the multispectral camera 10 can be estimated based on the multispectral image. This indicates that the type of soil covering the ground GR and the moisture content of the soil can be estimated without contacting the ground GR. The type of soil and the moisture content of the soil have a large effect on the cone index of the ground GR composed of the soil. Therefore, the cone index can be estimated by estimating the type of soil covering the ground GR and the moisture content of the soil.

The wavelength band of light that is the spectrum of the captured image includes wavelengths that are not affected or are relatively less affected by the moisture content of the soil, and wavelengths that are relatively more affected by the moisture content. When capturing a multispectral image used to estimate the type of soil, the wavelengths that are not affected or are relatively less affected by the moisture content of the soil are considered to be the spectrum of the captured image. In addition, both the wavelengths that are not affected or are relatively less affected by the moisture content of the soil and the wavelengths that are relatively more affected by the moisture content are considered to be the spectrum of the captured image.

In general, the deeper the soil is underground, the harder it tends to be due to pressure from the mass of the upper soil located closer to the ground. In other words, the exposed ground is often the softest part in the soil layer structure. In addition, in the same place, the soil exposed on the ground and the soil located deeper underground are often the same type of soil. In addition, the softer the soil, the lower the cone index tends to be. Due to these circumstances, the cone index estimated from the soil covering the ground rarely exceeds the cone index of the ground covered by the soil. Therefore, if the cone index of the soil covering the ground GR imaged by the multispectral camera 10 can be estimated, sufficient information can be obtained to estimate the traversability of the ground covered by the soil. This is because if the cone index of the softest soil exposed on the ground GR can withstand the weight of a construction machine, the cone index of the hard soil located deeper underground will naturally also be able to withstand the weight of a construction machine. In other words, if the construction machine's traversal capability is determined based on the cone index of the softest soil exposed on the ground surface GR, it is considered sufficient to determine the traversal capability of the ground surface GR.

The information processing device 20 performs processing related to estimating the type of soil, the moisture content of the soil, and the cone index based on a multispectral image. As shown in FIG. 1, the information processing device 20 includes a calculation unit 21, a storage unit 22, an interface 23, an input unit 24, and an output unit 25.

The calculation unit 21 includes a calculation device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The calculation unit 21 reads out and executes software programs and data stored in the storage unit 22. Hereinafter, when simply referred to as a program, this refers to the software program read out by the calculation unit 21 and the data referenced when executing the software program.

The storage unit 22 includes a secondary storage device that stores programs. Specifically, as shown in FIG. 1, the storage unit 22 stores a soil type estimation program 221, a moisture content estimation program 222, and a cone index estimation program 223. Specific configuration examples of such secondary storage devices include hard disk drives, solid state drives, flash memories, and other non-volatile storage devices. The storage unit 22 of the embodiment includes one or more of these. The storage unit 22 also includes a primary storage device that functions as a RAM (Random access memory) used as a storage area when the program is executed by the calculation unit 21. Specific configuration examples of such temporary storage devices include a semiconductor memory that functions as a DRAM (Dynamic Random Access Memory).

The interface 23 connects the information processing device 20 and the multispectral camera 10. Specifically, the interface 23 is an interface corresponding to the output form of the multispectral camera 10. For example, when the input/output interface of at least one of the first imaging device 11 and the second imaging device 12 is USB (Universal Serial Bus), the interface 23 includes a USB interface to which a terminal of a USB cable can be connected. In addition, when at least one of the first imaging device 11 and the second imaging device 12 outputs data via communication, the interface 23 includes a network interface corresponding to the standard of the communication. The communication path may be wired, may be wireless, or may be a mixture of wired and wireless. In addition, a part or all of the communication path may use a public communication line such as the Internet, or a part or all of the communication path may be provided exclusively for the embodiment. The specific form of the interface exemplified above does not limit the connection form between the multispectral camera 10 and the information processing device 20, and may be any form that allows the images captured and generated by the multispectral camera 10 to be transmitted to the information processing device 20.

The information processing device 20 may be mounted on the backhoe 100, or may be provided outside the backhoe 100 and connected to the multispectral camera 10 provided on the backhoe 100 via wire or wirelessly.

The input unit 24 includes an input device for receiving input to the information processing device 20, such as a touch panel, a keyboard, a mouse, and multiple switches. The output unit 25 includes a configuration for outputting corresponding to the processing results of the information processing device 20, such as a display and a printer.

Figure 4 is a functional block diagram of the estimation system 1. As shown in Figure 4, the information processing device 20 functions as an acquisition unit 31, a soil type estimation unit 32, a moisture content estimation unit 33, a cone index estimation unit 34, and an output unit 35. The information processing device 20 of the embodiment is an information processing device that has each of the components described with reference to Figure 1 and performs various functions described with reference to Figure 4, but is not limited to this. The information processing device 20 may be a dedicated device that performs various functions described with reference to Figure 4.

The acquisition unit 31 acquires a multispectral image. Specifically, the acquisition unit 31 is realized by acquiring the multispectral image that is output from the multispectral camera 10 and input to the information processing device 20 via the interface 23.

The soil type estimation unit 32 estimates the type of soil covering the ground GR captured by the multispectral camera 10. Specifically, the soil type estimation unit 32 is realized by the calculation unit 21 executing a soil type estimation program 221. The water content estimation unit 33 estimates the water content of the soil covering the ground GR captured by the multispectral camera 10. Specifically, the water content estimation unit 33 is realized by the calculation unit 21 executing a water content estimation program 222. The cone index estimation unit 34 estimates the cone index of the ground GR captured by the multispectral camera 10. Specifically, the cone index estimation unit 34 is realized by the calculation unit 21 executing a cone index estimation program 223. The functions of the soil type estimation unit 32, the water content estimation unit 33, and the cone index estimation unit 34 will be described in detail later.

The output unit 35 performs an output indicating the cone index estimated by the cone index estimation unit 34. Specifically, the output unit 35 performs an output indicating the cone index by at least one of the following: a display output by a display provided in the output unit 25, an image formation output by a printer, a sound by a speaker, and a voice.

The following describes in detail the estimation of soil type by the soil type estimation unit 32, the estimation of soil moisture content by the moisture content estimation unit 33, and the estimation of the cone index by the cone index estimation unit 34.

(Estimation of soil type)
In estimating the type of soil based on a multispectral image, it is utilized that each type of soil has a different spectral reflectance spectrum. When estimating the type of soil, a detailed shape of the spectral reflectance spectrum at a wavelength that is not absorbed by water molecules is required. Therefore, by increasing the number of spectra of the captured image specified for estimating the type of soil (first designated number), the estimation accuracy of the type of soil can be improved. In the embodiment, the first designated number is larger than the number of spectra of the captured image specified for estimating the water content (second designated number). Note that a multispectral image in which the spectra of the captured image are very large, such as the first designated number in the embodiment, is sometimes referred to as a hyperspectral image, especially among multispectral images.

Figure 5 is a diagram showing example visible light images of each of the ten types of soil. Figure 6 is a graph showing the spectral reflectance spectra in the wavelength band from 500 nm to 900 nm for three of the ten types of soil shown in Figure 5. In the graph of Figure 6, the vertical axis represents the spectral reflectance, and the horizontal axis represents the wavelength. Line L1 in Figure 6 shows the spectral reflectance spectrum of type A. Line L2 in Figure 6 shows the spectral reflectance spectrum of type B. Line L3 in Figure 6 shows the spectral reflectance spectrum of type C.

Figure 5 shows ten types of soil collected at different locations in Japan. Five of the ten types of soil are coarse-grained soils in which more than half of the soil particles have a diameter of 0.075 mm or more and less than 75 mm. The other five types are fine-grained soils in which more than half of the soil particles have a diameter of less than 0.075 mm. The other five types of fine-grained soils are further divided into clay soils and volcanic ash clay soils based on the origin of the soil. Specifically, types A, D, and G shown in Figure 5 are clay soils. Types B and F shown in Figure 5 are volcanic ash clay soils. Types C, E, H, I, and J shown in Figure 5 are coarse-grained soils.

As shown in FIG. 6, the spectral reflectance spectrum of line L3 (type C) is different from that of line L1 (type A) and line L2 (type B) over the entire wavelength range from 500 nm to 900 nm. Therefore, if the spectrum of the captured image is specified within the wavelength range from 500 nm to 900 nm, type C can be distinguished from types A and B based on the multispectral image.

In contrast, when comparing the spectral reflectance spectrum of type A with that of type B, there is almost no difference in the wavelength band from 600 nm to 700 nm. On the other hand, at a wavelength of 500 nm, the spectral reflectance of type A is higher than that of type B. Also, in wavelength bands exceeding 700 nm, the spectral reflectance of type A is higher than that of type B. In this way, if a wavelength of 500 nm or a wavelength band exceeding 700 nm is specified as the spectrum of the captured image, type A and type B can be distinguished based on the multispectral image.

Even if the soils are classified as the same type in terms of whether they are coarse-grained soil, clay soil, or volcanic ash clay soil, each of the 10 types of soils illustrated in FIG. 5 exhibits a different spectral reflectance spectrum within the spectral range of the captured image that can be specified by the first imaging device 11.

As described with reference to FIG. 6, if the spectrum of the specified captured image were limited to the wavelength band from 600 nm to 700 nm, it would be more difficult to distinguish between types A and B. Considering the relationship between the type of soil and the spectral reflectance spectrum, it is desirable for the number of spectra of the specified captured image and the wavelength bands included in the spectrum of the captured image to be large.

Figure 7 is a graph showing an example of the relationship between the number of spectra of the specified captured image and the accuracy of soil type discrimination. Hereinafter, unless otherwise specified, "designated number" refers to the number of wavelengths of light or types of wavelength bodies of light specified as the spectrum of the captured image. Figure 7 illustrates the high accuracy of soil type discrimination when the designated number is 3, when the designated number is 6, and when the designated number is 151. As shown in Figure 7, when the designated number is 151, the soil type can be discriminated with higher accuracy than when the designated number is 3 and when the designated number is 6. Also, when the designated number is 6, the soil type can be discriminated with higher accuracy than when the designated number is 3. Thus, the larger the designated number, the easier it is to increase the accuracy of soil type discrimination. Therefore, a larger designated number is preferable, and is not limited.

As described above, the first imaging device 11 of the embodiment is a multispectral camera 10 that is configured to be able to specify the spectrum of the captured image in increments of 5 nm within the range from 350 nm to 1100 nm. That is, the first imaging device 11 can acquire captured images for up to 151 types of light wavelengths by specifying the spectrum of the captured image in increments of 5 nm, such as 350 nm, 355 nm, 360 nm, ..., 1100 nm. In the embodiment, the accuracy of identifying the type of soil is ensured by acquiring captured images for each spectrum in such a wide range.

As described above, the first imaging device 11 is specified with 151 types of light wavelengths as the spectrum of the captured image. The first imaging device 11 captures an image of the ground GR (see FIG. 2) under the specified conditions. The calculation unit 21 acquires a multispectral image captured by the first imaging device 11. The soil type estimation unit 32 estimates the type of soil covering the ground GR captured by the first imaging device 11 based on the multispectral image. Specifically, the soil type estimation unit 32 estimates the type of soil by using a neural network NN.

Figure 8 is a schematic diagram showing an example of the relationship between the multispectral image and the input and output of the neural network NN. The soil type estimation unit 32 extracts the multispectral image in units of unit regions Cr, and inputs the components included in the vector of the spectral reflectance spectrum indicated by the unit regions Cr to the neural network NN individually. Here, a row vector in which the spectral reflectance indicated by the gradation values of each of the multiple imaging surfaces arranged in the z-axis direction in the unit region Cr is arranged from one end side (long wavelength side) to the other end side (short wavelength side) in the z-axis direction functions as a vector of the spectral reflectance spectrum indicated by the unit region Cr. That is, more specifically, the soil type estimation unit 32 derives a vector whose elements are the gradation values of each of the multiple imaging surfaces in each unit region Cr as a vector of the spectral reflectance spectrum indicated by each unit region Cr, and inputs the derived vector to the neural network NN.

In the embodiment, the vector of the spectral reflectance spectrum is a vector having components equal to the number of spectra specified as the spectrum of the captured image in the multispectral image captured by the first imaging device 11. Each component is represented by the gradation value at the position of the unit region Cr in each imaging plane of the multispectral image. The components are converted to have values between 0 and 1. Specifically, when the gradation value is treated as h-bit data in each imaging plane, the minimum value of the h-bit data is treated as 0, and the maximum value is treated as 1. As an example, when h=16, when the gradation value is 0, the component is 0, and when the gradation value is 65535, the component is 1.

The unit region Cr is a pixel region including one or more predetermined pixels. In the embodiment, the unit region Cr is one pixel. The unit region Cr may be a region of one pixel in the x-axis direction and a pixel column region extending from one end of the imaging surface in the y-axis direction. The unit region Cr may be a region of one pixel in the y-axis direction and a pixel row region extending from one end of the imaging surface in the x-axis direction. The unit region Cr may be each of a plurality of regions obtained by dividing the imaging surface in the x-axis and y-axis directions.

The first imaging device 11 of the embodiment is a multispectral camera 10 that captures images by scanning the imaging range SUB. Specifically, the first imaging device 11 of the embodiment records the spectral reflectance spectrum obtained by dispersing incident light for each unit area Cr constituting the multispectral image, and when that is complete, performs a process of dispersing incident light for the next unit area Cr and starting to record the spectral reflectance spectrum, sequentially from the first unit area Cr to the last unit area Cr. Therefore, the first imaging device 11 of the embodiment records the spectral reflectance spectrum of the material imaged in each unit area Cr during imaging for each unit area Cr. Therefore, in order to obtain the spectral reflectance spectrum from the multispectral image captured by the first imaging device 11 in the process performed by the soil type estimation unit 32, it is obtained for each unit area Cr.

The neural network NN includes an input layer, an output layer, and an intermediate layer. The input layer, output layer, and intermediate layer each have a plurality of nodes. In the embodiment, the neural network NN has an input layer that is a vector indicating the spectral reflectance spectrum derived by the soil type estimation unit 32. That is, the number of nodes in the input layer of the neural network NN corresponds to the number of "spectra of captured images" specified in capturing a multispectral image by the first imaging device 11. Each of the rectangular columns that make up the unit area Cr shown in the "Neural Network" column in Figure 8 is each of the multiple components included in the vector indicating the spectral reflectance spectrum.

The neural network NN is a neural network for estimating which of the i predetermined types of soil the soil of the ground GR captured as a multispectral image corresponds to. Therefore, the number of nodes (j) in the output layer (j class) of the neural network NN corresponds to the number of types of soil that can be distinguished in estimation using the neural network.

The number of nodes in the intermediate layer of the neural network NN is preferably smaller than the number of nodes in the input layer of the neural network NN. More preferably, the number of nodes in the intermediate layer is less than half the number of nodes in the input layer. In the embodiment, the number of nodes (i) in the intermediate layer of the neural network NN is approximately half the number of nodes in the input layer, but this is an example and is not limited to this, and can be changed as appropriate. More specifically, in the embodiment, the number of nodes (unit area Cr) in the input layer of the neural network NN is 151, the number of nodes (i) in the intermediate layer is 65, and the number of nodes (j) in the output layer is 10 (see FIG. 5).

The intermediate layer nodes are located between the input layer nodes and the output layer nodes. Each intermediate layer node receives input from all input layer nodes. That is, a vector of the spectral reflectance spectrum in one unit region Cr is input to each intermediate layer node. In addition, all output layer nodes are connected to each intermediate layer node. Each intermediate layer node outputs a value indicating the weighting result to each output layer node.

To give a more specific example, the intermediate layer nodes include a node that outputs a high weighting value to the node of the output layer corresponding to the estimation result that the soil is type A when the vector of the spectral reflectance spectrum is highly likely to be that of soil type A. The intermediate layer nodes also include a node that outputs a high weighting value to the node of the output layer corresponding to the estimation result that the soil is type B when the vector of the spectral reflectance spectrum is highly likely to be that of soil type B. Such nodes are provided for each type of soil. In addition, nodes that are not limited to simple types of soil A, B, ..., and that increase the accuracy of the estimation result of the soil type by weighting based on soil parameters such as the material and diameter distribution, shape, and surface roughness of the particles of each type of soil may be further provided. In addition, nodes that increase the accuracy of the estimation result of the soil type by a rougher distinction such as between coarse soil, fine soil, clayey soil among fine soil, and volcanic ash clayey soil among fine soil may be further provided. These are examples of intermediate layer nodes and are not limited to these. In an embodiment, nodes corresponding to weighting processes that can be used to further increase the accuracy of the soil type estimation results can be set as intermediate layer nodes.

In this way, a value indicating the result of the weighting process performed by each of the intermediate layer nodes is output for each of the nodes of the output layer corresponding to the distinguishable types of soil. For one imaging by the first imaging device 11, ideally, the input to the input layer is repeated as many times as the number of unit areas Cr included in the multispectral image, the intermediate layer performs weighting process for each input, and the output layer is given a value indicating the likelihood of the estimation result of the soil type. Here, for example, if the value of a node indicating that the estimation result is type A among the nodes of the output layer is higher than the values of the nodes of the other output layers, the soil type estimation unit 32 estimates that the type of soil covering the ground GR imaged by the first imaging device 11 is type A. Also, if the value of a node indicating that the estimation result is type B among the nodes of the output layer is higher than the values of the nodes of the other output layers, the soil type estimation unit 32 estimates that the type of soil covering the ground GR imaged by the first imaging device 11 is type B. In this way, the soil type estimation unit 32 of the embodiment estimates the type of soil using a neural network NN.

The reason why "ideally" is stated is that in reality, the vector of the spectral reflectance spectrum acquired from a part or all of the unit region Cr may be a vector that is not suitable for estimating the soil type. In the embodiment, the soil type estimation unit 32 performs a preprocessing (first preprocessing) in which the vector that is not suitable for estimating the soil type is excluded from the input to the neural network NN, and the vector that is suitable for estimating the soil type is input to the neural network NN. The details of the first preprocessing will be described later. In the embodiment, when a predetermined number or more of vectors suitable for estimating the soil type are obtained from the multispectral image obtained in one imaging, the soil type estimation unit 32 estimates the soil type based on the multispectral image. The predetermined number is, for example, 100, but is not limited to this and can be changed as appropriate.

The soil type estimation program 221 is a program or the like that causes the calculation unit 21 to execute the soil type estimation program 221 to function as a neural network NN. The soil type estimation program 221 also includes data indicating the spectral reflectance spectrum unique to each soil type, as described with reference to FIG. 6. This data is referenced in the weighting process performed in the intermediate layer nodes of the neural network NN. In addition, the soil type estimation program 221 further includes data that each intermediate layer node references for the weighting process.

To explain a more specific form, in the embodiment, ReLU (Rectified Linear Unit) is used as the activation function of the intermediate layer nodes in the neural network NN. A softmax function is used as the activation function of the output layer nodes in the neural network NN. Dropout is performed between the intermediate layer and the output layer in the neural network NN, and the dropout rate is set to 0.2. RMSprop is used to optimize the neural network NN, and the learning rate is set to 0.001. The soil type estimation program 221 in the embodiment is, for example, a program for functioning as a neural network NN that realizes these activation functions, dropout rates, optimization methods, and learning rates. These activation functions, dropout rates, optimization methods, and learning rates are merely examples and are not limited to these, and can be changed as appropriate.

Figure 9 is a schematic diagram showing an example of a verification experiment environment for estimating soil types. Figure 9 shows an example of a configuration in which three halogen lamps HL are arranged around a pot-shaped container CA containing soil, illuminating the soil exposed from the upper opening of the container CA. When capturing a multispectral image using the first imaging device 11 installed on a construction machine as illustrated in Figure 2, sunlight is used as a light source outdoors, but in the verification experiment, the multispectral image was captured using a halogen lamp HL, which has a spectrum similar to sunlight, as a light source. This makes it possible to obtain a spectral reflectance spectrum similar to that obtained when sunlight is used as the light source from the multispectral image. Furthermore, this verification experiment environment makes it possible to suppress the occurrence of adverse conditions such as insufficient light due to changes in environmental conditions, such as rainy weather or cloudy skies, and to capture a multispectral image in an environment where sufficient light is available.

By using the verification experiment environment as illustrated in FIG. 9, the spectral reflectance spectrum for each type of soil can be obtained. That is, the first image capture device 11 captures images of the soil in the container CA under different conditions, and the spectral reflectance spectrum for each type of soil can be obtained. The data indicating the spectral reflectance spectrum for each type of soil obtained in this manner can be used as data to be referenced in the weighting process performed by the intermediate layer node in the neural network NN. In the embodiment, the multispectral image for each type of soil is captured by the first image capture device 11 in the verification experiment environment, and the neural network NN is trained with a batch size of 128 and epoch number of 12. The batch size is the size of the divided data when the entire data (data set) is divided into multiple divided data (subsets) and the neural network NN processes the data. The number of epochs indicates the number of times the neural network NN repeats the process using the same data set. In the embodiment, the soil imaged in the verification experiment environment of FIG. 9 is standardized to have a water content of 0%, but this is not a required condition and is merely a measure taken to prevent images from being captured under different conditions for each type of soil.

10 is a confusion matrix showing the accuracy of soil type estimation when the neural network NN is trained with a batch size of 128 and the number of epochs of 12. In the confusion matrix, the vertical axis shows the actual soil type indicated by the spectral reflectance spectrum of each pixel of the multispectral image captured by the first imaging device 11, and the horizontal axis shows the result of the soil type estimation unit 32 estimating the soil type using the neural network NN based on the spectral reflectance spectrum. The letters (A, B, ..., J) on the vertical and horizontal axes correspond to the letters attached to the symbols of the soil types described with reference to FIG. 5. When the actual soil type detected by the first imaging device 11 matches the soil type estimated by the soil type estimation unit 32, the probability of the diagonal components of the confusion matrix increases. The higher the probability in this confusion matrix, the darker the color. In FIG. 10, the correspondence between the probability and the color intensity in the confusion matrix is shown by the color bar on the right side of the confusion matrix.

As shown in FIG. 10, the actual soil type can be estimated and identified with a high probability for all 10 types of soil exemplified with reference to FIG. 5 based on the spectral reflectance spectrum obtained from the multispectral image captured by the first imaging device 11. The soil type estimation unit 32 can estimate the soil type with the same estimation accuracy even if the light source is changed from a halogen lamp HL to sunlight.

(Estimation of water content)
FIG. 11 is an explanatory diagram of the moisture content of soil. The moisture content indicates the amount of water contained in a solid. The moisture content of soil is a soil parameter that indicates the percentage [%] of moisture contained in soil by dividing the weight of water by the weight of the solid component of soil (soil particles). If the moisture content of soil is w [%], w can be calculated by the following formula (2). mw in FIG. 11 and formula (2) indicates the mass of water contained in soil. Also, _ms in FIG. 11 and formula (2) indicates the mass of the solid component of soil (soil particles). Here, as shown in FIG. 11, _mw + _ms is _the mass of soil.

w=m _w /m _s ×100…(2)

In addition to solid components and moisture, soil also contains air (m _a ) between the soil particles. This air (m _a ) is not included in the mass of the soil and is not involved in the calculation of the moisture content.

The moisture content of soil is a soil parameter that significantly affects the cone index of the soil. The moisture content estimation unit 33 estimates the moisture content of the soil.

Figure 12 is a schematic graph showing an example of the relationship between the moisture content of soil and the spectral reflectance spectrum. In Figure 12, the vertical axis shows the spectral reflectance of light by the soil, and the horizontal axis shows the wavelength of the light. Lines L4, L5, and L6 shown in Figure 12 are spectral reflectance spectra of the same type of soil. Line L4 shown in Figure 12 shows the spectral reflectance spectrum when the moisture content of the soil is relatively low. Line L6 shown in Figure 12 shows the spectral reflectance spectrum when the moisture content of the soil is relatively high. Line L5 shown in Figure 12 shows the spectral reflectance spectrum when the moisture content of the soil is between the moisture content shown by line L4 and the moisture content shown by line L6.

The moisture content of soil affects the spectral reflectance spectrum of the soil. Therefore, even if the soil is the same type, if the moisture content of the soil is different, the spectral reflectance spectrum will be different. Specifically, water molecules absorb light through the vibrational contraction and bending motion between the oxygen and hydrogen atoms that make up the water molecule. Here, water molecules absorb light in some spectrums more significantly than in other spectrums. Specifically, it is known that water molecules absorb infrared light more significantly.

As the moisture content of soil increases, the rate at which water seeps into the gaps between soil particles increases. As the rate at which water seeps into the gaps between soil particles increases, the difference in refractive index between the soil and the air above the soil decreases. This decrease in the difference in refractive index means that the rate at which light incident on the soil from the air is not reflected but is refracted and enters the soil increases. Therefore, generally, the higher the moisture content of soil, the lower the spectral reflectance spectrum of light reflected by the soil tends to be overall. In addition to this tendency, as mentioned above, water molecules absorb infrared light more significantly. Therefore, the higher the moisture content of soil, the lower the spectral reflectance spectrum of light reflected by soil tends to be in the infrared wavelength band compared to the spectral reflectance of other wavelength bands.

12, a spectrum w1 corresponding to visible light and a spectrum w2 corresponding to infrared light are picked up, and the difference between the spectral reflectance in spectrum w1 and the spectral reflectance in spectrum w2 is shown. The difference _d1 shows the difference in the spectral reflectance spectrum shown by line L4. The difference _d2 shows the difference in the spectral reflectance spectrum shown by line L5. The difference _d3 shows the difference in the spectral reflectance spectrum shown by line L6.

In the spectral reflectance spectrum indicated by line L4, the spectral reflectance SP4 in spectrum w2 is relatively higher than the spectral reflectance SP1 in spectrum w1. Therefore, the difference _d1 is a positive value. In contrast, in the spectral reflectance spectrum indicated by line L5, the spectral reflectance SP5 in spectrum w2 is relatively lower than the spectral reflectance SP2 in spectrum w1. This is because the water content of the soil in the case of line L5 is relatively higher than that in the case of line L4, and the light in spectrum w2 is significantly absorbed by the water molecules contained in the soil. Therefore, the difference _d2 is a negative value. Therefore, the value of the difference _d2 is smaller than the value of the difference _d1 . In addition, in the spectral reflectance spectrum indicated by line L6, the spectral reflectance SP6 in spectrum w2 is relatively lower than the spectral reflectance SP3 in spectrum w1. Therefore, the difference _d3 is a negative value. Furthermore, the degree of deviation between the spectral reflectance SP3 and the spectral reflectance SP6 is greater than the degree of deviation between the spectral reflectance SP2 and the spectral reflectance SP5. This is because the water content of the soil in the case of line L6 is relatively higher than that in the case of line L5, and the light of spectrum w2 is significantly absorbed by the water molecules contained in the soil. Therefore, the value of the difference _d3 is even smaller than the value of the difference _d2 . In other words, _d1 > _d2 > _d3 holds. In this way, it can be seen that when the water content of the soil increases, the value of the difference obtained by subtracting the spectral reflectance of another wavelength band (for example, a wavelength band corresponding to visible light) from the spectral reflectance of the infrared wavelength band in which the water contained in the soil absorbs light more significantly decreases. The water content estimation unit 33 estimates the water content of the soil based on such a tendency of the spectral reflectance spectrum of the soil. In the description of the embodiment, a wavelength band corresponding to visible light (for example, a wavelength band including 570 [nm] described later) is used as the other wavelength band, but the other wavelength band may be any wavelength band of light having a shorter wavelength than infrared light. For example, the other wavelength band may be an ultraviolet wavelength band.

12, in order to make the explanation easier to understand, the spectrum w2 included in the infrared wavelength band IR is picked up and explained. However, in order to obtain the effect of the soil water content on the degree of absorption of light in the infrared wavelength band with higher accuracy, it is necessary to use a spectral image that captures light in the infrared wavelength band over a wider range. Although it is preferable to use a spectral image acquired by a single wavelength, in the embodiment, a multispectral camera 10 that can specify light in a wavelength band corresponding to infrared rays as the spectrum of the captured image is adopted as the second imaging device 12, and the same effect is obtained. In the embodiment, in capturing a multispectral image by the second imaging device 12, a wavelength band from 900 [nm] to 1700 [mm] corresponding to infrared rays is specified as the spectrum of the captured image. Also, in the embodiment, in capturing a multispectral image by the second imaging device 12, 570 [nm] is specified as the spectrum of the captured image that does not correspond to infrared rays. These exemplified spectra of the captured images of the second imaging device 12 are merely examples, and the range of the spectrum corresponding to infrared rays and other spectra that do not correspond to infrared rays are not limited to these examples. The wavelength of 570 nm is far enough away from the infrared wavelength band (900 nm to 1700 mm) in the wavelength range that can be obtained by the second imaging device 12, and furthermore, it was adopted because it was possible to obtain the most data showing that the difference between the spectral reflectance of the infrared wavelength band and the spectral reflectance of the visible light wavelength band monotonically decreases due to an increase in the moisture content of the soil compared to other visible light wavelengths (490 nm, 570 nm, 671 nm, 800 nm).

In the following description, when the term "first spectrum" is used, it refers to the spectrum of the captured image that corresponds to infrared light in the capture of a multispectral image by the second imaging device 12. Furthermore, when the term "second spectrum" is used in the following description, it refers to the spectrum of the captured image that does not correspond to infrared light in the capture of a multispectral image by the second imaging device 12. Furthermore, the spectral reflectance of the first spectrum obtained from the multispectral image by the second imaging device 12 is referred to as the first spectral reflectance. Furthermore, the spectral reflectance of the second spectrum obtained from the multispectral image by the second imaging device 12 is referred to as the second spectral reflectance.

Figure 13 is a graph showing an example of the relationship between the first spectral reflectance, the second spectral reflectance, the difference between the first and second spectral reflectances, and the moisture content of the soil. In Figure 13, the vertical axis indicates the moisture content, and the horizontal axis indicates the spectral reflectance. Plot SP11 shown in Figure 13 shows the second spectral reflectance when the moisture content of the soil is relatively low. Plot SP31 shown in Figure 13 shows the second spectral reflectance when the moisture content of the soil is relatively high. Plot SP21 shown in Figure 13 shows the second spectral reflectance when the moisture content of the soil is between the case shown by plot SP11 and the case shown by plot SP31 (relatively intermediate). Plot SP41 shown in Figure 13 shows the first spectral reflectance when the moisture content of the soil is relatively low. Plot SP61 shown in Figure 13 shows the first spectral reflectance when the moisture content of the soil is relatively high. Plot SP21 in FIG. 13 shows the first spectral reflectance when the soil moisture content is a relatively intermediate moisture content.

The difference _d11 indicates the difference between the plot SP11 and the plot SP41. That is, the difference _d11 indicates the difference between the first and second spectral reflectances when the soil moisture content is relatively low. The difference _d21 indicates the difference between the plot SP21 and the plot SP51. That is, the difference _d21 indicates the difference between the first and second spectral reflectances when the soil moisture content is a relatively intermediate moisture content. The difference _d31 indicates the difference between the plot SP31 and the plot SP61. That is, the difference _d31 indicates the difference between the first and second spectral reflectances when the soil moisture content is relatively high.

Figure 14 is a graph with plots showing the difference between the first and second spectral reflectances shown in Figure 13. In Figure 14, the vertical axis shows the moisture content, and the horizontal axis shows the magnitude of the difference between the first and second spectral reflectances. Plot SP71 shown in Figure 14 shows the difference between the first and second spectral reflectances when the moisture content of the soil is relatively low. Plot SP81 shown in Figure 14 shows the difference between the first and second spectral reflectances when the moisture content of the soil is relatively intermediate. Plot SP91 shown in Figure 14 shows the difference between the first and second spectral reflectances when the moisture content of the soil is relatively high.

Figure 15 is a graph showing a fitting curve FC1 generated based on plots SP71, SP81, and SP91 in Figure 14. Below, with reference to Figures 13 to 15, a mechanism for creating reference data used by the water content estimation unit 33 to estimate the water content of soil will be described. In the following description, the water content estimation unit 33 is used to create such reference data, but a separate program dedicated to creating reference data may also be prepared.

First, for the same type of soil, multispectral images are captured by the second imaging device 12 in each of the cases where the soil has a relatively low moisture content, a relatively intermediate moisture content, and a relatively high moisture content. The spectra of the captured images in capturing such multispectral images are the first spectrum and the second spectrum.

The water content estimation unit 33 obtains plots SP11, SP21, and SP31 from a spectral image corresponding to the first spectrum out of the multispectral images obtained by the second imaging device 12. The water content estimation unit 33 also obtains plots SP41, SP51, and SP61 from a spectral image corresponding to the second spectrum out of the multispectral images obtained by the second imaging device 12. The water content estimation unit 33 calculates a difference _d11 between the plots SP11 and SP41, a difference _d21 between the plots SP21 and SP51, and a difference _d31 between the plots SP31 and SP61.

The moisture content estimating unit 33 manages the differences _d11 , _d21 , _d31 as plots SP71, SP81, SP91 indicating the magnitudes of the differences _d11 , _d21 , _d31 relative to the same origin position as shown in Fig. 14. The moisture content estimating unit 33 then performs a curve fitting process to derive a fitting curve FC1 corresponding to the plots SP71, SP81, and SP91 as shown in Fig. 15. The relationship between the moisture content of the soil and the difference between the first and second spectral reflectances indicated by the fitting curve FC1 functions as reference data for estimating the moisture content of the soil captured in the multispectral image from which the plots SP11, SP21, SP31, SP41, SP51, and SP61 are obtained.

The curve that is fitted in the curve fitting process is, for example, an exponential approximation line, but is not limited to this and can be changed as appropriate. Also, the specific method of the curve fitting process can be a well-known method, such as estimating an optimal function using the least squares method, so details are omitted.

The plots SP11, SP21, SP31, SP41, SP51, SP61, SP71, SP81, and SP91 shown in Figures 13 and 14 include two plots that are illustrated as overlapping at approximately the same position, but this represents an inevitable error in which perfectly matching plots are not necessarily obtained even when imaging under the same conditions, and nothing more. In deriving the fitting curve FC1 based on the plots, all plots that overlap at approximately the same position may be taken into consideration (average, etc.), or the plots that are treated as representative values may be limited.

Next, a verification experiment environment in which the above-mentioned moisture content estimation unit 33 can generate reference data will be described with reference to Figs. 16 and 17. Fig. 16 is a schematic diagram showing an example of a verification experiment environment related to the estimation of soil moisture content. Fig. 17 is a schematic diagram showing an example of the configuration of the imaging target SS in Fig. 16. As shown in Fig. 16, the second imaging device 12 is installed so that the imaging target SS is included in the imaging range, and captures a multispectral image. The spectra of the captured image in capturing such a multispectral image are the first spectrum and the second spectrum.

As shown in FIG. 17, the image target SS includes a first sample SS1, a second sample SS2, a third sample SS3, and a calibration sheet PRS. The first sample SS1 is soil with a relatively low moisture content. The second sample SS2 is soil with a relatively medium moisture content. The third sample SS3 is soil with a relatively high moisture content. Note that the first sample SS1, the second sample SS2, and the third sample SS3 are the same type of soil.

As an example, the first sample SS1 is soil with a moisture content of 10%. The second sample SS2 is soil with a moisture content of 25%. The third sample SS3 is soil with a moisture content of 40%. These moisture contents are merely examples and are not limited to these, and can be changed as appropriate. As shown in FIG. 17, the first sample SS1, the second sample SS2, and the third sample SS3 are placed in the imaging target SS, for example, in a dish or bowl-shaped container with the second imaging device 12 side open. It is desirable that the shape of the dish or container is the same regardless of the difference in the moisture content of the soil.

The calibration sheet PRS is a sheet-like member for acquiring data indicating the conditions related to brightness during imaging. The calibration sheet PRS has a nearly uniform spectral reflectance on at least one surface facing the second imaging device 12, and exhibits a known spectral reflectance.

Based on a multispectral image of such an imaging target SS captured by the second imaging device 12, a fitting curve FC1 such as that described with reference to Figures 13 to 15 can be derived. That is, plots SP11 and SP41 can be obtained from the region in the multispectral image where the first sample SS1 is captured. Furthermore, plots SP21 and SP51 can be obtained from the region in the multispectral image where the second sample SS2 is captured. Plots SP31 and SP61 can be obtained from the region in the multispectral image where the third sample SS3 is captured.

To explain in more detail the process of deriving the spectral reflectance of soil performed in this embodiment, the second imaging device 12 captures an image of the soil (or a sample such as the first sample SS1) covering the ground GR and the calibration sheet PRS, whose spectral reflectance spectrum is known, and the luminance value of the part where the soil is captured in the infrared wavelength band is divided by the luminance value of the part where the calibration sheet PRS is captured. Furthermore, the value after this division is multiplied by the average spectral reflectance of the calibration sheet in that infrared wavelength band. This makes it possible to calculate the spectral reflectance of the soil.

In addition, the light source in the verification experiment environment described with reference to Figures 16 and 17 is sunlight, but a dedicated light source capable of sufficiently irradiating light of the first and second spectra may also be used.

Figure 18 is a graph showing examples of fitting curves FC1, FC2, FC3, FC4, and FC5 for types A, B, D, F, and G. Reference data relating to the moisture content of one type of soil has been described with reference to Figures 13 to 17, but in an embodiment, by changing the types of soil used as the first sample SS1, the second sample SS2, and the third sample SS3 and performing similar processing related to capturing multispectral images and deriving fitting curves, reference data relating to the moisture content of multiple types of soil can be generated, as illustrated in Figure 18. Of course, fitting curves can be derived similarly based on multispectral images for other types of soil not illustrated in Figure 18.

Using such a fitting curve indicating the relationship between the moisture content of the soil and the difference between the first and second spectral reflectances as reference data, the moisture content estimation unit 33 can estimate the moisture content of the soil covering the ground surface GR imaged by the second imaging device 12. That is, the moisture content estimation unit 33 obtains the first and second spectral reflectances from the multispectral image imaged by the second imaging device 12, applies the difference between the obtained first and second spectral reflectances to the fitting curve indicated by the reference data, and derives the moisture content where the difference coincides on the fitting curve as the estimated moisture content.

However, when estimating the soil moisture content, the multispectral image captured by the second imaging device 12 is subjected to preprocessing (second preprocessing) using the calibration sheet PRS. This second preprocessing will be described later.

In a multispectral camera such as the second imaging device 12, which captures a relatively small number of wavelength bands in a multispectral image compared to the first imaging device 11, a method of recording the spectral reflectance of each wavelength band using multiple imaging elements equipped with optical filters that transmit light of different spectra is generally used.

Therefore, depending on the specific configuration of the second imaging device 12, an event may occur in which the location of imaging differs depending on the type of spectrum of the captured image. When such an event occurs, an imaging element that captures images with an image spectrum corresponding to infrared light and an imaging element that captures images with the spectrum of another captured image may capture images separately, and the overlapping imaging range of each captured image may be extracted. Image processing such as so-called pattern matching may be applied to such extraction, and a neural network (not shown) may be used to perform such pattern matching with higher accuracy. As a specific example, a convolutional neural network (CNN) may be used for such extraction.

When the above-mentioned event occurs, the water content estimation unit 33 extracts an imaging range that overlaps in each of the images captured by the second imaging device 12 using multiple imaging elements. The water content estimation unit 33 acquires and calculates the spectral reflectance of the spectral image captured by each imaging element in the extracted imaging range from the gradation values of the pixels of the relevant spectral image. Then, based on the difference between the spectral reflectances of the spectral images captured by each imaging element, it estimates the water content of the soil covering the captured ground surface GR. Note that if the second imaging device 12 is a multispectral camera 10 that captures multiple spectral images corresponding to each of the spectra of different captured images using a single imaging element, similar to the first imaging device 11, the extraction process is omitted.

(Preprocessing)
Next, the pre-processing will be described in the order of the first pre-processing and the second pre-processing. First, the first pre-processing will be described with reference to FIG.

Figure 19 is a diagram showing the relationship between a multispectral image including a first partial area AA1 to which the first preprocessing is applied and a second partial area AA2 to which the first preprocessing is not applied, and a graph showing an example of a comparison of the spectral reflectance spectra of these partial areas. Note that the multispectral image shown in Figure 19 is a multispectral image captured by the first imaging device 11. Also, line L11 in the graph shows the average of the spectral reflectance spectra of multiple pixels included in the first partial area AA1. Also, line L12 in the graph shows the average of the spectral reflectance spectra of multiple pixels included in the second partial area AA2.

As described above, the soil type estimation unit 32 performs preprocessing (first preprocessing) to exclude vectors that are not suitable for estimating the type of soil from the input to the neural network NN and input vectors that are suitable for estimating the type of soil to the neural network NN. Specifically, the soil type estimation unit 32 calculates the average of the spectral reflectance in the wavelength band from 350 [nm] to 450 [nm] (first average) and the average of the spectral reflectance in the wavelength band from 700 [nm] to 750 [nm] (second average) for each unit area Cr of the multispectral image captured by the first imaging device 11. Here, if the first average is α and the second average is β, the soil type estimation unit 32 treats the unit area Cr in which the relationship that α multiplied by 10 is greater than β (α×10>β) holds as a vector that is not suitable for estimating the type of soil and excludes it from the input to the neural network NN. In other words, the soil type estimation unit 32 treats the unit area Cr for which the relationship that 10 times α is equal to or less than β (α×10≦β) holds as a vector suitable for estimating the type of soil, and adopts it as an input to the neural network NN.

To give a more specific example, the first partial area AA1 in the multispectral image of FIG. 19 is a partial area in which the shadows of soil particles are captured. The average of the spectral reflectance spectra of the pixels included in the first partial area AA1 is a flat graph with almost no difference between the first average and the second average, as shown by the line L11 in the graph. In contrast, the average of the spectral reflectance spectra of the pixels included in the second partial area AA2, which has almost no shadowed areas, is a mountain-shaped graph with the second average significantly higher than the first average, as shown by the line L12 in the graph. It is desirable to estimate the type of soil based on the spectral reflectance spectrum of a partial area such as the second partial area AA2, where the second average is significantly higher than the first average and β is equal to or greater than 10 times α (α×10≦β). Therefore, in this embodiment, pixels included in a partial area such as the first partial area AA1 where α multiplied by 10 is greater than β (α×10>β) are excluded from the input to the neural network NN.

In the explanation with reference to FIG. 19, the first partial area AA1 and the second partial area AA2 are described as partial areas including multiple pixels, but the first partial area AA1 and the second partial area AA2 may be a single unit area Cr.

Next, the second pre-processing will be described with reference to Figs. 20 and 21. In the embodiment, when the second pre-processing is performed, when the imaging range SUB is imaged by the second imaging device 12, the calibration sheet PRS is placed within the imaging range SUB so that the ground GR and the calibration sheet PRS are imaged simultaneously.

20 is a diagram showing examples of multispectral images that are adopted and not adopted in the process related to the estimation of the moisture content of soil when the second pre-processing is performed. In the second pre-processing, an image in which the average of the brightness value of the partial image in which the calibration sheet PRS is captured and the brightness value of the partial image in which other than the calibration sheet PRS is captured in the multispectral image captured by the second imaging device 12 does not fall within a range determined by a predetermined lower limit value to a predetermined upper limit value is not adopted as a multispectral image used in the process related to the estimation of the moisture content of soil. Here, a multispectral image in which the average falls below the predetermined lower limit value is rejected as an image that is too dark, as shown by "Rejected (too dark)" in FIG. 20. A multispectral image in which the average exceeds the predetermined upper limit value is rejected as an image that is too bright, as shown by "Rejected (too bright)" in FIG. 20. A multispectral image in which the average falls within a range determined by a predetermined lower limit value to a predetermined upper limit value is adopted in the process by the moisture content estimation unit 33 related to the estimation of the moisture content of soil. The predetermined lower limit value and the predetermined upper limit value can be expressed as values when the maximum luminance value that the imaging element of the second imaging device 12 can detect based on its performance is 100%. The predetermined lower limit value is, for example, 7.62% but is not limited to this and can be changed as appropriate. The predetermined upper limit value is, for example, 91.6% but is not limited to this and can be changed as appropriate.

Whiteout due to being too bright and blackout due to being too dark may be determined by the threshold value of the multispectral image. In this case, the brightness threshold value for whiteout is, for example, a 16-bit gradation value where the average gradation value of the pixels of the multispectral image exceeds 60,000. Also, the brightness threshold value for blackout is, for example, a 16-bit gradation value where the average gradation value of the pixels of the multispectral image falls below 5,000. These threshold values are merely examples and can be set arbitrarily.

Note that what is referred to as the luminance value in the second pre-processing is the level of spectral reflectance indicated by the gradation value of the pixel of the multispectral image captured by the second imaging device 12.

In addition, in the second pre-processing, a multispectral image that does not satisfy a specified adoption condition is not adopted in the processing by the moisture content estimation unit 33 for estimating the moisture content of the soil, based on the coefficient of variation of the brightness values in a partial image in which the calibration sheet PRS is captured in the multispectral image captured by the second imaging device 12. Hereinafter, when simply referred to as "coefficient of variation," this refers to the coefficient of variation of the brightness values in a partial image in which the calibration sheet PRS is captured. Furthermore, when simply referred to as "specified adoption condition," this refers to the adoption condition of a multispectral image based on the coefficient of variation.

FIG. 21 is a diagram showing an example of a graph of the coefficient of variation of the brightness values of a partial image in which the calibration sheet PRS is captured in a multispectral image, and a relationship between the propriety of adoption in processing by the moisture content estimation unit 33 for estimating the moisture content of the soil. Specifically, the predetermined adoption condition in the embodiment is that the coefficient of variation is less than 0.05. If the coefficient of variation is 0.05 or more, in the embodiment, the moisture content estimation unit 33 determines that a multispectral image has been captured in which the appropriate spectral reflectance of the calibration sheet PRS has not been obtained, and does not adopt the multispectral image in processing by the moisture content estimation unit 33 for estimating the moisture content of the soil.

When a multispectral image that satisfies the specified adoption conditions is captured, for example, as shown in the "graph" in the "Adopted" column of FIG. 21, the partial image in which the calibration sheet PRS is captured has a single luminance value peak, and the degree of variance of the luminance values around the luminance value peak is extremely small. In contrast, when a multispectral image that does not satisfy the specified adoption conditions is captured, for example, as shown in the "graph" in the "Not Adopted" column of FIG. 21, the partial image in which the calibration sheet PRS is captured has multiple luminance value peaks, and the luminance values have clear undulations, and the degree of variance of the luminance values around the luminance value peak is larger than in the "Adopted" case. An example of a case in which the specified adoption conditions are not satisfied in this way is when a light reflector that functions as a secondary light source is included within the imaging range of the multispectral image captured by the second imaging device 12. In the "multispectral image" in the "rejected" column of Figure 21, there is grass growing close to the calibration sheet PRS, and it is believed that this grass acts as a secondary light source by reflecting sunlight, resulting in multiple luminance peaks and other undulations.

Note that, although neither the first nor the second preprocessing is required, performing some or all of these preprocessing steps can improve the accuracy of various estimations based on multispectral images.

(Estimation of the Corn Index)
Fig. 22 is a graph showing an example of the relationship between the moisture content of soil and the cone index. Each of the plots CP11, CP12, ..., CP19 in the graph of Fig. 22 shows a measurement value of the cone index measured using a cone penetrometer. Each of the plots CP11, CP12, ..., CP19 shows a measurement value measured on a ground surface where a common type of soil is layered and each soil has a different moisture content. In other words, the cone index shown in Fig. 22 shows the relationship between the moisture content of a certain type of soil and the cone index.

In FIG. 22, the cone index is represented by plots CP11, CP12, ..., CP19 and a straight line connecting these plots. The straight line connects two plots out of the plots CP11, CP12, ..., CP19 that have closer moisture content at the time of measurement.

As illustrated in FIG. 22, even for the same type of soil, the cone index varies according to the variation in the soil moisture content. The cone index estimation unit 34 stores data (cone index data) showing the relationship between the cone index actually measured using a cone penetrometer as described with reference to FIG. 22 and the soil moisture content for each type of soil. The relationship between the soil moisture content and the cone index differs for each type of soil.

Therefore, in the embodiment, the soil type estimation unit 32 estimates the type of soil, and the moisture content estimation unit 33 estimates the moisture content of the soil based on the multispectral image captured by the multispectral camera 10. The cone index estimation unit 34 then refers to the cone index data corresponding to the type of soil estimated by the soil type estimation unit 32, and identifies the cone index corresponding to the moisture content of the soil estimated by the moisture content estimation unit 33. The cone index estimation unit 34 treats the cone index identified in this manner as the estimated cone index.

The data that directly corresponds to the soil moisture content estimated by the moisture content estimation unit 33 is not necessarily a direct measurement value such as plots CP11, CP12, ..., CP19 in the cone index data. Therefore, in the embodiment, an interpolation process is performed as shown by the straight lines connecting the above-mentioned plots CP11, CP12, ..., CP19 to estimate a cone index that corresponds to the soil moisture content that does not correspond to a direct measurement value.

Figure 23 is a diagram showing a specific example of the interpolation process. The cone index estimation unit 34 picks up two water contents of the soil when the measured values actually measured using a cone penetrometer are obtained, such as plots CP11, CP12, ..., CP19 in Figure 22, during the interpolation process. One of the two water contents is a water content that is less than the water content of the soil estimated by the water content estimation unit 33 (estimated water content) and is closest to the estimated water content. The other of the two water contents is a water content that exceeds the estimated water content and is closest to the estimated water content. The cone index estimation unit 34 calculates the ratio mw:nw that divides the two water contents by the estimated water content. The cone index estimation unit 34 identifies the cone index indicated by the point VP that divides the line connecting the two measured values corresponding to the two moisture ratios, such as the plots CP11, CP12, ..., CP19, at the internal division ratio mqc:nqc where mw:nw = mqc:nqc, as the estimated cone index. Figure 23 shows an example in which CP(q+1) and CP(q+2) are treated as two measured values corresponding to the two moisture ratios picked up from the measured values of the cone index arranged in the order CPq, CP(q+1), CP(q+2), and CP(q+3) in order from the soil with the lowest moisture content. q is a natural number.

Note that errors may occur in the cone index measured by a cone penetrometer. Specific examples of such errors include errors caused by the user of the cone penetrometer, such as the speed or force with which the cone is pressed into the soil. Another specific example of such errors is errors caused by the condition of the soil, such as when a stone or cavity happens to be present in the soil into which the cone is pressed. It is difficult to completely eliminate these errors, but the measurement accuracy of the cone index can be ensured by measuring the cone index using a cone penetrometer a sufficient number of times and recording the average value of the measured cone index.

The above describes the estimation of soil type by the soil type estimation unit 32, the estimation of soil moisture content by the moisture content estimation unit 33, and the estimation of the cone index by the cone index estimation unit 34. Next, the flow of processing related to the estimation of the cone index performed by the information processing device 20 in the embodiment will be described with reference to Figures 24 to 27.

Figure 24 is a flowchart showing an example of the flow of processing related to the estimation of the cone index performed by the information processing device 20 in an embodiment. In the processing related to the estimation of the cone index performed by the information processing device 20, first, the soil type estimation unit 32 performs processing to estimate the type of soil (step ST1). Next, the moisture content estimation unit 33 performs processing to estimate the moisture content of the soil (step ST2). Then, the cone index estimation unit 34 performs processing to estimate the cone index (step ST3).

Figure 25 is a flowchart showing an example of the flow of the soil type estimation process (step ST1) shown in Figure 24. In the soil type estimation process, first, a multispectral image is captured by the first imaging device 11 (step ST11). Next, the soil type estimation unit 32 derives a vector of a spectral reflectance spectrum from each of the multiple unit areas Cr included in the multispectral image captured in step ST11 (step ST12). The soil type estimation unit 32 performs a first pre-processing. That is, the soil type estimation unit 32 excludes vectors that satisfy the above-mentioned α × 10 > β from the vectors acquired in step ST12 (step ST13). In addition, the soil type estimation unit 32 determines whether a predetermined number or more of spectral reflectance spectrum vectors suitable for estimating the type of soil have been obtained from the multiple unit areas Cr included in the multispectral image captured in step ST11 (step ST14). Here, if it is determined that the predetermined number or more of the vectors have not been obtained (step ST14; No), the process proceeds to step ST11. That is, the first imaging device 11 captures a multispectral image again. On the other hand, if it is determined that the vector has been obtained a predetermined number or more (step ST14; Yes), the soil type estimation unit 32 estimates the type of soil using the vector of the spectral reflectance spectrum obtained from the multispectral image and the neural network NN (step ST15). That is, the soil type estimation unit 32 inputs the vector of the spectral reflectance spectrum obtained from the multispectral image to the neural network NN, and estimates the type of soil based on the node values of the output layer of the neural network NN.

Figure 26 is a flow chart showing an example of the flow of the soil moisture ratio estimation process (step ST2) shown in Figure 24. First, the second imaging device 12 captures a multispectral image (step ST21). Next, the moisture ratio estimation unit 33 performs a second pre-processing. That is, the moisture ratio estimation unit 33 performs a judgment on the luminance value using a partial image of the calibration sheet PRS included in the multispectral image captured in step ST21, and checks whether the judgment result on the luminance value indicates that the multispectral image is a multispectral image suitable for estimating the moisture ratio of soil (step ST22). More specifically, the judgment on the luminance value using the partial image of the calibration sheet PRS may be, for example, one or both of the judgment described with reference to Figure 20 and the judgment described with reference to Figure 21. The determination described with reference to Fig. 20 refers to a determination as to whether the average of the luminance values of the partial image in which the calibration sheet PRS is captured and the luminance values of the partial image in which other than the calibration sheet PRS is captured in the multispectral image captured by the second imaging device 12 falls within a range determined by a predetermined lower limit value and a predetermined upper limit value. The determination described with reference to Fig. 21 refers to a determination as to whether the variation coefficient of the luminance values of the partial image in which the calibration sheet PRS is captured in the multispectral image captured by the second imaging device 12 satisfies a predetermined adoption condition.

If it is determined in step S22 that the multispectral image is not suitable for estimating the moisture content of the soil (step ST22; No), the process proceeds to step ST21. That is, the second imaging device 12 captures a multispectral image again. On the other hand, if it is determined in step S22 that the multispectral image is suitable for estimating the moisture content of the soil (step ST22; Yes), the moisture content estimation unit 33 acquires the spectral reflectance of each of the infrared wavelength band and the visible light wavelength band of the multispectral image (step ST23). The moisture content estimation unit 33 estimates the moisture content of the soil based on the difference between the spectral reflectance of each of the infrared wavelength band and the visible light wavelength band of the multispectral image (step ST24). Here, in the embodiment, the moisture content of the soil is estimated using reference data corresponding to the type of soil estimated in step ST1.

Figure 27 is a flow chart showing an example of the flow of the cone index estimation process (step ST3) shown in Figure 24. The cone index estimation unit 34 reads out cone index data corresponding to the type of soil estimated in the process of step ST1 (step ST31). The cone index data is included in advance in the cone index estimation unit 34. The cone index estimation unit 34 identifies a cone index corresponding to the water content of the soil estimated in the process of step ST2 based on the cone index data (step ST32), and treats the identified cone index as an estimated cone index.

In the description with reference to Figures 25 and 26, the multispectral image is captured once, but in order to perform estimation with higher accuracy, the multispectral image may be captured multiple times and the multiple multispectral images thus obtained may be used to perform divided cross-validation multiple times (k). k is a natural number of 2 or more. Specifically, k is, for example, 5, but is not limited to this, and may be 4 or less or 6 or more.

Next, a verification experiment for estimating the cone index will be described with reference to Figs. 28 to 31. Fig. 28 is a schematic diagram showing the relationship between the backhoe 100 and the imaging range SUB (see Fig. 2) in the verification experiment. The imaging range SUB located in front of the backhoe 100 has a depth SUB1 in the front-rear direction of, for example, 5 m. The imaging range SUB has a width SUB2 along the ground GR that is perpendicular to the depth SUB1 of, for example, 3 m. In the verification experiment, the imaging range SUB with these dimensions was excavated 0.5 m underground, and then the soil resulting from the excavation was backfilled with soil whose moisture content had been adjusted by adding water.

Figure 29 is a graph showing the relationship between the moisture content and the cone index of the soil excavated and backfilled in the verification experiment described with reference to Figure 28. In Figure 29, the cone index is represented by plots CP21, CP22, ..., CP30 showing the cone index measured using a cone penetrometer, and lines connecting these plots. Although not shown, reference data for the soil excavated and backfilled in the verification experiment is previously included in the moisture content estimation unit 33. In addition, the soil type estimation unit 32 has already implemented machine learning capable of estimating the type of the soil.

In the verification experiment, three patterns of imaging ranges SUB with different soil moisture contents were created, and the three patterns of imaging ranges SUB were each imaged with the multispectral camera 10, and the cone index was estimated individually. Note that in the verification experiment, imaging was performed under clear skies and in an environment with sufficient sunlight.

Figure 30 is a table showing the relationship between the measured and estimated values of the soil moisture content and cone index for three patterns. As shown in Figure 30, the measured value of the soil moisture content for pattern 1 was 38.8%. In contrast, the soil moisture content for pattern 1 estimated by the moisture content estimation unit 33 based on the multispectral image captured by the second imaging device 12 was 42%. Therefore, in pattern 1, the error between the measured and estimated values of the soil moisture content was 3.3%. In addition, the measured value of the soil moisture content for pattern 2 was 46.2%. In contrast, the soil moisture content for pattern 1 estimated by the moisture content estimation unit 33 based on the multispectral image captured by the second imaging device 12 was 47.4%. Therefore, in pattern 1, the error between the measured and estimated values of the soil moisture content was 1.2%. The actual moisture content of the soil in pattern 3 was 45.3%. In contrast, the moisture content of the soil in pattern 1 estimated by the moisture content estimation unit 33 based on the multispectral image captured by the second imaging device 12 was 48.5%. Therefore, in pattern 1, the error between the actual moisture content and the estimated value was 3.2%. The average error between the actual moisture content and the estimated value of the soil moisture content for the three patterns was approximately 2.6%.

The actual measured value of the cone index for pattern 1 was 461 [kN/m ² ]. On the other hand, the estimated value of the cone index for pattern 1 estimated by the cone index estimation unit 34 was 233 [kN/m ² ]. Therefore, in pattern 1, the error between the actual measured value and the estimated value of the soil moisture content was 228 [kN/m ² ]. The actual measured value of the cone index for pattern 2 was 72.1 [kN/m ² ]. On the other hand, the estimated value of the cone index for pattern 1 estimated by the cone index estimation unit 34 was 71.7 [kN/m ² ]. Therefore, in pattern 1, the error between the actual measured value and the estimated value of the soil moisture content was 0.4 [kN/m ² ]. On the other hand, the actual measured value of the cone index for pattern 3 was 64.1 [kN/m ² ]. In contrast, the estimated value of the cone index for pattern 1 estimated by the cone index estimation unit 34 was 71.4 [kN/ ^m2 ]. Therefore, the error between the actual and estimated values of the soil moisture content was 7.3 [kN/ ^m2 ] for pattern 1. In addition, the average error between the actual and estimated values of the cone index for the three patterns was approximately 79 [kN/ ^m2 ].

Figure 31 is a graph showing the relationship between the measured and estimated values of the cone index for three patterns. In the soil excavated and backfilled in the verification experiment, the cone index fluctuates by 50 or more with a moisture content fluctuation of about 1% in the moisture content range of about 40% (38% to 42%). Therefore, in pattern 1, in which the soil with a moisture content of about 40% was used as the subject of the cone index estimation, the degree of influence of the error between the measured and estimated moisture content of the soil on the error between the measured and estimated moisture content of the cone index is large. The verification experiment results for pattern 1 were an example in which such an influence was extremely noticeable. On the other hand, the cone index of the soil hardly fluctuates even if the moisture content fluctuates from about 45%. For this reason, in patterns 2 and 3, in which soil with a moisture content of over 45% was used to estimate the cone index, the degree to which the error between the actual and estimated moisture content of the soil affects the error between the actual and estimated cone index is small. The results of the verification experiments for patterns 2 and 3 were examples that support this.

The above describes the estimation of the cone index, but the estimation system 1 may further determine the travelling ability of the backhoe 100 based on the estimation of the cone index. Specifically, the cone index estimation unit 34 may further have a function of comparing the estimated cone index with the cone index required for the backhoe 100 to travel, and estimating that the backhoe 100 can travel if the estimated cone index is equal to or greater than the cone index required for the backhoe 100 to travel, and estimating that the backhoe 100 can travel if the estimated cone index is not equal to or greater than the cone index required for the backhoe 100 to travel, and estimating that the backhoe 100 can travel if this is not the case. In this case, the cone index estimation program 223 further includes a program corresponding to this function, etc.

FIG. 32 is a table showing an example of the determination result of the traversability of the backhoe 100 performed based on the cone index estimated in the three patterns shown in FIG. 30. For example, if the backhoe 100 is a construction machine classified as a so-called super-wetland bulldozer, the backhoe 100 can traverse a ground surface having a cone index of 200 [kN/m ² ] or more. Therefore, as shown in FIG. 32, the cone index estimation unit 34 determines that the backhoe 100 can traverse the ground surface of pattern 1 in which the cone index of 228 [kN/m ² ] is estimated. On the other hand, the cone index estimation unit 34 determines that the backhoe 100 cannot traverse the ground surface of pattern 2 in which the cone index of 71.7 [kN/m ² ] is estimated and the ground surface of pattern 3 in which the cone index of 71.4 [kN/m ² ] is estimated.

In addition, in the estimation of the cone index and the determination of the traversability based on the estimated cone index, the type of soil for which the cone index is estimated may be limited. For example, while sandy soil and gravel soil maintain a cone index at a level that allows construction machinery to traverse almost regardless of the increase or decrease in water content, it is known that clayey soil, including volcanic ash clayey soil, tends to have a lower cone index as the water content increases, making it difficult for construction machinery to traverse. Therefore, if the type of soil estimated by the soil type estimation unit 32 is clayey soil, the estimation of the water content and the cone index may be performed, and if the type of soil estimated by the soil type estimation unit 32 is not clayey soil, the estimation of the water content and the cone index may be omitted. If the estimation of the water content and the cone index is omitted, the determination of the traversability automatically determines that the traversability is possible. Also, if the estimation of the water content and the cone index is performed, the determination of the traversability may be performed based on the estimated cone index.

As described above, the object estimation method of the embodiment includes an imaging step (step ST11) of capturing an image of an object using a multispectral camera (e.g., first imaging device 11), a derivation step (step ST12) of deriving a vector of a spectral reflectance spectrum from the image captured in the imaging step, and an object estimation step (step ST15) of estimating the object based on the vector and a trained neural network NN based on images of each type of object captured in advance by the multispectral camera. This allows estimation of the imaged object, etc., to be performed without contact. By using the trained neural network NN, the accuracy of the object estimation can be further improved.

In addition, in the object estimation step (step ST15), among the vectors of the spectral reflectance spectrum, when the average of the spectral reflectance in the wavelength band from 350 [nm] to 450 [nm] is α and the average of the spectral reflectance in the wavelength band from 700 [nm] to 750 [nm] is β, the vectors for which α×10>β holds are not used in the process related to the estimation of the object, thereby further improving the accuracy of the estimation of the object.

In addition, the calculation unit 21 realizes a process of deriving a vector of the spectral reflectance spectrum from a unit area included in the image of the object, and estimating the object based on the vector and a neural network NN that has been trained based on images of each type of object previously captured by a multispectral camera (e.g., the first image capture device 11), so that a process related to estimating the object of an image captured in a non-contact manner can be realized by so-called software processing.

In addition, since the number of nodes in the intermediate layer of the neural network NN is less than the number of nodes in the input layer of the neural network NN, processing related to estimating the object can be achieved with a lower load.

In addition, by mounting a multispectral camera 10 on a vehicle such as a backhoe 100, it is possible to capture images of the ground that the vehicle is expected to travel over, and to make non-contact estimates of objects related to the vehicle.

Furthermore, the water content estimation method of the embodiment includes an imaging step (step ST21) of capturing an image of the object using a multispectral camera (e.g., the second imaging device 12), a spectral reflectance acquisition step (step ST23) of acquiring a first spectral reflectance, which is the spectral reflectance of the object in the infrared wavelength band, and a second spectral reflectance, which is the spectral reflectance of the object in another wavelength band (e.g., the wavelength band of visible light), based on the image, and a water content estimation step (step ST24) of estimating the water content of the object based on the difference between the first spectral reflectance and the second spectral reflectance. This makes it possible to estimate the water content of the object in a non-contact manner with high accuracy.

Furthermore, in the water content estimation step described above, the water content of the object is estimated based on data (e.g., the above-mentioned reference data) that indicates the correspondence between the difference between the first and second spectral reflectances for each type of object stored in advance and the water content. This further improves the accuracy of estimating the water content of the object.

Furthermore, the moisture content estimation method of the embodiment includes a determination step (step ST22) of determining whether the brightness of the multispectral image captured by the second imaging device 12 is appropriate based on the calibration sheet PRS that shows a known spectral reflectance, and does not estimate the moisture content of the object based on a multispectral image that is not appropriate in brightness. This further improves the accuracy of estimating the moisture content of the object.

Furthermore, in the embodiment, the target object is ground GR covered with soil. The cone index estimation method of the embodiment includes a cone index estimation step (step ST32) of estimating the cone index of the ground GR based on the estimated type of soil and the moisture content estimated in the moisture content estimation step described above. Therefore, by capturing an image of the ground GR covered with soil, the cone index of the ground GR can be estimated.

Furthermore, in the above-mentioned cone index estimation step, the cone index of the ground surface GR is estimated based on pre-stored data showing the correspondence between the moisture content and the cone index for each type of soil. This further improves the accuracy of the cone index estimation.

Note that the specific application of the estimation system 1 is not limited to the application in which the multispectral camera 10 is provided on the backhoe 100. For example, the multispectral camera 10 may be provided so that an image of the ground below can be captured from a flying aircraft. This allows for the acquisition of estimated information on the type of soil covering the ground in the area where the aircraft has flown, the soil's moisture content, and the cone index. Therefore, in areas where it is necessary to enter construction machinery or the like to deal with disasters, etc., information on the drivability of construction machinery or the like in the area can be obtained without contact. Note that the aircraft also includes unmanned aerial vehicles known as drones. The multispectral camera 10 may also be provided on an artificial satellite instead of an aircraft.

As an example, if estimated information for a specified range (e.g., 10 m x 10 m) can be obtained based on a multispectral image, it is possible to determine whether a machine such as the backhoe 100 can travel through the specified range. As a different example, the multispectral image may be divided into multiple divided regions, and estimation may be performed for each divided region. For example, a multispectral image of a specified range may be divided into 100 in the x-axis direction and 100 in the y-axis direction to set 10,000 divided regions. This makes it possible to determine whether a vehicle such as the backhoe 100 can travel through each divided region.

The number of divided regions is not limited, and the minimum unit of the divided regions can be one pixel, and the maximum unit is not limited. If the area included in the multispectral image is determined by the altitude and angle with respect to the ground when the multispectral camera 10 captures the image, the area range of the ground that can be estimated in divided regions can be easily determined. Therefore, by making the divided regions from which the estimated information is to be obtained smaller, the traveling route can be selected more precisely and appropriately. In addition, making the divided regions from which the estimated information is to be obtained smaller is suitable for making an estimation based on a multispectral image captured by a multispectral camera 10 mounted on a vehicle including a caterpillar or a multispectral camera 10 mounted on an aircraft or artificial satellite located at a higher altitude. In this way, a configuration that can use an aircraft or the like that can easily capture a wider range of multispectral images from a higher altitude than a vehicle traveling on the ground will be described with reference to FIG. 33.

Figure 33 is a diagram showing an example of the configuration of a backhoe 100 to which a modified embodiment of the present invention is applied. In the backhoe 100 shown in Figure 33, an aircraft takeoff and landing unit 105 is provided on the upper side of the roof 101 instead of the multispectral camera 10 shown in Figure 2. The aircraft takeoff and landing unit 105 is a platform-like structure having a flat surface on which an aircraft 200 can take off and land.

The aircraft 200 is an unmanned aircraft that can fly by rotating a plurality of rotors 202 rotatably supported by a housing 201. As shown in FIG. 33, the aircraft 200 functions as an aircraft that flies with a multispectral camera 10 mounted thereon and captures an image of the imaging range SUB by directing the angle of view of the multispectral camera 10 downward. The imaging device 14 shown in FIG. 33 is a multispectral camera 10 that performs the functions of both the first imaging device 11 and the second imaging device 12 described above. Compared to a case in which the multispectral camera 10 is fixed to the backhoe 100, the aircraft 200 can more easily capture a wider range of the imaging range SUB. Therefore, information on the type of soil covering the imaging range SUB, the moisture content and cone index of the soil, and the traversability when the backhoe 100 is in the imaging range SUB can be obtained more quickly. In addition, the aircraft 200 shown in FIG. 33 is connected to the backhoe 100 via a cable 205. According to the configuration shown in FIG. 33, the data of the multispectral image obtained by the imaging device 14 can be transmitted to the backhoe 100 in real time via the cable 205. The cable 205 is not limited to transmitting the data of the multispectral image, and may be used to supply power to rotate the rotor 201, or may be used for other purposes. In addition, the transmission path of the multispectral image data from the aircraft 200 to the backhoe 100 may be partially or entirely wireless. In addition, by using the cable 205 for power supply, it is not necessary to install a battery on the aircraft 200. Therefore, the weight that the multispectral camera 10 installed on the aircraft 200 can occupy can be more generous than the maximum takeoff weight of the aircraft 200.

The object of imaging by the multispectral camera 10 is not limited to soil. For example, concrete may be imaged by the multispectral camera 10 to estimate the water content of the cement. This makes it possible to estimate whether the amount of water mixed into cement hydrate, such as concrete, is appropriate. In addition, if the water content of a substance, not limited to cement, affects its quality, it is possible to obtain estimated information regarding the quality of the substance by imaging the substance with the multispectral camera 10.

Furthermore, the specific form of the multispectral camera 10 is not limited to the first imaging device 11 and the second imaging device 12. For example, a single multispectral camera having the functions of both the first imaging device 11 and the second imaging device 12 may be adopted as the multispectral camera 10. In that case, the imaging of a multispectral image by the above-mentioned first imaging device 11 or the second imaging device 12 can be the imaging of a multispectral image by such a single multispectral camera.

In addition to the backhoe 100, other vehicles that run on caterpillar tracks or wheels can also be equipped with a multispectral camera 10, similar to the backhoe 100 described with reference to Figures 3 and 33, and can utilize the cone index estimation method by the estimation system 1 and various estimation methods included in the cone index estimation method.

Reference Signs List 1 Estimation system 10 Multispectral camera 11 First imaging device 12 Second imaging device 20 Information processing device 21 Calculation unit 31 Acquisition unit 32 Soil type estimation unit 33 Moisture content estimation unit 34 Cone index estimation unit 100 Backhoe PRS Calibration sheet

Claims (8)

capturing an image of an object using a multispectral camera;
a spectral reflectance acquisition step of acquiring a first spectral reflectance, which is a spectral reflectance of the object in an infrared wavelength band, and a second spectral reflectance, which is a spectral reflectance of the object in another wavelength band, based on the image;
a moisture content estimating step of estimating a moisture content of the object based on a difference between the first spectral reflectance and the second spectral reflectance;
Including,
the other wavelength band is a wavelength band of light having a shorter wavelength than the infrared wavelength band,
A derivation step of deriving a vector of a spectral reflectance spectrum from an image of the object captured by a multispectral camera;
and an object estimation step of estimating a type of the object based on the vector and a pre-stored spectral reflectance spectrum for each type of the object,
In the water content estimation step, the water content of the object is estimated based on data indicating a correspondence relationship between the water content and a difference between the first spectral reflectance and the second spectral reflectance for each type of the object, the data being stored in advance;
a vector for which α×10>β holds when the average spectral reflectance in the wavelength band from 350 [nm] to 450 [nm] is α and the average spectral reflectance in the wavelength band from 700 [nm] to 750 [nm] is β, is not used in the object estimation step. The water content estimation method according to claim 1 , wherein in the object estimation step, the type of the object is estimated using a trained neural network based on images of the object for each type captured in advance by a multispectral camera. a determining step of determining whether the brightness of the image is appropriate based on a calibration sheet showing a known spectral reflectance;
The water content estimating method according to claim 1 or 2, further comprising the step of: not estimating the water content of the object based on the image having inappropriate brightness. An imaging step of capturing an image of ground including soil by a multispectral camera;
a spectral reflectance acquisition step of acquiring a first spectral reflectance, which is a spectral reflectance of the ground in an infrared wavelength band, and a second spectral reflectance, which is a spectral reflectance of the ground in another wavelength band, based on the image;
A derivation step of deriving a vector of a spectral reflectance spectrum from an image of the ground captured by a multispectral camera;
an object estimation step of estimating the type of soil based on the vector and a pre-stored spectral reflectance spectrum for each of the soil types;
a moisture content estimating step of estimating the moisture content of the ground based on data indicating a correspondence relationship between the difference between the first spectral reflectance and the second spectral reflectance for each type of soil, the difference being stored in advance, and the moisture content;
a cone index estimating step of estimating a cone index of the ground based on the type of soil estimated in the object estimating step and the moisture content estimated in the moisture content estimating step;
Including,
the other wavelength band is a wavelength band of light having a shorter wavelength than the infrared wavelength band ,
a vector for which α×10>β holds when the average spectral reflectance in the wavelength band from 350 [nm] to 450 [nm] is α and the average spectral reflectance in the wavelength band from 700 [nm] to 750 [nm] is β, is not used in the object estimation step. The cone index estimating method according to claim 4 , wherein in the cone index estimating step, the cone index of the soil is estimated based on pre-stored data indicating a correspondence between the moisture content and the cone index for each type of soil. A multispectral camera that captures images of the soil;
and an information processing device that estimates a cone index of the soil based on the image,
the information processing device includes a calculation unit that acquires a first spectral reflectance which is a spectral reflectance of the soil in an infrared wavelength band and a second spectral reflectance which is a spectral reflectance of the soil in another wavelength band based on the image, estimates the moisture content of the soil based on a difference between the first spectral reflectance and the second spectral reflectance based on pre-stored data indicating a correspondence relationship between the difference between the first spectral reflectance and the second spectral reflectance for each type of soil and the moisture content, and uses information indicating the estimated moisture content of the soil as information related to estimating a cone index of the soil,
the other wavelength band is a wavelength band of light having a shorter wavelength than the infrared wavelength band,
The calculation unit is
Deriving a vector of a spectral reflectance spectrum from an image of the soil captured by a multispectral camera;
a vector for which α×10>β holds, where α is the average spectral reflectance in a wavelength band from 350 nm to 450 nm and β is the average spectral reflectance in a wavelength band from 700 nm to 750 nm, and the vectors for which α×10>β holds are excluded, and the soil type is estimated based on the pre-stored spectral reflectance spectra for each soil type. The multispectral camera is mounted on a vehicle.
The cone index estimation system of claim 6 . The multispectral camera is mounted on an unmanned aerial vehicle.
The cone index estimation system of claim 6 .

Images