JP7587282B2 - Method, program and information processing device - Google Patents

Description

The present disclosure relates to a method, a program, and an information processing device.

There is known technology for capturing images of land on which agricultural crops (hereinafter generally referred to as "crops" in this specification) are grown using a spectral camera to obtain a spectral image of the land, with the aim of understanding the growth of the crops, pests, and soil conditions (see, for example, Patent Document 1).

International Publication No. 2017/179378

For agricultural businesses, when it comes to growing and harvesting crops, it is assumed that the growers know where the crops were planted. However, when it comes to harvesting and shipping the crops, it may not be possible to determine whether the crops are growing well simply by looking at the density of vegetation.

Therefore, businesses that grow and harvest crops need technology that allows them to more appropriately manage the growing conditions.

The present disclosure has been made to solve the above problem, and its purpose is to provide a method, program, and information processing device that enables harvesting businesses to more appropriately manage the growing conditions.

According to one embodiment, a method for operating a computer is provided. The method causes a processor of the computer to execute the following steps: a first step of storing in a memory information for detecting that a crop being observed is in a specific state, for comparison with a spectral image captured by a multispectral camera; a second step of generating a spectral image in the observed region by capturing an image with the multispectral camera at a wavelength for detecting that the crop is in a specific state; a third step of identifying a region in the observed region where a crop in a specific state is located, based on the spectral image resulting from the capturing and the information for detecting that the crop is in a specific state; and a fourth step of outputting information on the identified region.

This disclosure enables harvesting businesses to more appropriately manage the growing conditions.

1 is a diagram showing an overview of a system according to an embodiment; 1 is a block diagram showing a configuration of a system according to an embodiment. FIG. 1 is a front view showing a drone according to an embodiment. FIG. 1 is a plan view showing a drone according to an embodiment. FIG. 2 is a block diagram showing the hardware configuration of the drone according to the embodiment. FIG. 2 is a block diagram showing the functional configuration of a drone according to an embodiment. FIG. 1 is a conceptual diagram showing an image capturing state of land by a drone according to an embodiment. FIG. 2 is a block diagram showing a hardware configuration of a satellite according to the embodiment. FIG. 2 is a block diagram showing the functional configuration of a satellite according to the embodiment. FIG. 2 is a conceptual diagram showing an image capturing state of land by a satellite according to the embodiment. 1 is a block diagram showing a hardware configuration of an information processing device according to an embodiment. FIG. 2 is a diagram illustrating a functional configuration of an information processing device according to an embodiment. FIG. 4 is a diagram showing a data structure of a first imaging condition DB stored in the satellite according to the embodiment. FIG. 2 is a diagram showing a data structure of an image DB stored in a satellite according to the embodiment. A diagram showing the data structure of a second imaging end time DB stored in the drone according to the embodiment. A diagram showing the data structure of a flight plan DB stored in the drone of the embodiment. 4 is a diagram showing a data structure of a wide-area image DB stored in the information processing device according to the embodiment; FIG. 4 is a flowchart for explaining an example of an operation of a satellite according to an embodiment. 10 is a flowchart illustrating an example of an operation of the information processing device according to the embodiment. 11 is a flowchart illustrating an example of an operation of a drone according to an embodiment. 10 is a flowchart for explaining another example of the operation of the information processing device according to the embodiment. 11A to 11C are diagrams illustrating an example of an operation of generating a wide-area image in the information processing device according to the embodiment. FIG. 1 is a conceptual diagram showing a procedure for determining whether a crop is in a particular state using a spectral image. FIG. 11 is a diagram illustrating an example of a screen displayed on the information processing device according to the embodiment. FIG. 11 is a diagram showing another example of a screen displayed on the information processing device according to the embodiment.

Embodiments of the present disclosure will be described below with reference to the drawings. In all figures describing the embodiments, common components are given the same reference numerals, and repeated explanations will be omitted. Note that the following embodiments do not unduly limit the contents of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each figure is a schematic diagram, and is not necessarily a precise illustration.

In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be a single-core or multi-core.

In addition, at least one processor may be a processor in the broad sense, such as a hardware circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs some or all of the processing.

In the following explanation, information that produces an output for an input may be described using expressions such as "xxx table," but this information may be data of any structure, or may be a learning model such as a neural network that produces an output for an input. Therefore, a "xxx table" may be called "xxx information."

In addition, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or two or more tables may all or partly be one table.

In the following explanation, the processing may be described with the "program" as the subject, but since the program is executed by a processor to perform a predetermined process using a memory unit and/or an interface unit as appropriate, the subject of the processing may be the processor (or a device such as a controller having the processor).

The program may be installed on a device such as a computer, or may be, for example, on a program distribution server or on a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

In addition, in the following description, identification numbers are used as identification information for various objects, but other types of identification information (e.g., identifiers including letters or codes) may also be used.

In the following description, when elements of the same type are described without distinction, reference signs (or common signs among reference signs) are used, and when elements of the same type are described with distinction, the identification numbers (or reference signs) of the elements are used.

In addition, in the following explanation, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

In this specification, a "spectral image" refers to an image consisting of only reflected light of a specific wavelength band (including a specific wavelength) from an object. As an example, as shown in the embodiment described below, a spectral image is generated based on an image signal output from an image sensor by passing reflected light from an object (crop) through a filter (preferably with a variable transmission wavelength band) that passes only a specific wavelength band. Alternatively, the reflected light from an object is imaged on an image sensor without passing through a filter, and the image signal output from the image sensor is converted (filtered) into an image signal of only a specific wavelength band by signal processing. Note that an image of a collection of multiple spectral images consisting of only reflected light of a specific wavelength band from the same object may also be referred to as a spectral image.

In addition, in this specification, the term "aircraft" refers to a moving object that flies through the air and can be controlled without a crew member (for example, an aircraft, drone, satellite, or airship that can be remotely controlled by radio or autonomously).

Furthermore, in this specification, a crop being in a "specific state" includes the following.
- Whether the crop is diseased, and what disease it has, or is likely to develop - The state of growth of the crop (including whether it is ready to be harvested)
- If the target area is forest, the state of vegetation and distribution of vegetation

Crops include not only agricultural crops but also vegetation. Therefore, when a crop is in a particular state, it also includes the state of the vegetation on which the crop is growing (for example, if the crop is apples, the state of the leaves on the tree on which the apples are growing).

<Embodiment>
<Overview of the embodiment>
In the system according to the embodiment, the ground surface including the area where crops are grown is imaged by a first multispectral camera, and an operator checks the spectral image obtained by the image capture, and associates the difference in the appearance of the spectral image in a specific wavelength band in a certain area (which corresponds to the intensity of reflected light in a specific wavelength band and reflected light in other wavelength bands for the same object) with a specific state of the crops in that area. This association is stored as a library. Next, the area to be observed is similarly imaged by a second multispectral camera to obtain a spectral image. The spectral image of the area to be observed is then analyzed according to the above-mentioned library, and an area in this area where crops are in a specific state is identified.

The first and second multispectral cameras are preferably mounted on a mobile object, and the spectral images are acquired by moving the mobile object over the ground surface and the area to be observed. Suitable examples of the mobile object include an airborne drone and a satellite, but there are no particular limitations as long as the mobile object can acquire spectral images by moving the first and second multispectral cameras. Furthermore, the mobile object does not have to be an airborne object, and the mobile object on which the first and second spectral cameras are mounted may move over the ground surface. For example, the spectral images may be acquired by a means such as an operator holding the first and second spectral cameras moving on foot over the ground surface, or a vehicle on which the first and second spectral cameras are mounted driving over the ground surface. One example is a mode in which a multispectral camera that realizes the spectroscopic terminal device disclosed in Japanese Patent No. 6342594 using a smartphone is used.

The moving body on which the first multispectral camera is mounted and the moving body on which the second multispectral camera is mounted do not have to be the same. In the embodiment described below, the first multispectral camera is mounted on a satellite and the second multispectral camera is mounted on a drone, but the first multispectral camera may be mounted on a drone and the second multispectral camera may be mounted on a satellite. Furthermore, both the first and second multispectral cameras may be mounted on a satellite, or both the first and second multispectral cameras may be mounted on a drone. Furthermore, as described above, at least one of the first and second multispectral cameras may be mounted on a moving body moving on the ground surface.

Both the first and second multispectral cameras receive light reflected from the earth's surface in multiple wavelength bands and generate spectral images based on the reflected light. Preferably, the first multispectral camera has a member capable of switching the wavelength to be imaged, such as a liquid crystal wavelength tunable filter, and can generate spectral images for a greater number of wavelengths than the second multispectral camera. On the other hand, the second multispectral camera can generate spectral images for a specific wavelength. In this case, it is preferable that the specific wavelength cannot be switched while the area to be observed by the second multispectral camera is being imaged. In other words, the specific wavelength is a predetermined wavelength. Naturally, there is no particular restriction on the magnitude relationship between the number of wavelength bands of the spectral images that can be imaged by the first multispectral camera and the number of wavelength bands of the spectral images that can be imaged by the second multispectral camera.

In this specification, the term "library" refers to a highly versatile data set and/or program. Preferably, the library is realized as training data in artificial intelligence technology. More preferably, in the system of this embodiment, a learning model is generated using the training data, and an area where crops are in a specific state is identified based on this training model. Therefore, in the following description, the term "library" is used mainly to refer to training data, but may be referred to as "library" without making any particular distinction between training data and training models. Note that in the following description, training models also include models that have already been trained, i.e., trained models.

As will be described in more detail below, when creating a library, it is necessary to take into account the solar altitude at the time the spectral images were captured. This is because the wavelength distribution of light reflected from crops (especially crop leaves) differs depending on the angle at which the sun hits the crop, so it is necessary to consider the effect of the solar altitude (sun angle) on the library. However, when creating a library based on spectral images captured by the first multispectral camera, there may be a lack of variation in the spectral images captured by the first multispectral camera (especially the variation in spectral images based on different solar angles) at the beginning of operation of the system, making it difficult to create a library that takes the solar angle into account.

Therefore, the spectral image captured by the first multispectral camera is taken at a certain solar angle (for example, a spectral image at noon), and a library is created based on the spectral image at the certain solar angle. Next, when acquiring a spectral image with the second multispectral camera, the spectral image is acquired at the same solar angle as when the spectral image was acquired by the first multispectral camera, and the influence of the solar angle can be eliminated as much as possible to identify the area where the crop is in a specific state.

Then, the variation of the spectral images acquired by the first and second multispectral cameras can be increased, i.e., spectral images at various solar angles can be acquired, and the effect of the solar angle on the library can be formulated.

When creating a library, and also when identifying areas where crops are in a specific state, there is a need to create a library for a wide range of observation target areas and to perform area identification work. On the other hand, the spectral images captured by the first and second multispectral cameras have a predetermined imaging range depending on the angle of view of each multispectral camera and the altitude from the ground surface of the mobile body carrying the multispectral camera. In this case, it is preferable to generate a wide range of spectral images by linking multiple spectral images. Methods for linking spectral images are well known, and one example is a method of generating orthoimages from spectral images and linking these orthoimages. The method of linking spectral images may be applied to spectral images acquired from either the first multispectral camera or the second multispectral camera.

With reference to FIG. 1, an overview of a system to which a method according to an embodiment is applied will be described. In the system according to this embodiment, an example in which a first multispectral camera is mounted on a satellite and a second multispectral camera is mounted on a drone will be mainly described, but the moving body on which the first and second multispectral cameras are mounted and the method of moving the first and second multispectral cameras are not limited to the disclosure of the embodiment. Furthermore, the configuration of the first and second multispectral cameras (particularly the number of wavelength bands of the spectral images that can be acquired) is also not limited to the disclosure of the embodiment.

In the system 1 according to the embodiment, spectral images for creating the library are generated by a first multispectral camera mounted on a satellite 2, which is an example of an aircraft, and the area to be observed is imaged by a second multispectral camera mounted on a drone 3, which is an example of an aircraft, and the obtained spectral images are used to identify areas where crops are in a particular state.

However, the aircraft for capturing the spectral images for creating the library and the aircraft for capturing the spectral images for identifying the area may be the same, and there are no restrictions on the type. However, capturing the spectral images for creating the library using a satellite 2 has the advantage of being able to efficiently obtain spectral images from a wider area of the earth's surface. On the other hand, capturing the spectral images for identifying the area using a drone 3 has the advantage of being able to easily obtain spectral images by flying the drone 3 only in the area where the condition needs to be identified, and also of being able to reduce the cost of capturing images of the same area multiple times.

Although details will be described later, the satellite 2 is equipped with a first multispectral camera using a liquid crystal tunable filter (LCTF), while the drone 3 is equipped with a second multispectral camera (preferably a multispectral camera equipped with multiple cameras that can capture only reflected light in specific wavelength bands) that can capture spectral images with a combination of multiple wavelength bands that can clearly identify that the crop is in a specific state in library creation. Naturally, there are no limitations on the configuration of the multispectral camera as long as it can capture images based on reflected light in multiple wavelength bands. Alternatively, the drone 3 may be equipped with a first multispectral camera using an LCTF, and the satellite 2 may be equipped with a second multispectral camera that can capture spectral images with a combination of multiple wavelength bands.

The first multispectral camera mounted on satellite 2 can capture reflected light in a specific wavelength band among the reflected light in a wide wavelength band from the visible range to the near infrared range by the action of a liquid crystal wavelength tunable filter. In capturing reflected light from the Earth's surface, system 1 of this embodiment scans a wide wavelength band and captures a spectral image based on reflected light in multiple wavelength bands for the same area.

The spectral image captured by the first multispectral camera mounted on the satellite 2 is transmitted to the information processing device 4. The operator of the information processing device 4 compares multiple (preferably three or more wavelength bands) spectral images captured by the first multispectral camera mounted on the satellite 2 of the same area, identifies a shading pattern indicating that the crop is in a specific state from the shading pattern in each spectral image, and associates this shading pattern with information indicating the specific state. Therefore, it is assumed that the operator of the information processing device 4 understands the growth state of the crops on the ground surface in the same area described above, and has previously understood what specific state the crops in which area are in based on this growth state. The above-mentioned association is stored in the memory of the information processing device 4 as a library. Preferably, this library is realized as teaching data in artificial intelligence technology. More preferably, when creating the library, the angle of the sun with respect to the ground surface is obtained or calculated from the time when the ground surface is imaged by the satellite 2, and the library is created by associating this sun angle as well. However, as already explained, the dependence on the sun angle is not essential, even at the beginning of operation of System 1.

Next, the drone 3 is flown in the observation area where it is to be determined whether the crops are in a specific state based on a predetermined flight plan, and images are taken with the second multispectral camera mounted on the drone 3. As described above, the imaging range of the second multispectral camera mounted on the drone 3 is relatively narrow, so it is preferable to fly multiple drones 3 at the same time and capture images in parallel with the second multispectral camera.

The flight plan is preferably created in advance by the operator of the information processing device 4. In this case, it is preferable for the operator to grasp the topography of the area to be observed in advance and create a flight plan that focuses the drone 3 on flying in areas where crops are expected to be in a specific state. In this case, it is also possible for the information processing device 4 to recommend areas where crops are expected to be in a specific state. Areas where crops are expected to be in a specific state include, for example, areas along rivers when crops are affected by humidity, areas where fog is expected to occur for long periods of time, and areas within a specific altitude range when crops are expected to be in a specific state within this altitude range.

In the system 1 of this embodiment, multiple wavelength bands that can clearly determine whether or not a specific state is present are selected in advance, and multiple cameras (preferably three or more, more preferably four) that can capture the reflected light of each of these wavelength bands are prepared, and a multispectral camera is formed from these multiple cameras.

The spectral image obtained by capturing an image of the observation target area using such a second multispectral camera is input to the information processing device 4, and compared with the library to identify the area where the crop is in a specific state. In FIG. 1, the hatched area is identified as the area where the crop is in a specific state. In this case, it is preferable that the information processing device 4 connects the individual spectral images captured by the second multispectral camera mounted on the drone 3 to generate a single spectral image, and uses this spectral image to identify the area where the crop is in a specific state. This is because the drone 3 flies at a lower altitude than the satellite 2, and when the observation target area is wide, it is difficult to identify the entire observation target area with a single spectral image. The meaning of connecting the spectral images is similarly applicable to the spectral images obtained from the first multispectral camera mounted on the satellite 2, so as already explained, it is preferable to appropriately connect the spectral images obtained from the first multispectral camera. Preferably, when identifying an area where the crop is in a particular state, the angle of the sun in the area at the time of capturing the spectral image is acquired and calculated, and the area where the crop is in a particular state is identified taking this angle of the sun into consideration.

<Basic configuration of System 1>
The basic configuration of a system 1 to which the method according to the embodiment is applied will be described with reference to FIG.

The system 1 of this embodiment has an information processing device 10, a drone 20, and a satellite 30, which are connected via a network N. The hardware configuration of the information processing device 10 is shown in FIG. 11, the hardware configuration of the drone 20 in FIG. 5, and the hardware configuration of the satellite 30 in FIG. 8. The drone 20 and the satellite 30 are equipped with an information processing device.

The information processing device is composed of a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration are described below.

The network N is composed of the Internet, LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of a wired connection, the network also includes those directly connected by a USB (Universal Serial Bus) cable, etc.

The information processing device 10 accepts the spectral images captured by the satellite 30 and creates a library based on the spectral images. The information processing device 10 then accepts the spectral images captured by the drone 20 and identifies areas in the observation area where crops are in a particular state based on the spectral images and the library.

The configuration and operation of each device are explained below.

<Hardware configuration of drone 20>
The drone 20 of this embodiment is configured to be capable of stationary flight, and is equipped with a spectral camera control system 23 having a spectral camera control device 21 and a spectral camera 22 controlled by the spectral camera control device 21, as shown in Figures 3 and 4. Each component will be described in detail below.

The drone 20 is an aircraft with a function of flying stationary in the air, a so-called hovering function, and in this embodiment, it is a multicopter type drone with multiple rotors. The drone 20 in this embodiment also has a function of flying autonomously along a pre-specified flight path and a function of flying by remote control from a communication device or the like. Furthermore, although not shown in Figures 3 and 4, the drone 20 has a GPS (Global Positioning System) receiver for detecting the position (latitude and longitude) and altitude of the drone during flight, and an attitude sensor for detecting the attitude of the drone during flight.

As shown in FIG. 5, the drone 20 is equipped with a spectral camera control system 23 having a spectral camera control device 21 and a spectral camera 22, which is a second multispectral camera controlled by the spectral camera control device 21.

As shown in FIG. 5, the spectral camera control system 23 mainly includes a spectral camera 22, an attitude position detector 24 that detects the attitude and position information of the spectral camera 22, a spectral camera control device 21 that controls the spectral camera 22, and a battery 25 that supplies power to each of these devices.

As shown in FIG. 5, the spectral camera 22 is mounted on the drone 20 and faces vertically downward so that the ground surface is the subject of imaging while the drone 20 is in stationary flight.

The spectral camera 22 mounted on the drone 20 of this embodiment has an image sensor capable of detecting reflected light in wavelength bands corresponding to each of the combinations of multiple wavelength bands that can detect that the crop is in a specific state as a result of library generation described below. In detail, the spectral camera 22 has an image sensor and multiple filters that can pass only light in multiple wavelength bands, and when the spectral camera 22 captures an image of the ground surface, the multiple filters are appropriately switched (replaced) to allow only reflected light in each wavelength band to reach the image sensor, and the filters are switched continuously to obtain an imaging result based on reflected light in a wavelength band corresponding to each of the combinations of multiple wavelength bands.

In this case, before the drone 20 captures an image of the area to be observed, it is determined in advance which specific state the crops are to be detected in, and the spectral camera 22 consisting of a combination of filters determined based on the determination result is mounted on the drone 20. However, the spectral camera 22 may be configured by mounting multiple cameras capable of capturing images of only light of a single wavelength band. In this way, the spectral camera 22 generates a spectral image 2104 based on reflected light of a specific predetermined wavelength band, and is configured not to switch wavelength bands while the drone 20 is capturing an image of the area to be observed. Note that the above description is not intended to exclude the spectral camera 22 being configured to be able to arbitrarily switch between multiple wavelength bands, similar to the spectral camera 32 mounted on the satellite 30.

The image sensor captures a spectral image using a snapshot method. In this embodiment, the image sensor 222 is a two-dimensional image sensor that can capture images within the field of view at the same time, such as a CMOS image sensor or a CCD image sensor. The image sensor 222 also performs imaging based on an imaging command signal transmitted from the spectral camera control device 21.

The attitude position detector 24 is a device that detects the attitude and position state of the spectral camera 22. In this embodiment, the attitude position detector 24 has a GPS receiver 240 that detects the position information and altitude information of the spectral camera 22, and an attitude sensor 241 that detects the attitude information of the spectral camera 22.

The GPS receiver 240 acquires current position information and altitude information by capturing the positions of multiple artificial satellites. In this embodiment, the GPS receiver 240 acquires longitude information and latitude information as position information, and altitude information as altitude information. Note that the position information and altitude information are not limited to those acquired by the GPS receiver 240, and may be acquired by other methods. For example, a reference point may be set and distance and altitude information from the reference point may be acquired by a distance measuring device using laser light or acoustic reflection.

The attitude sensor 241 detects attitude information of the tilt angle, angular velocity, and acceleration of the spectral camera 22. Although not shown, the attitude sensor 241 in this embodiment is composed of a gyro sensor that utilizes gyro characteristics and an acceleration sensor, and is configured to obtain the tilt angle, angular velocity, and acceleration in three axial directions as attitude information.

In this embodiment, the position information and attitude information are acquired from the attitude position detector 24 provided as the spectral camera control system 23, but this configuration is not limited to this. For example, the position information and attitude information may be acquired from a GPS receiver and attitude sensor that the drone 20 already has.

<Functional configuration of the spectrum camera control device 21>
6 shows the functional configuration realized by the spectral camera control device 21 of the drone 20. The spectral camera control device 21 includes a storage unit 210, a control unit 211, and a communication unit 212. The communication unit 212 is configured by a communication IF (not shown) of the spectral camera control device 21, the storage unit 210 is configured by a main storage device and an auxiliary storage device (not shown) of the spectral camera control device 21, and the control unit 211 is mainly configured by a processor (not shown) of the spectral camera control device 21.

The communication unit 212 communicates with the information processing device 10 etc. via the network N (not shown).

<Configuration of the storage unit 210 of the spectral camera control device 21>
The memory unit 210 of the spectroscopic camera control device 21 has a flight plan DB (DataBase) 2101 , a second imaging condition DB 2102 , an image DB 2103 , and a spectroscopic image 2104 .

Of these flight plan DB 2101 etc., all except for the spectral image 2104 are databases. The database referred to here refers to a relational database, which is used to manage and correlate data sets called tables in a tabular format that are structurally defined by rows and columns. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.

Typically, each table has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 211 can cause the processor to add, delete, and update records in specific tables stored in the storage unit 210 according to various programs.

The flight plan DB 2101 is a database in which a flight plan for flying the drone 20 along a predetermined route when flying the drone 20 in the air is stored, the second imaging condition DB 2102 is a database related to conditions when imaging an observation target area by the spectral camera 22 mounted on the drone 20, and the image DB 2103 is a database for managing a spectral image 2104 obtained as a result of imaging by the spectral camera 22 according to the second imaging condition DB 2102, and the spectral image 2104 is a spectral image obtained as a result of imaging by the spectral camera 22 according to the second imaging condition DB 2102. Details of the flight plan DB 2101, the second imaging condition DB 2102, and the image DB 2103 will be described later.
<Configuration of the control unit 211 of the spectral camera control device 21>

The control unit 211 of the spectral camera control device 21 includes a reception control unit 2110, a transmission control unit 2111, a flight control unit 2112, an attitude position information acquisition unit 2113, and an imaging control unit 2114. The control unit 211 executes an application program 2100 stored in the memory unit 210 to realize the functional units such as the reception control unit 2110.

The reception control unit 2110 controls the process in which the spectral camera control device 21 receives signals from external devices according to a communication protocol.

The transmission control unit 2111 controls the process in which the spectral camera control device 21 transmits signals to external devices according to a communication protocol.

The flight control unit 2112 controls the flight of the drone 20 toward the target point specified by the flight plan DB 2101 based on the geographical information of the target point specified by the flight plan DB 2101 and the attitude position information acquired by the attitude position information acquisition unit 2113.

The attitude position information acquisition unit 2113 acquires the position information, altitude information, and attitude information of the drone 20 detected by the attitude position detector 24, and provides the acquired results to the flight control unit 2112 and the imaging control unit 2114.

The imaging control unit 2114 refers to the second imaging condition DB 2102, captures an image of the observation target area using the spectral camera 22 at the imaging position specified by the second imaging condition DB 2102, stores the captured spectral image 2104 in the memory unit 210, and stores the conditions under which the spectral image 2104 was captured in the image DB 2103.

Figure 7 conceptually illustrates the state in which the drone 20 of this embodiment captures an image of an observation target area to obtain a spectral image. The area captured by the spectral camera 22 at one time is, for example, a rectangular (square) area with each side measuring approximately 150 m, and the altitude of the drone 20 at this time is approximately 150 m.

<Hardware configuration of satellite 30>
The satellite 30 of this embodiment revolves around the earth at a substantially constant speed on an orbit above the earth. As shown in FIG. 8, the satellite 30 is equipped with a spectral camera control system 33 having a spectral camera control device 31 and a spectral camera 32, which is a first multispectral camera controlled by the spectral camera control device 31. The configurations of the spectral camera control device 31, the spectral camera 32, and the spectral camera control system 33 are similar to those of the spectral camera control device 21, the spectral camera 22, and the spectral camera control system 23 of the drone 20. Therefore, the description of the substantially identical configurations will be omitted, and the description will be focused on the main differences.

The spectral camera control system 33 mainly comprises a spectral camera 32 equipped with a liquid crystal tunable filter 34, an attitude position detector 35 that detects information on the attitude and position of the spectral camera 32, a liquid crystal tunable filter control circuit 36 that controls the liquid crystal tunable filter 34 of the spectral camera 32, a spectral camera control device 31 that controls the spectral camera 32, and a battery 37 that supplies power to each of these devices.

The spectral camera 32 is used to capture spectral images using a snapshot method, and mainly comprises a lens group 320, a depolarizer 321 for converting polarized light to unpolarized light, a liquid crystal wavelength-variable filter 34 that can select any transmission wavelength, and an image sensor 322 for capturing two-dimensional spectral images.

The lens group 320 utilizes the refraction of light to transmit light from the image subject through the liquid crystal wavelength tunable filter 34 and to focus the transmitted light on the image sensor 322. In this embodiment, the lens group 320 is composed of an entrance lens 320a that focuses the light from the image subject and causes it to enter the liquid crystal wavelength tunable filter 34, and a focusing lens 320b that focuses only the light of the transmitted wavelength after passing through the liquid crystal wavelength tunable filter 34 on the image sensor 322. The type and number of lenses are not particularly limited and may be selected appropriately depending on the performance of the spectral camera 32, etc.

The depolarizer 321 is used to eliminate polarization and make the light non-polarized. In this embodiment, the depolarizer 321 is provided on the light-entering side of the liquid crystal tunable filter 34, and is designed to eliminate the polarization of the light before it passes through the liquid crystal tunable filter 34, thereby reducing the polarization characteristics.

The liquid crystal tunable filter 34 is an optical filter that can select any transmission wavelength from within a predetermined wavelength range. Although not shown, the liquid crystal tunable filter 34 is configured by stacking multiple plate-shaped liquid crystal elements and plate-shaped polarizing elements alternately. The orientation state of each liquid crystal element is independently controlled by the applied voltage supplied from the liquid crystal tunable filter control circuit 36. Therefore, the liquid crystal tunable filter 34 is capable of transmitting light of any wavelength by combining the orientation state of the liquid crystal elements with the polarizing elements.

In this embodiment, the transmission wavelength width of the liquid crystal wavelength tunable filter 34 is approximately 20 nm or less, the transmission center wavelength can be set in 1 nm increments, and the wavelength switching time is approximately 10 ms to several hundreds of ms.

The image sensor 322 captures a spectral image using a snapshot method. In this embodiment, the image sensor 322 is a two-dimensional image sensor that can capture images within the field of view at the same time, such as a CMOS image sensor or a CCD image sensor. The image sensor 322 also performs imaging based on an imaging command signal transmitted from the spectral camera control device 31.

The liquid crystal tunable filter control circuit 36 controls the liquid crystal tunable filter 34. In this embodiment, when the liquid crystal tunable filter control circuit 36 receives a wavelength-specific signal transmitted from the spectral camera control device 31, it supplies an applied voltage corresponding to the wavelength-specific signal to the liquid crystal element of the liquid crystal tunable filter 34. The wavelength-specific signal also includes information on the transmission wavelength to be transmitted by the liquid crystal tunable filter 34, and the liquid crystal tunable filter control circuit 36 determines which liquid crystal element to supply the applied voltage to based on the information on the transmission wavelength, and supplies the applied voltage to the identified liquid crystal element.

In this embodiment, the liquid crystal wavelength tunable filter control circuit 36 is configured independently from other components such as the spectral camera control device 31, but is not limited to this and may be provided in, for example, the spectral camera control device 31 or the spectral camera 32.

<Functional configuration of the spectrum camera control device 31>
9 shows the functional configuration realized by the spectral camera control device 31 of the satellite 30. The spectral camera control device 31 includes a storage unit 310, a control unit 311, and a communication unit 312. The communication unit 312 is configured by a communication IF (not shown) of the spectral camera control device 31, the storage unit 310 is configured by a main storage device and an auxiliary storage device (not shown) of the spectral camera control device 31, and the control unit 311 is mainly configured by a processor (not shown) of the spectral camera control device 31.

The communication unit 312 communicates with the information processing device 10 etc. via the network N (not shown).

<Configuration of the storage unit 310 of the spectroscopic camera control device 31>
The storage unit 310 of the spectroscopic camera control device 31 includes a first imaging condition DB (DataBase) 3101 , an image DB 3102 , and a spectroscopic image 3103 .

Of these first imaging condition DB 3101 etc., all except for the spectral image 3103 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tables in a tabular format structurally defined by rows and columns in association with each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.

Typically, each table has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 311 can cause the processor to add, delete, or update records in a specific table stored in the storage unit 310 according to various programs.

The first imaging condition DB 3101 is a database relating to conditions for imaging the earth's surface by the spectral camera 32 mounted on the satellite 30, the image DB 3102 is a database for managing spectral images obtained as a result of imaging by the spectral camera 32 according to the first imaging condition DB 3101, and the spectral images 3103 are spectral images obtained as a result of imaging by the spectral camera 32 according to the first imaging condition DB 3101. Details of the first imaging condition DB 3101 and the image DB 3102 will be described later.
<Configuration of the control unit 311 of the spectral camera control device 31>

The control unit 311 of the spectral camera control device 31 includes a reception control unit 3110, a transmission control unit 3111, an image capture determination unit 3112, an attitude position information acquisition unit 3113, an image capture control unit 3114, and a wavelength setting unit 3115. The control unit 311 executes an application program 3100 stored in the memory unit 310 to realize the functional units such as the reception control unit 3110.

The reception control unit 3110 controls the process in which the spectral camera control device 31 receives signals from external devices according to a communication protocol.

The transmission control unit 3111 controls the process in which the spectral camera control device 31 transmits signals to external devices according to a communication protocol.

The imaging determination unit 3112 determines whether or not to image the ground surface using the spectral camera 32 mounted on the satellite 30 based on the imaging start time and imaging end time stored in the first imaging condition DB 3101, and sends the determination result to the imaging control unit 3114.

The attitude position information acquisition unit 3113 acquires the position information, altitude information, and attitude information of the satellite 30 detected by the attitude position detector 35, and provides the acquired results to the imaging control unit 3114.

The imaging control unit 3114 receives the judgment result from the imaging judgment unit 3112, and if the judgment result is to perform imaging, it images the ground surface using the spectral camera 32, stores the captured spectral image 3103 in the memory unit 310, and stores the conditions under which the spectral image 3103 was captured in the image DB 3102.

Figure 10 conceptually illustrates how the satellite 30 of this embodiment captures an image of an observation target area to obtain a spectral image. The area captured by the spectral camera 32 at one time is, for example, a rectangular (square) area with one side measuring approximately 2 to 10 km, and the altitude of the satellite 30 at this time is approximately 500 km.

<Hardware configuration of information processing device 10>
11 is a block diagram showing a basic hardware configuration of an information processing device 10. The information processing device 10 has at least a processor 101, a main memory device 102, an auxiliary memory device 103, a communication IF (Interface) 104, an input IF 105, and an output IF 106. These are electrically connected to each other by a communication bus 107. In addition, an input device 110 is connected to the information processing device 10 via the input IF 105, and an output device 111 is connected to the information processing device 10 via the output IF 106.

The processor 101 is hardware for executing a set of instructions written in a program. The processor 101 is composed of an arithmetic unit, registers, peripheral circuits, etc.

The main memory device 102 is used to temporarily store programs and data processed by the programs. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

The auxiliary storage device 103 is a storage device for saving data and programs. For example, it is a flash memory, a solid state drive (SSD), a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc.

The communication IF 104 is an interface for inputting and outputting signals for communicating with other computers over a network using wired or wireless communication standards.

The input IF 105 functions as an interface with the input device 110 for receiving input operations from an operator of the information processing device 10. The output IF 106 functions as an interface with the output device 111 for presenting information to the operator. The input device 110 is an input device (e.g., a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving input operations from the operator. The output device 111 is an output device (a display, a speaker, etc.) for presenting information to the operator.

In addition, the information processing device 10 can be virtually realized by distributing all or part of each hardware configuration across multiple computers and connecting them to each other via a network. In this way, the information processing device 10 is a concept that includes not only a computer housed in a single housing or case, but also a virtualized computer system.

<Functional configuration of information processing device 10>
12 shows a functional configuration realized by the hardware configuration of the information processing device 10. The information processing device 10 includes a storage unit 120, a control unit 130, and a communication unit 140. The communication unit 140 is configured by a communication IF 104, the storage unit 120 is configured by a main storage unit 102 and an auxiliary storage unit 103 of the information processing device 10, and the control unit 130 is mainly configured by a processor 101 of the information processing device 10.

The communication unit 140 communicates with the drone 20 etc. via the network N.

<Configuration of storage unit 120 of information processing device 10>
The memory unit 120 of the information processing device 10 has a flight plan DB (DataBase) 122, an image DB 123, a wide area image DB 124, teacher data 125, a learning model 126, a spectral image 127, and a wide area spectral image 128.

Of these flight plan DB122 etc., flight plan DB122, image DB123 and wide area image DB124 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.

Typically, each table has a column that serves as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 130 can cause the processor 101 to add, delete, or update records in a specific table stored in the storage unit 120 according to various programs.

The flight plan DB 122 is similar to the flight plan DB 2101 stored in the storage unit 210 of the drone 20. The image DB 123 is similar to the image DB 2103 stored in the storage unit 210 of the drone 20 and the image DB 3102 stored in the storage unit 310 of the satellite 30. In the information processing device 10 of this embodiment, the image DB 123, which is a combination of the image DB 2103 and the image DB 3102, is stored in the storage unit 120. The wide area image DB 124 is a database for managing the wide area spectral image 128 generated by the image synthesis unit 136 of the control unit 130 based on the spectral image 127.
<Configuration of control unit 130 of information processing device 10>

The control unit 130 of the information processing device 10 includes a reception control unit 131, a transmission control unit 132, a learning model generation unit 133, a flight plan creation unit 134, an aircraft control unit 135, an image synthesis unit 136, an area identification unit 137, and a type identification unit 138. The control unit 130 executes an application program 121 stored in the memory unit 120 to realize these functional units such as the reception control unit 131.

The reception control unit 131 controls the process in which the information processing device 10 receives signals from external devices according to a communication protocol.

The transmission control unit 132 controls the process in which the information processing device 10 transmits signals to external devices according to a communication protocol.

The learning model generation unit 133 generates teacher data 125 based on the spectral image 127 captured by the spectral camera 32 mounted on the satellite 30 and the image DB 123 sent from the satellite 30, as well as the spectral camera 22 mounted on the drone 20 and the image DB 123 sent from the drone 20, and generates a learning model 126 based on this teacher data 125.

The following describes in detail the operation of the learning model generation unit 133 when using the spectral image 127 captured by the spectral camera 32 mounted on the satellite 30. Note that the operation is similar when using the spectral image 127 captured by the spectral camera 22 mounted on the drone 20.

The spectral camera 32 mounted on the satellite 30 captures an image of the same earth surface area for each wavelength band into which a wide range of wavelength bands are subdivided, and obtains a large number of spectral images 3103 (spectral images 127) of different wavelength bands for the same area. The learning model generation unit 133 presents a plurality of spectral images 127 of different wavelength bands for the same area to the operator of the information processing device 10, and allows the operator to identify a spectral image 127 of a wavelength band that can clearly distinguish an area in this area where crops are in a specific state. Next, the learning model generation unit 133 allows the operator to identify a specific state of the crop that can be identified by this wavelength band for the spectral image 127 of the identified wavelength band, and associates the spectral image 127 of the identified wavelength band with the specific state of the identified crop. Then, the learning model generation unit 133 stores the identified wavelength band, the spectral image 127 of the identified wavelength band, the specific state of the identified crop, and the associations between them in the storage unit 120 as teacher data 125.

For example, as shown in FIG. 23, for an original image (in this embodiment, multiple spectral images 127 can be obtained for the same area, but only one image is shown for simplicity of explanation) captured on the same ground surface, multiple wavelength bands (three types of wavelengths α, β, and γ are shown in FIG. 23, but the number of wavelength bands is not limited) of spectral images 127 can be obtained. Among these spectral images 127, a shading distribution different from that of the other wavelengths α and γ may occur for a certain wavelength (wavelength β in FIG. 23). Since the spectral image 127 is for a specific wavelength band, it is assumed that it will be displayed in a black and white shading pattern (i.e., grayscale display) when displayed on a display or the like. The operator of the information processing device 10 knows in advance what specific state the crops are in for the same ground surface area (for example, by a field survey, etc.), and inputs the association between the shading distribution and the specific state of the crops by looking at the unique shading distribution (shown by hatching in FIG. 23) for wavelength β that is not seen for the other wavelengths α and γ.

At this time, the learning model generation unit 133 refers to the image DB 123 and calculates the angle between the earth's surface and the sun when the spectral image of the specified wavelength band was captured. This is because the spectrum of reflected light from the crop differs depending on the angle of the sun. The learning model generation unit 133 then stores the sun angle in the storage unit 120 as teacher data 125. Alternatively, the learning model generation unit 133 calculates a correction formula (or correction value) for correcting the spectral image to a spectral image of a spectrum captured at a sun angle at a specified time based on the calculated sun angle, generates teacher data 125 using the spectral image corrected by this correction formula or the like, and stores it in the storage unit 120. At this time, the correction formula itself may be stored in the learning model 126 described later, or the correction formula may be stored separately in the storage unit 120. However, correction based on the sun angle is not essential.

Next, the learning model generation unit 133 generates a learning model 126 in artificial intelligence technology based on the teacher data 125 generated by the above-mentioned procedure. Since known methods for generating the learning model 126 and types of the learning model 126 can be suitably applied, further explanation will be omitted here.

The learning model generation unit 133 then repeats the above-mentioned procedure for generating the teacher data 125 to update the teacher data 125, and re-learns the learning model 126 after updating the teacher data 125 or at predetermined time intervals. This improves the accuracy with which the type identification unit 138, described later, identifies that a crop is in a specific state.

The above-mentioned updating of the teacher data 125 by the learning model generation unit 133 and the re-learning of the learning model 126 are preferable operations, particularly when the spectral image 127 at a limited solar angle is acquired at the beginning of the operation of the system 1, and the teacher data 125 and the learning model 126 are generated based on this spectral image 127. In other words, the teacher data 125 and the learning model 126 may be generated based on the spectral image 127 at a limited solar angle at the beginning, and then the spectral images obtained from the spectral camera 32 mounted on the satellite 30 and/or the spectral images obtained from the spectral camera 22 mounted on the drone 20 may be acquired at various solar angles, teacher data 125 including the influence of the solar angle may be generated, and the learning model 126 may be re-learned based on this teacher data 125. This allows the start of operation of the system 1 to be accelerated, and the accuracy of the identification result by the type identification unit 138 to be progressively improved. In addition, the identification result by the type identification unit 138, which will be described later, may be compared with the actual identified state of the crop, and the actual identified state of the crop may be fed back to create/modify the teacher data 125, and re-learn the learning model 126. This may also progressively improve the accuracy of the identification result by the type identification unit 138.

At this time, it is preferable that the learning model generation unit 133 acquires topographical data of the ground surface captured by the spectral camera 32 of the satellite 30 and/or the spectral camera 22 of the drone 20, identifies an area where the crop is estimated to be in a specific state based on the topographical data, extracts a spectral image 127 obtained by capturing the area with the spectral cameras 22, 32, and presents it to the operator. Whether the crop is in a specific state often depends on the topography. As an example, it is known that if the humidity of the place where the crop is grown is high, the crop is more likely to become ill. Therefore, the learning model generation unit 133 acquires topographical data of the ground surface, estimates the environment (sunlight, humidity, wind direction, etc.) of the place where the crop is grown based on the topographical data, and presents the spectral image 127 obtained by capturing an image of the place where the environment is specific to the operator. The topographical data may be stored in advance in the storage unit 120 of the information processing device 10, or may be acquired from an external service.

In addition, without using the learning model generation unit 133, the teacher data 125, and the learning model 126, it is also possible to determine whether or not the crop is in a specific state based only on the identified wavelength band, the spectral image 127 of the identified wavelength band, the specific state of the identified crop, the angle of the sun when the spectral image 127 was captured, and the correlation between them.

The flight plan creation unit 134 creates a flight plan for flying the drone 20 over the area to be observed and capturing images of the area to be observed using the spectral camera 22 mounted on the drone 20, and stores the flight plan in the storage unit 120 as a flight plan DB 122. The flight plan creation unit 134 then transmits the created flight plan DB 122 to the drone 20. Note that when the flight of the drone 20 is controlled (the drone 20 is piloted) by the aircraft control unit 135 described below, the flight plan creation unit 134 and the flight plan DB 122 do not need to be provided.

In this case, it is preferable that the flight plan creation unit 134 creates a flight plan in which multiple drones 20 fly simultaneously over the area to be observed and the area to be observed is imaged in parallel by the spectral cameras 22 mounted on the multiple drones 20.

Furthermore, it is preferable that the flight plan creation unit 134 acquires topographical data of the area to be observed and creates a flight plan based on this topographical data. As described above, whether a crop is in a specific state often depends on the topography. As an example, it is known that if the humidity of the area where the crop is grown is high, the crop is more susceptible to disease. Therefore, the flight plan creation unit 134 acquires topographical data of the area to be observed, estimates the environment (sunlight, humidity, wind direction, etc.) of the crop growing area based on this topographical data, flies the drone 20 focusing on the area where this environment is specific, and acquires a spectral image by the spectral camera 22. The topographical data may be stored in advance in the memory unit 120 of the information processing device 10, or may be acquired from an external service.

The flight plan creation unit 134 may create a flight plan by having the operator of the information processing device 10 individually specify the flight target position of the drone 20, or the operator may identify only the area to be observed, and the flight plan creation unit 134 may generate the flight target position based on the flight speed of the drone 20 and the speed at which the spectral image is acquired by the spectral camera 22.

The air vehicle control unit 135 controls the flight of the drone 20 based on the flight plan DB 122. However, if the drone 20 is capable of flying autonomously based on the flight plan DB 2101 stored in its own memory unit 210, the air vehicle control unit 135 does not need to be provided. Alternatively, if the drone 20 is controlled by the information processing device 10, the air vehicle control unit 135 controls the drone 20 based on control signals input by an operator via a remote control, which is an example of an input device 110 provided in the information processing device 10.

The image synthesis unit 136 stitches together the spectral images 2104 and 127 captured by the spectral camera 22 of the drone 20 to generate a wide-spectrum image 128 that preferably captures the entire observation area. This is because the imaging range of the spectral camera 22 of the drone 20 is often narrower than the observation area in which it is necessary to determine whether the crop is in a specific state, and therefore it is preferable to generate a wide-spectrum image 128 that covers a wide range in order to collectively identify the specific state of the crop. The image synthesis unit 136 then stores the generated wide-spectrum image 128 in the storage unit 120 and updates the wide-spectrum image DB 124. The image synthesis unit 136 also stitches together the spectral images 3103 and 127 captured by the spectral camera 32 of the satellite 30 to generate a wide-spectrum image 128 that preferably captures the ground surface over a wide range.

In this case, the image synthesis unit 136 preferably generates the broad spectrum image 128 based on a known method for generating an orthoimage. An orthoimage is an image obtained by converting an aerial photograph into an image that is displayed at the correct size and position, without tilt, as if viewed from directly above, like a map, by eliminating positional misalignment of the image on the photograph (hereinafter referred to as "orthotransformation"). To orthotransform an aerial photograph, it is necessary to match the position on the aerial photograph with the horizontal position on the ground. This orthotransformation is performed using a digital elevation model (elevation data) that represents the three-dimensional shape of the earth's surface. Since the method for generating an orthoimage is known, further explanation will be omitted here.

Figure 22 is a diagram showing an example of image synthesis of spectral images 2104, 127 by the image synthesis unit 136. Spectral images 127 obtained by capturing an image of an area 2200 (a square in the figure, but not limited to this) by one drone 20 are arranged horizontally in the figure with a predetermined overlapping area, and these spectral images 127 are synthesized to generate a wide-band spectral image 128. This operation is also performed on a spectral image 127 obtained by capturing an image of an area 2201 by another drone 20, to obtain a wide-band spectral image 128 that encompasses the area to be observed.

The area identification unit 137 refers to the broad spectrum image 128 created by the image synthesis unit 136 and receives from the operator of the information processing device 10 the identification of an area for the type identification unit 138 to identify whether or not the crop is in a specific state.

The type identification unit 138 identifies whether the crop is in a specific state for the area identified by the area identification unit 137, and if so, what state it is, based on the broadband spectrum image 128 and the learning model 126. At this time, the type identification unit 138 may accept an input of the specific state of the crop to be identified based on an instruction input from the operator of the information processing device 10, and identify whether the crop is in a specific specific state for the area identified by the area identification unit 137. At this time, the type identification unit 138 calculates the angle between the earth's surface and the sun when the spectral image 127 that is the basis of the broadband spectrum image 128 was captured, and corrects the spectrum using the learning model 126 or a solar angle correction formula stored in the memory unit 120 so that it becomes the broadband spectrum image 128 captured at a specified time.

Next, the type identification unit 138 displays the identification results on a display, which is an example of the output device 111. There are no particular limitations on the display format used by the type identification unit 138, but one example is a format in which map data of the observation target area is displayed by referencing a map database, and areas where crops are in a specific state are superimposed on this map data.

In addition, the type identification unit 138 may receive a time axis designation input from the operator of the information processing device 10, and display the spread of the area on the time axis where the crops are in a specific state. Furthermore, the type identification unit 138 may cause the display to display the fact that the crop is in a specific state, the crop, and an estimated result of the cost required to restore the specific state (for example, cutting down the crop in the specific state, spraying pesticides, etc.), and may calculate the estimated amount of damage and display it on the display.
<Data structure>
FIG. 13 is a diagram showing the data structure of the first imaging condition DB 3101 stored in the storage unit 310 of the satellite 30. As shown in FIG.

The first imaging condition DB 3101 is a table having columns for imaging start time, imaging end time, and wavelength conditions, with an imaging condition ID as a key for identifying the imaging conditions for the spectral camera 32 of the satellite 30.

"Imaging condition ID" is information for identifying the imaging conditions of the spectral camera 32 of the satellite 30. "Imaging start time" is information related to the time when imaging by the spectral camera 32 of the satellite 30 starts. "Imaging end time" is information related to the time when imaging by the spectral camera 32 of the satellite 30 ends. "Wavelength condition" is information related to the wavelength condition when imaging by the spectral camera 32 of the satellite 30 is performed. The spectral camera 32 mounted on the satellite 30 is capable of imaging in multiple wavelength bands (and therefore capable of generating spectral images in multiple wavelength bands), and in the example shown in FIG. 13, information related to the range of the wavelength band and the wavelength width that changes the wavelength band is stored as the wavelength condition.

Each column in the first imaging condition DB3101 is generated by an information processing device including the information processing device 10, and is stored in the memory unit 310 by sending it to the satellite 30.

Figure 14 is a diagram showing the data structures of image DBs 2103, 3102, and 123 stored in the memory unit 210 of the drone 20, the memory unit 310 of the satellite 30, and the memory unit 120 of the information processing device 10, respectively.

Image DBs 2103, 3102, and 123 are tables that use an image ID as a key to identify the spectral images captured by the spectral cameras 22 and 32 of the drone 20 and satellite 30, and have columns for the image file name, capture time, location information, altitude information, municipal information, and wavelength.

"Image ID" is information for identifying the spectral images captured by the spectral cameras 22, 32 of the drone 20 and the satellite 30. "Image file name" is information indicating the file names of the spectral images 2104, 3103, 127 identified by the image ID and stored in the memory unit 210 of the drone 20, the memory unit 310 of the satellite 30, and the memory unit 120 of the information processing device 10. "Image capture time" is information indicating the time when the spectral images 2104, 3103, 127 identified by the image ID were captured. "Location information" is location information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, 127 identified by the image ID were captured. "Altitude information" is altitude information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, 127 identified by the image ID were captured. "Attitude information" is attitude information of the drone 20 and satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured. "Wavelength" is information indicating the wavelength bands set in the spectral cameras 22 and 32 of the drone 20 and satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured.

Each column in the image DB 2103, 3102, 123 is generated by the spectral camera control device 21, 31 of the drone 20 and the satellite 30 when the spectral images 2104, 3103 are generated by the spectral cameras 22, 32 of the drone 20 and the satellite 30.

Figure 15 shows the data structure of the second imaging condition DB 2102 stored in the memory unit 210 of the drone 20.

The second imaging condition DB2102 is a table having columns for imaging position, imaging altitude, and imaging attitude, with an imaging condition ID as a key for identifying the imaging conditions of the spectral camera 22 of the drone 20.

The "imaging condition ID" is information for identifying the imaging conditions of the spectral camera 22 of the drone 20. The "imaging position" is position information of the drone 20. The "imaging altitude" is altitude information of the drone 20. The "imaging attitude" is attitude information of the drone 20.

Each column in the second imaging condition DB2102 is generated by an information processing device including the information processing device 10, and is stored in the memory unit 210 by sending it to the drone 20.

Figure 16 is a diagram showing the data structure of flight plan DBs 122 and 2101 stored in the memory unit 120 of the information processing device 10 and the memory unit 210 of the drone 20.

Flight plan DB122, 2101 is a table having columns for elapsed time, flight position, and flight attitude, with a flight plan ID as a key for identifying the flight plan of drone 20.

"Flight plan ID" is information for identifying the flight plan of the drone 20, more specifically, the target of the drone 20. "Elapsed time" is information indicating the elapsed time from the start of the flight of the drone 20 by which the drone 20 should reach the target of the drone 20 identified by the flight plan ID. "Flight position" is target position information that the drone 20 should reach at the target of the drone 20 identified by the flight plan ID. "Flight altitude" is target altitude information that the drone 20 should reach at the target of the drone 20 identified by the flight plan ID. "Flight attitude" is target attitude information that the drone 20 should reach at the target of the drone 20 identified by the flight plan ID.

Each column in the flight plan DB122, 2101 is generated by the flight plan creation unit 134 of the control unit 130 of the information processing device 10, and is stored in the memory unit 120 and sent to the drone 20. The spectral camera control device 21 of the drone 20 stores the sent flight plan DB122, 2101 in the memory unit.

Figure 17 is a diagram showing the data structure of the wide-area image DB 124 stored in the memory unit 120 of the information processing device 10.

The wide-area image DB 124 is a table having columns for the wide-area image file name, image capture time, area information, solar angle, position information, altitude information, attitude information, wavelength, and image ID, with a wide-area image ID as a key for identifying the wide-area spectrum image 128 stored in the memory unit 120 of the information processing device 10.

The "wide area image ID" is information for identifying the wide area spectrum image 128 stored in the storage unit 120 of the information processing device 10. The "wide area image file name" is information indicating the file name of the wide area spectrum image 128 identified by the wide area image ID and stored in the storage unit 120 of the information processing device 10.

"Image capture time" is information indicating the time when the wide-area spectrum image 128 identified by the wide-area image ID was captured. Here, the wide-area spectrum image 128 is formed by combining multiple spectral images 127, and the capture times of the individual spectral images 127 are strictly different. Therefore, when the image synthesis unit 136 synthesizes multiple spectral images 127 to generate the wide-area spectrum image 128, it uses the capture time of one of the multiple spectral images 127 that were used for synthesis as the capture time of the wide-area spectrum image 128. This operation is also performed for "sun angle," "position information," "altitude information," and "attitude information."

"Region information" is information indicating the region where the wide-area image 128, which is identified by the wide-area image ID and stored in the storage unit 120 of the information processing device 10, was captured. "Sun angle" is information indicating the angle of the sun when the wide-area image 128 identified by the wide-area image ID was captured. The value of the "sun angle" field may be blank, such as at the beginning of operation of the system 1. "Location information" is the location information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured. "Altitude information" is altitude information of the drone 20 when the wide-area image 128 identified by the wide-area image ID was captured. "Attitude information" is attitude information of the drone 20 when the wide-area image 128 identified by the wide-area image ID was captured. "Wavelength" is information indicating the wavelength band in which the wide-area image 128 identified by the wide-area image ID was captured. The "image ID" is information for identifying the multiple spectral images 127 that were used to synthesize the wide-area spectral image 128 identified by the wide-area image ID, and is the same as the image ID in the image DB 2103.

Each column in the wide-area image DB 124 is generated by the image synthesis unit 136 of the control unit 130 of the information processing device 10 and stored in the memory unit 120.

<Operation of System 1>
The processing of the system 1 will be described below with reference to the flowcharts of FIGS.

Figure 18 is a flowchart showing the spectral image capture process performed by the spectral camera control device 31 of the satellite 30.

First, the spectral camera control device 31 refers to the first imaging condition DB 3101 and waits for the imaging start time described in the first imaging condition DB 3101 to arrive (S1800). Then, when the imaging start time arrives (YES in S1800), the spectral camera control device 31 refers to the first imaging condition DB 3101 and sets the wavelength band for imaging by the spectral camera 32 (S1801), causes the spectral camera 32 to image the ground surface in the set wavelength band, obtains a spectral image 3103 and stores it in the memory unit 310, and stores the imaging conditions in the image DB 3102 of the memory unit 310 (S1802).

Next, the spectral camera control device 31 refers to the first imaging condition DB 3101 and determines whether the imaging operation in S1802 has been completed for all wavelength bands (S1803). If it is determined that the imaging operation in all wavelength bands has been completed (YES in S1803), the process proceeds to S1804. If it is determined that there is a wavelength band for which imaging operation has not yet been performed (NO in S1803), the process returns to S1801, sets the next wavelength band, and performs subsequent processing.

In S1804, the spectral camera control device 31 refers to the first imaging condition DB 3101 and determines whether the imaging end time has been reached. If it is determined that the imaging end time has been reached (YES in S1804), the process proceeds to S1805, and if it is determined that the imaging end time has not yet been reached (NO in S1804), the process returns to S1800 and waits for the next imaging start time.

In S1805, the spectral camera control device 31 determines whether the imaging operation has been completed for all imaging conditions stored in the first imaging condition DB 3101. If it is determined that the imaging operation has been completed for all imaging conditions (YES in S1805), the spectral image 3103 and image DB 3102 stored in the storage unit 310 are transmitted to the information processing device 10 (S1806). If it is determined that there is an imaging condition for which the imaging operation has not yet been completed (NO in S1805), the process returns to S1800 and waits for the next imaging start time.

Next, FIG. 19 is a flowchart showing the learning model generation process performed by the learning model generation unit 133 of the control unit 130 of the information processing device 10.

First, the learning model generation unit 133 refers to the image DB 123 of the storage unit 120, and specifies/selects the location (area, region) where the spectral image 127 that is the source for generating the learning model was captured (S1900). Next, the learning model generation unit 133 specifies/selects the wavelength band where the spectral image 127 was captured, in order to identify the spectral image 127 that is the source for generating the learning model from among the spectral images 127 stored in the storage unit 120 (S1901).

Next, the learning model generation unit 133 extracts from the memory unit 120 a spectral image 127 that matches the conditions specified/selected in S1900 and S1901, and displays the extracted spectral image 127 on a display, which is an example of the output device 111 (S1902).

Furthermore, the learning model generation unit 133 refers to the image DB 123, obtains the time when the spectral image 127 extracted in S1902 was captured from the item "imaging time", and calculates the angle (sun angle) between the earth's surface and the sun when the spectral image 127 was captured from this imaging time. Next, the learning model generation unit 133 calculates a correction formula (or a correction value) for the spectrum when this spectral image 127 is assumed to have been captured at a specific time (e.g., noon), and corrects the spectrum of the spectral image 127 using this correction formula (S1903). This correction formula etc. are stored as the learning model 126.

Then, the learning model generation unit 133 receives a selection input of the area to be learned from the spectral image 127 displayed on the display from the operator of the information processing device 10, using a pointing device such as a mouse, which is an example of the input device 110 (S1904). The learning model generation unit 133 then receives a designation input of the specific state of the crop for the area designated in S1903 from the operator of the information processing device 10, using a pointing device such as a mouse, which is an example of the input device 110 (S1905).

Then, the learning model generation unit 133 receives the input specifications from S1900 to S1905 from the operator of the information processing device 10, and waits for an instruction to the effect that various inputs for generating the learning model have been completed (S1906). Upon receiving the instruction (YES in S1906), the learning model generation unit 133 generates teacher data 125 based on the input specifications from S1900 to S1905 and the image data of the target spectral image 127 (S1907). The learning model generation unit 133 then generates a learning model 126 based on the teacher data 125 generated in S1907 (S1908). The learning model generation unit 133 stores the teacher data 125 and the learning model 126 generated in S1907 and S1908 in the storage unit 120.

Next, FIG. 20 is a flowchart showing the spectral image capture process performed by the spectral camera control device 21 of the drone 20.

First, the spectral camera control device 21 flies the drone 20 so that the drone 20 reaches the target position sequentially according to the flight plan DB 2101 stored in the memory unit 210 (S2000).

Next, the spectral camera control device 21 refers to the second imaging condition DB 2102 stored in the memory unit 210, and waits for the drone 20 to reach the imaging position described in the second imaging condition DB 2102 (S2001). Then, when the drone 20 reaches the imaging position (YES in S2001), the spectral camera control device 21 causes the spectral camera 22 to capture an image of the observation target area, stores the captured spectral image 2104 in the memory unit 210, and stores the imaging conditions in the image DB 2103 of the memory unit 210 (S2002).

Then, the spectral camera control device 21 judges whether the imaging operation has been completed for all imaging positions stored in the second imaging condition DB 2102 (S2003). If it is judged that the imaging operation has been completed for all imaging conditions (YES in S2003), it transmits the spectral image 2104 and the image DB 2103 stored in the storage unit 210 to the information processing device 10 (S2004). If it is judged that there is an imaging position for which the imaging operation has not yet been completed (NO in S2003), it returns to S2001 and waits for the next imaging position to be reached.

After this, the spectral camera control device 21 ends the flight of the drone 20 (S2005).

Figure 21 is a flowchart showing the process of identifying an area where a crop is in a specific state by the area identification unit 137 and type identification unit 138 of the control unit 130 of the information processing device 10.

First, the area identification unit 137 refers to the image DB 123 of the memory unit 120 to specify and select the location (area, region) where the spectral image 127 was captured, which is used to identify the area where the crop is in a specific state (S2100).

Next, the type identification unit 138 extracts from the memory unit 120 a spectral image 127 that matches the conditions specified/selected in S2100, and displays the extracted spectral image 127 on a display, which is an example of the output device 111 (S2101).

Next, the image synthesis unit 136 generates a wide-band spectrum image 128 based on the spectral image 127 extracted in S2101 (S2102), and displays the generated wide-band spectrum image 128 on a display or the like (S2103). Note that the operation of generating the wide-band spectrum image 128 by the image synthesis unit 136 may be performed prior to the processing shown in FIG. 21.

Next, the type identification unit 138 refers to the image DB 123 to obtain the time when the spectral image 127, which is the basis of the wide-band spectrum image 128, was captured from the item "Capture time", and calculates the angle (sun angle) between the earth's surface and the sun when the spectral image 127 was captured from this capture time. Then, based on the learning model 126 or a correction formula stored in the memory unit 120, the type identification unit 138 corrects the spectrum of the spectral image 127 so that it becomes the spectrum when this spectral image 127 is captured at a specific time (e.g. noon), and corrects the spectrum of the wide-band spectrum image 128 as a whole (S2104).

Then, the type identification unit 138 uses the learning model 126 stored in the memory unit 120 to perform an estimation process (inference operation) to determine whether crops are in a specific state in the observation area where the wide-band spectrum image 128 was captured, and if so, which area it is (S2105), and displays the inference result on a display or the like (S2106).

<Screen example>
24 is a diagram showing an example of a screen displayed on a display, which is an example of the output device 111 of the information processing device 10, in the learning model generation process of the learning model generation unit 133 shown in the flowchart of FIG.

Displayed on screen 2400 is a button 2401 for specifying a spectral image 127 that is the basis for generating a learning model, and the operator of information processing device 10 presses this button using a pointing device, which is an example of input device 110, to specify spectral images 2402 to 2404. The specified spectral images 2402 to 2404 are displayed on screen 2400 for each wavelength band.

The operator of the information processing device 10 views these spectral images 2402-2404, and based on the previously acquired information on the specific state of the crops in this region and the areas where the crops are in the specific state, specifies the area 2405 where the crops are in the specific state for the spectral images 2402-2404 (spectral image 2403 of wavelength β in the example shown in FIG. 24) that best indicate that the crops are in the specific state, using a pointing device or the like, which is an example of the input device 110. Next, the operator of the information processing device 10 uses a pull-down menu 2406 to specify the specific specific state of the crops in the area 2405. Then, after specifying the area 2405 and the specific specific state of the crops using the pull-down menu 2406, the operator of the information processing device 10 clicks the OK button 2407 with a pointing device or the like to input the end of the specification operation.

Next, FIG. 25 is a diagram showing an example of a screen displayed on a display, which is an example of an output device 111 of the information processing device 10, during the process of identifying an area where a crop is in a specific state by the area identification unit 137 and the type identification unit 138, as shown in the flowchart of FIG. 21.

Screen 2500 displays a spectral image 2501, and also has an area 2502 in which information indicating the image date and time and imaged area of this spectral image 2501 is displayed. Also displayed is map data 2503 corresponding to the area in which the spectral image 2501 was captured. When the operator of the information processing device 10 uses pull-down menu 2504 to specify the specific state of the crop that the operator wishes to identify, an area 2505 that is presumed to be in the specified specific state is displayed in the spectral image 2501. This area 2505 is also the shading distribution of the spectral image 2501.

Effects of the embodiment
As described above in detail, the system 1 of this embodiment enables business operators who carry out harvesting businesses to more appropriately manage the growing conditions.

<Modification>
In addition, the above-described embodiments are described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. In addition, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

As an example, in the system 1 of the embodiment described above, the spectral camera 32 mounted on the satellite 30 captures the ground surface to obtain the spectral image 3103, and the spectral camera 22 mounted on the drone 20 captures the observation target area to obtain the spectral image 2104. However, the moving body on which the spectral camera that captures the spectral image is mounted is not limited to the satellite 30 or the drone 20, and is not limited as long as it is a moving body that can capture the ground surface from the air. Furthermore, a method in which a moving body mounted with a spectral camera moves on the ground surface, such as an operator holding a spectral camera moving on foot on the ground surface or a car mounted with a spectral camera driving on the ground surface, may be used. One example is a mode in which a multispectral camera that realizes the spectroscopic terminal device disclosed in Patent No. 6342594 using a smartphone is used.

As such, there are various combinations and variations in the methods for acquiring the spectral images 3103 for creating the library and the methods for capturing the observation target area and acquiring the spectral images 2104. Modifications of these variations are described in detail below.

The LTCF camera, which is the spectral camera 32 mounted on the satellite 30, can receive reflected light in many (for example, hundreds) wavelength bands and obtain a spectral image 3103 with just one spectral camera 32. On the other hand, the spectral camera 22 mounted on the drone 20 obtains a spectral image 2104 from the reflected light in multiple wavelength bands, limited to (multiple) wavelength bands that can detect that the crops in the observation area are in a specific state based on the library.

These differences in the configuration of the spectral cameras 22, 32 are also reflected in the weight and price of the spectral cameras 22, 32. In other words, the spectral camera 32 is more expensive and heavier than the spectral camera 22. The difference in weight between the spectral cameras 22, 32 also affects the configuration of the aircraft on which the spectral cameras 22, 32 are mounted. In other words, when mounting the spectral camera 32 on the drone 20, the configuration of the drone 20 must be enlarged, and further, the time that the drone can fly at one time may be limited to a short time.

On the other hand, if the satellite 30 is equipped with a spectral camera 32, the restrictions imposed by the weight of the spectral camera 32 are somewhat alleviated, but the frequency with which the spectral image 3103 of the observation target area can be acquired is inevitably low. In other words, the satellite 30 may pass over the observation target area only once every few months, and therefore the frequency with which the spectral image 3103 can be acquired may also be only once every few months.

For this reason, when the system 1 is first put into operation, the number of times that the spectral camera 32 acquires the spectral image 3103 of the observation target area can be limited to a certain number, and the time at which the spectral camera 32 captures the observation target area can also be fixed (for example, noon), and a library can be created using the meridian altitude at that time for that area as the solar angle. In this case, no correction of the solar angle is performed when the library is created.

Next, if the time at which the spectral camera 22 images the observation area is set to coincide with the time at which the spectral camera 32 images the observation area (12:00 noon in the above example), it is possible to detect whether the crops in the observation area are in a specific state without correcting for the solar altitude. After this, the drone 20 or the like is repeatedly flown at different times at which the spectral camera 22 images the observation area to obtain spectral images 2104 of the observation area at multiple times, and the solar angle is corrected for this spectral image 2104 to detect whether the crops in the observation area are in a specific state. As described above, the correction value for the solar angle can be stored in the teacher data 125 and the learning model 126 can be retrained.

The advantage of acquiring the spectral image 3103 for creating the library by the spectral camera 32, which is an LTCF camera, is that the spectral image 3103 of the reflected light of the above-mentioned many wavelength bands can be acquired by a few (even one) shots. By acquiring the spectral image 3103 of the reflected light of many wavelength bands, it is possible to accurately select a suitable wavelength band for detecting whether or not the crops in the observation area are in a specific state. This allows the number of wavelength bands detected by the spectral camera 22 to be reduced without lowering the detection accuracy of whether or not the crops in the observation area are in a specific state. This also leads to a reduction in the weight of the spectral camera 22, which leads to the use of a highly versatile aircraft as an aircraft such as a drone 20 equipped with the spectral camera 22, and further leads to flying the drone 20 for a long time and frequently. If the drone 20 can be flown for a long time, the cost of acquiring the spectral image 2104 can be reduced, and if the drone 20 can be flown frequently, it also leads to the detection of whether or not the crops in the observation area are in a specific state.

In addition, by aligning the acquisition time of spectral image 2104 with the acquisition time of spectral image 3103 without correcting the sun angle when creating the initial library, the effort required for starting up system 1 can be reduced, and the time and date required for introducing system 1 can be reduced. Furthermore, by acquiring spectral images 2104 and 3103 at multiple times as system 1 is operated, the accuracy of detecting whether or not the crop is in a specific state can be improved.

In addition, in order to identify the area where the crops are in a specific state, the system 1 in the above-described embodiment uses a spectral image, but the temperature distribution of the area may be used in addition to the spectral image. To this end, when acquiring the spectral image that is the basis for creating the library, the temperature of this area may be acquired by a sensor or the like, and when acquiring the spectral image of the area to be observed, the temperature of this area may also be acquired by a sensor or the like. In this case, it is preferable to provide multiple temperature measurement points so that the temperature distribution has a two-dimensional spread. In this case, the library also includes temperature distribution data for the area.

Furthermore, in the above-described embodiment of system 1, the learning model 126 was used to identify areas in the observed region where crops are in a specific state, but so-called image analysis technology may also be used to identify areas where crops are in a specific state.

Furthermore, the system 1 according to the present disclosure can also be configured as follows:

When storing information in memory to detect that the crop being observed is in a specific state for comparison with a spectral image captured by the multispectral camera, a first multispectral camera-equipped aircraft is flown to capture images of the ground surface and generate a spectral image.

Next, an aircraft equipped with a second multispectral camera different from the first multispectral camera is flown in the observation area according to the operation plan, and a spectral image is generated by having the second multispectral camera capture the crops at a wavelength for detecting that the crops are in a specific state. Preferably, the aircraft equipped with the first multispectral camera and the aircraft equipped with the second multispectral camera are different aircraft.

The first multispectral camera has a component capable of switching the wavelength to be imaged, such as a liquid crystal wavelength tunable filter 34, and can generate spectral images for multiple wavelengths. On the other hand, the second multispectral camera can generate spectral images for a specific wavelength. The specific wavelength cannot be switched while the area to be observed by the second multispectral camera is being imaged. In other words, the specific wavelength is a predetermined wavelength.

The second multispectral camera may not have a specific member for capturing a spectral image, unlike the first multispectral camera, which has a member capable of switching wavelengths. For example, the second multispectral camera may generate a spectral image for a specific wavelength without having a member capable of switching wavelengths, unlike the first multispectral camera, which has a member capable of switching wavelengths. The second aircraft carrying the second multispectral camera may have a low load weight, unlike the first aircraft carrying the first multispectral camera. As a result, the first aircraft and the second aircraft may have different weights (the second aircraft has a lighter aircraft weight than the first aircraft), and based on the difference in weight, the first aircraft and the second aircraft may have different airspaces (geographical ranges determined by latitude and longitude, and altitudes) in which they can fly. The second aircraft may be equipped with multiple (for example, about four) second multispectral cameras, and each of the second multispectral cameras may generate a spectral image for a different wavelength.

The first multispectral camera 32, which can acquire spectral images for many wavelength bands, such as the multispectral camera 32 equipped with the liquid crystal wavelength tunable filter 34, tends to be expensive and heavy. On the other hand, the second multispectral camera 22, which can generate spectral images only for a specific wavelength, can be constructed to be inexpensive and lightweight.

Since the first multispectral camera 32 generates spectral images for creating a library, it is preferable to generate spectral images based on reflected light in a large number of wavelength bands. On the other hand, the second multispectral camera 22 is used to identify areas in the observation area where crops are in a specific state, and it is preferable to take frequent images using an aircraft such as a drone 20. Therefore, the second multispectral camera 22 has a great advantage in being lighter and less expensive than the first multispectral camera 32.

The above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor of the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C/C++, perl, Shell, PHP, Java (registered trademark), etc.

Furthermore, the program code of the software that realizes the functions of the embodiment may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

<Additional Notes>
The matters described in the above embodiments will be supplemented below.

(Appendix 1)
A method for operating a computer (10), comprising: a first step (S1908) of causing a processor (101) of the computer (10) to store in a memory (120) information (126) for detecting that a crop being observed is in a specific state, for comparison with a spectral image (127) captured by a multispectral camera (22); a second step (S2002) of generating a spectral image (127) in the observed region by capturing an image with the multispectral camera (22) at a wavelength for detecting that the crop is in a specific state; a third step (S2105) of identifying a region in the observed region where a crop in the specific state is located, based on the spectral image (127) resulting from the capturing and the information (126) for detecting that the crop is in a specific state; and a fourth step (S2106) of outputting information on the identified region.
A method for executing the above.
(Appendix 2)
The method described in Appendix 1, in a second step, an aircraft (20) equipped with a multispectral camera (22) is flown over the observation area according to an operation plan, and a spectral image (127) is generated by having the multispectral camera (22) capture images at wavelengths for detecting that the crop is in a specific state.
(Appendix 3)
The method according to claim 1, in which in a first step (S1908), as information for detection, information indicating the area in which the crop to be observed is grown, a spectral image (127) obtained by photographing the area with a multispectral camera (22) and information on the wavelengths for the photographing, and information on the angle of the sun when photographing the crop in the area are associated with each other, are stored in a memory (120) to generate a trained model (126) by performing machine learning using as training data a database in which such information is associated, and in a third step (S2105), the area in which the crop is in a specific state is identified based on the spectral image (127) resulting from the photographing and the trained model (126).
(Appendix 4)
The method according to Appendix 3, in which in the first step (S1908), a trained model (126) is generated by performing machine learning using a database associated with information on the temperature of the area when the area is imaged with a multispectral camera (22) as training data (125) as information for detection, and a trained model (126) is stored in the memory (120), and in the third step, an area in which crops in a specific state are located is identified based on the temperature of the observed area when the observed area is imaged with the multispectral camera (22).
(Appendix 5)
The method described in Appendix 3, in a first step (S1908), a database that is training data (125) is further associated with information on a specific type of state, and a trained model (126) generated by the database containing the information on the specific type of state is stored in a memory (120), and in a third step (S2105), an area in which crops in a specific state are located and the type of specific state are identified based on the trained model (126).
(Appendix 6)
The method described in Appendix 2, in a second step (S2002), as an operation plan, includes flying multiple aircraft (20) over different areas, and overlaying the spectral images (127) taken by each aircraft based on the area taken to generate a spectral image (128).
(Appendix 7)
The method described in Appendix 2, in a second step (S2002), as an operation plan, specifies a wavelength of the multispectral camera (22) for each observation area according to the crop being observed, and generates a spectral image (127) by the multispectral camera (22) of the aircraft by switching the wavelength for each area in accordance with the specified wavelength.
(Appendix 8)
The method described in Appendix 2, in a second step (S2002), as an operation plan, if the terrain surrounding the crop growing location satisfies specified conditions, a spectral image (127) is generated by photographing the area that satisfies the specified conditions with the multispectral camera (22) of the aircraft.
(Appendix 9)
9. The method according to any one of appendices 1 to 8, wherein in a third step (S2105), a geographical area where the crop is in a particular state is distinguished from a geographical area where the crop is not in a particular state, and further, the spectral image (127) is overlaid on a map image based on the point where the image was taken to display the area where the crop is in a particular state.
(Appendix 10)
A method according to any one of appendices 1 to 9, in which in a third step (S2105), a designation of the photographing time is accepted from a user, and a geographical area in which crops are in a particular state is displayed based on a spectral image (127) taken at the designated time.
(Appendix 11)
A method as described in any of Appendices 1 to 10, in which in a third step (S2105), based on the result of detecting that the crop is in a specific state, information about the crop that is in the specific state and information about the costs of dealing with the crop being in the specific state are displayed.
(Appendix 12)
12. The method according to any one of appendices 1 to 11, in which in a first step (S1908), an aircraft (30) equipped with a multispectral camera (32) is flown to photograph the earth's surface, and based on the obtained spectral image (127), information indicating the area in which the crop to be observed is grown is associated with a spectral image (127) obtained by photographing the area with the multispectral camera (32) and information on wavelengths for the photographing and stored in a memory (120); further, a processor (101) of a computer (10) executes a fifth step (S1904) of accepting from a user a designation that the crop is in a specific state in association with the spectral image (127) by having a user view the stored spectral image (127); and a sixth step (S1906) of storing in the memory (120) a database (125) for detecting that the crop is in a specific state in accordance with the user's designation.
(Appendix 13)
13. The method according to claim 12, wherein the spectral images for creating the database are images of the same region using a first wavelength group consisting of a plurality of wavelengths, and the spectral images in the second step are images of the same observation target region using a second wavelength group consisting of a plurality of wavelengths, and the number of wavelengths in the first wavelength group is greater than the number of wavelengths in the second wavelength group.
(Appendix 14)
A method according to claim 12 or 13, in which in a first step (S1908), information indicating a region is associated with a spectral image (127) and information on the wavelength for capturing the image, and information on a type of a specific state is further stored in a memory (120).
(Appendix 15)
A method according to any one of appendices 12 to 14, further comprising identifying wavelengths for detecting that the crop is in a particular state using a spectral image (127) based on a specification received from a user in a fifth step (S1904).
(Appendix 16)
The method according to any one of appendices 12 to 15, wherein in the first step (S1908), information on the temperature of the area when the area is imaged by the multispectral camera (23) is further associated and stored in the memory.
(Appendix 17)
A method according to any one of appendices 12 to 16, wherein, if the topography surrounding a location where a crop is grown satisfies a predetermined condition from the information stored in the memory (120), a spectral image (127) of an area that satisfies the predetermined condition is extracted, and the extracted spectral image (127) is presented to a user in a selectable manner.
(Appendix 18)
A method according to any one of appendices 12 to 17, in response to receiving input of information about a region from a user, extracting a spectral image (127) of the region related to the input, and presenting the extracted spectral image (127) to the user in a manner that allows the user to specify it.
(Appendix 19)
A program for causing a computer (10) to execute the method according to any one of appendices 1 to 18.
(Appendix 20)
19. An information processing device for executing the method according to any one of claims 1 to 18.

1...System 2, 30...Satellite 3, 20...Drone 4, 10...Information processing device 101...Processor 102...Main memory device 103...Auxiliary memory device 120, 210, 310...Memory unit 121, 2100, 3100...Application program 122...Flight plan DB 123, 2103, 3102...Image DB 124...Wide-area image DB 125...Teacher data 126...Learning model 127, 2104, 3103...Spectral image 128...Wide-area spectral image 130, 211, 311...Control unit 133...Learning model generation unit 134...Flight plan creation unit 135...Aircraft control unit 136...Image synthesis unit 137...Area identification unit 138...Type identification unit 21, 31...Spectral camera control device 22, 32...Spectral camera N...Network

Claims (18)

1. A method for operating a computer, comprising:
A first step of storing information for detecting a growth state of a crop to be observed in a memory in advance, the information being to be compared with a spectral image captured by a multispectral camera;
a second step of generating the spectral image by capturing images of the crops in a region to be observed using the multispectral camera in a plurality of wavelength bands determined in advance for detecting the growth state of the crops;
a third step of identifying the area in which the crop in the particular growth state is located in the observation target area based on the spectral image obtained by the photographing and information for detecting the growth state of the crop;
and a fourth step of outputting information on the identified area.
In the first step, an aircraft equipped with the multispectral camera is flown to photograph the ground surface, and based on the obtained spectral image, information indicating the region in which the crop to be observed is grown is associated with the spectral image obtained by photographing the region with the multispectral camera and information on the wavelength band for photographing the spectral image , and the associated information is stored in advance in the memory;
In the second step, the aircraft is flown in the area to be observed according to an operation plan , and the spectral image is generated by photographing the crop with the multispectral camera using the wavelength band for detecting that the crop is in the specific growth state;
The first step comprises:
a receiving step of receiving, from the user, a designation that the crop is in the specific growth state in association with the spectral image by allowing the user to view the stored spectral image;
and a storage step of storing in the memory a database for detecting that the crop is in the specific growth state in response to the user's designation received in the receiving step . In the first step, the information for the detection includes:
Information indicating the region in which the crop to be observed is grown; and
The spectral image obtained by photographing the area with the multispectral camera and information on the wavelength band for photographing the spectral image;
A trained model is generated by performing machine learning using a database in which information on the angle of the sun when photographing the crops in the region is associated as training data, and the trained model is stored in the memory;
The method according to claim 1 , wherein in the third step, the area in which the crop in the particular growth state is located is identified based on the spectral image resulting from the photographing and the trained model. In the first step, the learned model is generated by performing the machine learning using a database associated with temperature information of the area when the area is imaged by the multispectral camera as training data, as the information for the detection, and is stored in the memory;
3. The method according to claim 2, further comprising: identifying the area in which the crop is in the particular growth state based on a temperature of the observed area when the observed area is imaged by the multispectral camera in the third step. In the first step, the database serving as the teacher data is further associated with information on the specific growth state type, and the trained model generated by the database including the information on the specific growth state type is stored in the memory;
The method according to claim 2 , in the third step, the area in which the crop in the particular growth state is located and the type of the particular growth state are identified based on the trained model. The method according to claim 1, wherein in the second step, as the operation plan, a plurality of the flying vehicles are flown over different areas, and the spectral images taken by each of the flying vehicles are superimposed based on the area in which the images were taken to generate the spectral image. The method according to claim 1, wherein in the second step, the operation plan specifies the wavelength band of the multispectral camera for each region to be observed according to the crop to be observed, and the wavelength band is switched for each region according to the specification of the wavelength band, to generate the spectral image by the multispectral camera of the aircraft. The method according to claim 1, wherein in the second step, if the terrain surrounding the growing area of the crop satisfies a predetermined condition as part of the operation plan, the spectral image is generated by photographing the area that satisfies the predetermined condition with the multispectral camera of the aircraft. The method according to claim 1, wherein in the fourth step, a geographical area in which the crop is in the specific growth state is distinguished from a geographical area in which the crop is not in the specific growth state, and the area in which the crop is in the specific growth state is displayed by overlaying the spectral image on a map image based on the point at which the spectral image was taken. The method according to claim 1, wherein in the fourth step, a designation of the photographing time is accepted from a user, and a geographical area in which the crop is in the specific growth state is displayed based on the spectral image taken at the designated time. The method according to claim 1, wherein in the fourth step, based on the result of detecting that the crop is in the specific growth state, information about the crop being in the specific growth state and information about the cost of dealing with the fact that the crop is in the specific growth state are displayed. the spectral images for creating the database are images of the same region for a first wavelength group consisting of a plurality of wavelength bands;
the spectral images in the second step are images of the area of the same observation target for a second wavelength group consisting of a plurality of wavelength bands;
The method of claim 1 , wherein the number of wavelength bands in the first wavelength group is greater than the number of wavelength bands in the second wavelength group. The method according to claim 1 , wherein in the first step, information indicating the region is associated with information on the spectral image and the wavelength band for capturing the spectral image, and information on the type of the specific growth condition is further stored in the memory. The method according to claim 1 , further comprising identifying a wavelength band for detecting, from the spectral image, that the crop is in the specific growth state, based on the designation received from the user in the receiving step. The method according to claim 1 , further comprising: storing in the memory in association with information on temperature of the area when the area is imaged by the multispectral camera, in the first step. 2. The method according to claim 1, further comprising: extracting the spectral image of the area that satisfies a predetermined condition from the information stored in the memory, and presenting the extracted spectral image to the user in a selectable manner when the topography surrounding the location where the crop is grown satisfies the predetermined condition. The method according to claim 1 , further comprising: extracting a spectral image of the area related to the input from the user; and presenting the extracted spectral image to the user in a selectable manner. A program for causing a computer to execute the method of claim 1. An information processing device that executes the method according to claim 1.