JP7514559B2 - Crop growth management device and method - Google Patents

Description

The present invention relates to a crop growth management device and a crop growth management method.

As a background technique of this technical field, there is JP 2007-124932 A (Patent Document 1). This publication states that "the computer 3 has a communication unit, a growth information generation unit that generates growth information based on an image, and an information providing unit that transmits the growth information generated by the growth information generation unit to a portable communication terminal device corresponding to the identification information via the communication unit and displays the growth information on the display unit of the portable communication terminal device 2. The portable communication terminal device 2 has a display unit 25, a communication unit, a filter, an image capture unit 21 that captures the reflected light of the plant to be measured through the filter to obtain an image, and a control circuit that associates the spectral image and the identification information and transmits them to the computer 3 via the communication unit, and displays the growth information and growth diagnosis information of the plant P1 to be measured received from the computer 3 on the display unit 25" (see abstract).

JP 2007-124932 A

The above-mentioned Patent Document 1 describes a system for performing growth diagnosis based on images captured by a portable communication terminal device. However, the Patent Document 1 only analyzes image information transmitted from the portable communication terminal device by a computer, and does not take into consideration the shooting conditions suitable for determining the growth state of the crop.
Therefore, the present invention provides a mechanism for acquiring images that are more suitable for determining the growth state of crops.

In order to solve the above problems, for example, the configurations described in the claims are adopted.
The present application includes multiple means for solving the above-mentioned problems, and one example is a crop growth management device having an acquisition unit that acquires image information captured by a camera, a judgment unit that judges the growth state of the crop based on the acquired image information, and an output unit that outputs information, wherein the output unit outputs information relating to the shooting conditions for judging the growth state of the crop.

According to the present invention, it is possible to provide a mechanism for acquiring images that are more suitable for determining the growth state of crops.
Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments.

1 is an example of a connection configuration diagram of an entire crop growth management system 100. 2 is an example of a hardware configuration of the mobile terminal 102. 4 is an example of another hardware configuration of the mobile terminal 102. 2 is an example of a hardware configuration of a management server 103. 5 is an example of farm field management information 500. 6 is an example of imaging condition management information 600. 7 is an example of imaging information management information 700. 8 is an example of growth state management information 800. 9 illustrates an example of a photographing information acquisition process flow 900. 1 is an example of a growth state determination process flow 1000. 11 illustrates an example of a shooting condition calculation process flow 1100. 12 is an example of a flow of displaying imaging conditions and imaging processing; 13 is an example of a shooting schedule information display screen 1300 displayed on the mobile terminal 102. 14 is an example of a shooting position list display screen 1400 displayed on the mobile terminal 102. 15 is an example of a shooting location details display screen 1500 displayed on the mobile terminal 102. 16 is an example of a screen 1600 when photographing with the mobile terminal 102.

Hereinafter, the embodiments will be described with reference to the drawings.
It is envisioned that photographs of plants and other crops will be taken using camera equipment or mobile devices such as smartphones and tablet computers, and the captured images will be analyzed to diagnose the crop growth.
FIG. 1 is an example of a connection configuration diagram of an entire crop growth management system 100.
The growth management system 100 includes a drone 101, a mobile terminal 102, and a base station 110, each of which is connected to a management server 103 via a network. Regardless of whether the network is wired or wireless, each terminal can transmit and receive information via the network.

The network may be a network that communicates according to one communication standard, or a network that combines networks of multiple communication standards. For example, the drone 101 and the mobile terminal 102 may each be network-connected via Wi-Fi (registered trademark) provided by the base station 110, or the drone 101 and the mobile terminal 102 may each be network-connected via a mobile communication network such as LTE. Alternatively, the drone 101 may be connected via Wi-Fi provided by the base station 110, and the base station 110 and the mobile terminal 102 may be connected via a mobile communication network.

The mobile terminal 102 is realized by, for example, a tablet terminal having a camera, a mobile information device such as a smartphone, or a camera device having a communication function.
An image of the crop captured by the mobile terminal 102 or processed data of the image processed by the mobile terminal is transmitted to the management server 103 via a network, and the management server 103 executes a crop growth diagnosis.
The drone 101 performs autonomous flight according to instructions from the management server 103 or by the mobile terminal 102, but can also be manually operated by the mobile terminal 102. The mobile terminal 102 is connected to a base station 110 and can communicate with the management server 103 via the base station 110 or directly.

The management server 103 is, for example, a server located on the cloud, and can perform various analyses of the condition of fields and crops by analyzing images and sensor information transmitted from cameras and various sensors mounted on the mobile terminal 102 and drone 101.
The base station 110 is a device installed in the field 120 that provides a base station function for Wi-Fi communication and the like, and also functions as an RTK-GPS base station, and is capable of providing the accurate position of the drone 101 (the base station function for Wi-Fi communication and the RTK-GPS base station may be independent devices). The base station 110 can also communicate with the management server 103 using a mobile communication network such as 3G, 4G, or LTE.

Each terminal of the crop growth management system 100 and the management server 103 may be, for example, a mobile terminal such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, wristwatch, or clothing. They may also be stationary or mobile computers, or servers located on the cloud or network. In terms of function, they may be VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, or MR (Mixed Reality) terminals. Or they may be a combination of multiple terminals. For example, a combination of one smartphone and one wearable terminal may function logically as one terminal. They may also be other information processing terminals.

Each terminal and management server 103 of the growth management system 100 includes a processor (control unit) that executes an operating system, applications, programs, etc., a main memory device such as a RAM (Random Access Memory), an auxiliary memory device such as an IC card, a hard disk drive, an SSD (Solid State Drive), a flash memory, etc., a communication control unit such as a network card, a wireless communication module, a mobile communication module, etc., an input device such as a touch panel, a keyboard, a mouse, voice input, or input by motion detection by imaging of a camera unit, and an output device such as a monitor or display. The output device may be a device or terminal that transmits information to be output to an external monitor, display, printer, device, etc.

The main memory stores various programs, applications, etc. (modules), and the processor executes these programs and applications to realize each functional element of the overall system. Each of these modules may be implemented in hardware, for example by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

In this specification, each module is described as an entity (subject) that performs processing, but in reality, the processing is carried out by a processor that processes various programs, applications, etc. (modules).
Various databases (DB) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a data set so that it can handle any data operation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The method of implementing the database is not limited, and may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.
The growth management mechanism in this embodiment can be implemented by the management server 103 in the growth management system 100 in cooperation with the mobile terminal 102, or can be implemented by the mobile terminal 102 alone. Therefore, the management server 103 may be called a growth management device, and the mobile terminal 102 may be called a growth management device.

FIG. 2 shows an example of the hardware configuration of the mobile terminal 102.
The mobile terminal 102 is, for example, a tablet, a smartphone, a head-mounted display, a camera, or the like.
The main memory device 201 stores programs and applications such as a photography information management module 211, a growth status management module 212, and a photography management module 213, and each functional element of the mobile terminal 102 is realized by the processor 203 executing these programs and applications.
The photography information management module 211 manages image information captured by the camera unit 206 and information on the fields and crops acquired by the input device 204 such as various sensors, and stores the information in the photography information management information 700. The photography information management module 211 also transmits the stored information to the management server 103.

The growth state management module 212 displays the growth state management information 800 such as crop growth diagnosis information received from the management server 103 on the output device 205 such as a monitor or panel.
In addition, in the case where the mobile terminal 102 is configured to analyze images and diagnose growth by itself, the growth status management module 212 analyzes the images, determines the growth status of the crop, and calculates growth status management information 800 such as growth diagnosis information.
The photography management module 213 displays the photography conditions described in the photography condition management information 600 transmitted from the management server 103, and takes a photographic image that matches the conditions.
The auxiliary storage device 202 stores various information such as map information 400 in which information on the map to be displayed on the screen is stored, farm field management information 500, photography condition management information 600, photography information management information 700, and growth state management information 800.

FIG. 3 shows another example of the hardware configuration of the mobile terminal 102. In FIG.
For example, the mobile terminal 102 is a camera device 20 equipped with a communication function, such as a single-lens reflex camera or a compact camera. The camera device 20 has a camera unit 21 equipped with a lens and an image capturing unit.
The camera device 20 also has a camera controller (camera control unit) 22 that performs overall control of the camera unit 21, and a camera capture unit 23 that issues instructions to the camera unit 21 to take pictures, etc.
The camera device 20 also has an image processor 24 that pre-processes images captured by the camera unit.
The camera device 20 also has a storage device 25 such as an SSD, and is capable of storing data of captured images or pre-processed images.

The camera device 20 further includes a transmission device 26, which can transmit raw data of a captured image or data of a pre-processed image to an external management server 103 or the like.
For example, it is possible to connect to a cloud device 28 such as the management server 103 by using application software 27 that synchronizes files and directories between remote locations.
Furthermore, the camera device 20 can be mounted on the drone 101, and can also be provided with a connection part for connecting to the drone 101. The camera device 20 may be mounted on the drone 101 when the drone 101 is manufactured, or may be mounted on the drone 101 later.

When the camera device 20 is mounted on the drone 101, it can be connected to the flight controller 501 and transmit data such as captured images to the flight controller 501. In addition, in cooperation with the flight controller 501, it can also capture images while the drone 101 is flying.
The camera device 20 can receive information such as the position, speed, attitude, time, flight status (mission status), and altitude above ground of the drone 101 from the flight controller 501. In addition, the camera unit can execute shooting processing in response to instructions from the flight controller 501.
The camera unit may be an ordinary visible light camera, an infrared light camera, an ultraviolet light camera, or a multispectral camera capable of acquiring light of multiple wavelengths.

FIG. 4 shows an example of the hardware configuration of the management server 103 .
The management server 103 is configured, for example, by a server placed on a cloud.
The main memory device 401 stores programs and applications such as a shooting information management module 411, a growth status management module 412, and a shooting condition management module 413, and each functional element of the management server 103 is realized by the processor 403 executing these programs and applications.
The photography information management module 411 receives image information captured by the camera unit 206 of the mobile terminal 102 and information regarding the field and crops acquired by the input device 204 such as various sensors from the mobile terminal 102, and stores and manages the information in the photography information management information 700.

The growth state management module 412 analyzes images and information related to the field and crops received from the mobile terminal 102 to determine the growth state of the crops. The growth state management module 412 stores the determined growth state as growth state management information 800 in the auxiliary storage device 402, and also sends the image and other information to the mobile terminal 102 that sent it.
The image capturing condition management module 413 calculates and manages the image capturing conditions used to determine the growth state.
The auxiliary storage device 402 stores various types of information such as map information 400, farm field management information 500, photography condition management information 600, photography information management information 700, and growth state management information 800.

Although the same information is stored in the mobile terminal 102 and the management server 103, this may be configured such that the respective pieces of information are synchronized, or the information on one side is simply copied to the other. Also, some or all of the information may be stored on the management server 103, and the mobile terminal 102 may download information from the management server 103 as necessary.

The camera device 20 may also be configured to cooperate with a device such as a smartphone to form a single mobile terminal 102. In this case, even if the camera device 20 itself does not have a communication function via a mobile phone data communication network, it is possible to transfer information such as images from the camera device 20 to the smartphone and transmit shooting information such as images to the management server 103 via the communication function of the smartphone.
Conversely, even if the camera device 20 does not have a sensor for acquiring location information, directional information, etc., the camera device 20 can utilize information acquired using the smartphone's GPS function or directional measurement function using an electronic compass, and store and transmit images in association with this information.

For example, the following process is executed.
The camera device 20 processes the captured image in the image processing unit 24, and transmits the processed image information to the smartphone via a wireless or wired network.
The smartphone displays the received processed image, and transmits the image information to the management server 103 in association with the location information acquired by the smartphone (GPS location information, shooting direction, distance from the subject, etc.).
The management server 103 analyzes and diagnoses the growth condition of the crop based on the received image information and location information, and transmits the determination result of the growth condition (growth diagnosis result) to the smartphone.
The smartphone displays the growth diagnosis results received from the management server 103.

As another cooperation method, for example, the smartphone transmits location information acquired by a GPS function to the camera device 20, and the camera device 20 transmits a processed image and location information to the management server 103. The growth state of the crop is analyzed and diagnosed based on the received image information and location information, and a determination result of the growth state (growth diagnosis result) is transmitted to the smartphone or the camera device 20.
As another method of collaboration, for example, the camera device 20 sends processed images to the management server 103, the smartphone sends location information to the management server 103, the management server 103 matches this information, and then analyzes and diagnoses the crop growth condition based on the received image information and location information, and sends the growth condition judgment result (growth diagnosis result) to the smartphone or the camera device 20.

FIG. 5 is an example of farm field management information 500.
The field management information 500 stores various information related to the field in which the crop for which the growth state is to be determined is grown, such as the field ID, the field name, the field position 511, the field perimeter coordinates 512, the field area 513, topographical information 514, soil information 515, and the cultivated crop 516. The field management information 500 may also be simply referred to as field information.
The field ID is identification information that uniquely identifies a field.
The field position 511 indicates the position coordinates of the field, and has information on the latitude and longitude of the center of the field, for example.
The field surrounding coordinates 512 indicate the coordinates around the field, for example, the position coordinates of the four corners of a rectangular field. The sample value GC007 indicates a link to information in which consecutive position coordinates are stored, separated by commas, for example.

The field area 513 is the total area of the field corresponding to the field ID.
The topographical information 514 is information on the topography of the field, such as the location of a water gate, the location of a sunny or shady area at a specific date and time, etc. This information can be calculated based on the field location 511 and the field periphery coordinates 512 in the map information 400 and the field management information 500, and information confirmed in advance by surveying or the like is also stored.
The soil information 515 indicates a link to information about the soil of the field. The information about the soil is, for example, detection information about soil components.
The cultivated crop 516 indicates information for identifying the crop cultivated in the field. In addition, information on the cultivation of the crop, such as the row direction of the crop and the general growth process, is also stored.

FIG. 6 shows an example of imaging condition management information 600.
The photography condition management information 600 stores the conditions for photographing crops or fields, and stores information such as a condition ID 611, a condition name 612, a photography position 613, a photography direction 614, a photography angle 615, a photography procedure 616, a photography distance 617, a photography location 618, a photography time 619, and photography settings 621.
The condition ID 611 is identification information that uniquely identifies a condition.
The condition name 612 stores a name for easy understanding when displaying the condition.
The photographing position 613 stores the position coordinates where photographing is performed, for example, position information, latitude, longitude, height information from the ground, etc., for a field specified by a field ID. An example of "farm007-360613.58926, 1400516.27816-80cm" written in the photographing position 613 indicates that the position is 36 degrees 06 minutes 13.58926 seconds, the longitude is 140 degrees 05 minutes 16.27816 seconds, and the height from the ground is 80 cm for a field with a field ID of farm007.
The direction of photography is stored in the photography direction 614. For example, "270" written in the photography direction 614 indicates 270 degrees clockwise from north, that is, due west.

The shooting angle 615 stores the angle at which shooting is performed. For example, the horizontal direction is set to 0, and positive angles are used upward and negative angles are used downward. For example, "-30" written in the shooting angle 615 indicates a direction 30 degrees downward from the horizontal direction.
The photographing procedure 616 stores a procedure for photographing. By displaying the photographing procedure on the mobile terminal 102, it is possible to encourage the photographer to photograph according to the procedure. Examples of the photographing procedure include photographing the sky light and then the crop, or photographing the base of the plant, the whole plant, and the tips of the plant in that order.
The shooting distance 617 indicates the distance from the crops or ground surface when shooting.
The photographing location 618 indicates the location of the subject to be photographed (photographing location). For example, the base of a stump or the tip of an ear of rice can be specified.
The shooting time 619 stores the date and time when the image was taken. An example of "0810-0820" written in the shooting time 619 indicates that the image was taken between August 10 and August 20. It is also possible to set a specific date and time instead of a period.
The shooting settings 621 store setting information for acquiring an image when shooting, such as exposure settings such as shutter speed and aperture value, number of pixels, histogram adjustment, and the like.

FIG. 7 shows an example of shooting information management information 700.
The photographing information management information 700 stores information about the photographed image and the conditions under which the image was photographed, and stores information such as a photographing information ID 711, a photographing information name 712, a photographing position 713, a photographing direction 714, a photographing angle 715, a photographing procedure 716, a photographing distance 717, a photographing location 718, a photographing date and time 719, a photographing setting 721, a photographing brightness 722, an image 723, a photographing condition 724, and a photographing condition determination 725. Here, since a person takes a photograph while holding the camera in his/her hand, it is considered that the photographing position, photographing direction, photographing angle, photographing distance, etc. may vary during photographing (during sensor exposure), so it is desirable to store the amount of variation for this information as well, and if there is further variation, it is desirable to store an average value.

The photographing information ID 711 is identification information that uniquely identifies a photographed image or the like.
The photographing information name 712 stores a name for easy identification when displaying the photographing information.
The photographing position 713 stores the position coordinates where the photograph was taken, for example, position information, latitude, longitude, height information from the ground, etc., regarding the field identified by the field ID.
The photographing direction 714 stores the direction in which the photograph was taken.
The shooting angle 715 stores the angle at which the shooting was performed.
The photographing procedure 716 stores the procedure for photographing.
The shooting distance 717 indicates the distance from the crops or ground surface when the image was taken.
The photographed location 718 indicates the location (photographed location) of the subject where the photograph was taken. For example, the base of a stump or the tip of an ear of rice can be specified.

The photographing date and time 719 stores the date and time when the photograph was taken.
The shooting settings 721 store image acquisition setting information such as exposure settings, pixel count, and histogram adjustment values at the time of shooting.
The brightness at the time of shooting 722 stores information about the brightness at the time of shooting.
Image 723 indicates a link to captured image information or image information preprocessed by the mobile terminal 102 after capture.
The photographing conditions 724 stores the photographing condition management information 600 corresponding to the photographed image, and the condition ID 611 of the photographing condition management information 600 is referenced.
The photographing condition determination 725 is information for storing whether or not the photographing conditions are satisfied at the time of photographing. Information of the photographing time determination 1630 in FIG. 16 is stored.

FIG. 8 is an example of the growth state management information 800.
The growth state management information 800 stores the growth state of the crop obtained by analyzing the photography information management information 700, and stores information such as a growth state ID 811, a photography information ID 812, and a growth state 813.
The growth state ID 811 is information for uniquely identifying a growth state.
The photographing information ID 812 indicates the original data used to determine the growth state, and the value of the photographing information ID 711 of the photographing information management information 700 is stored.
The growth state 813 stores information regarding the determination result of the analyzed growth state.

FIG. 9 shows an example of a shooting information acquisition process flow 900 .
The image capture information acquisition process flow 900 is a process in which the image capture information management module 411 of the management server 103 receives and stores image capture information including an image of a crop from the mobile terminal 102 .
The photographing information management module 411 receives photographing information including image information from the mobile terminal 102 (step 910).
The imaging information management module 411 stores the received imaging information in the imaging information management information 700 (step 920).

The shooting information received from the mobile terminal 102 includes, in addition to image information, information about the conditions when the image was taken, such as the shooting position, shooting direction, shooting angle, shooting procedure, shooting distance, shooting location, shooting date and time, shooting settings, and brightness at the time of shooting, which is stored in the shooting information management information 700. Information about these conditions is acquired by various sensors equipped in the mobile terminal 102.
Also, as mentioned above, when the mobile terminal 102 is a combination of a camera device 20 and a device such as a smartphone, information about the conditions when the image was taken is acquired by either the camera device 20 or the device such as a smartphone, and is sent to the management server 103 either together or separately.
When the photographing information management module 411 receives an image and information on the conditions under which the image was photographed separately, the photographing information management module 411 associates them and stores them in the photographing information management information 700 .

FIG. 10 shows an example of a growth state determination process flow 1000.
The growth state determination process flow 1000 is a process in which the growth state management module 412 of the management server 103 analyzes the photography information management information 700 such as images to determine the growth state of the crop.
The growth state management module 412 reads and acquires the photographing information management information 700 received from the mobile terminal 102 and stored in the auxiliary storage device 402 (step 1010). Note that the information may be directly received from the mobile terminal 102 and acquired without being stored in the auxiliary storage device 402.

The growth state management module 412 analyzes various types of photography information in the acquired photography information management information 700, and determines the growth state of the crop (step 1020).
The growth state management module 412 associates the determined growth state with the analyzed photographing information and stores them in the growth state management information 800 (step 1030). The growth state management information 800 is associated with the photographing information management information 700 by the photographing information ID 812, and the photographing information management information 700 stores photographing conditions 724 indicating the photographing condition management information 600 referred to at the time of photographing and a photographing condition judgment 725 indicating whether the photographing conditions were satisfied.
The growth state may be analyzed, for example, by judging whether the crop is growing normally based on the color and length of the leaves, stems, ears, etc. in the captured image information, or by judging whether symptoms of disease (e.g., early disease spots) are present. The growth state may also be judged by calculating the degree of growth of the crop by calculating a normalized difference vegetation index based on the ratio of visible red light and near infrared light of the leaves.

FIG. 11 shows an example of a shooting condition calculation process flow 1100.
The photographing condition calculation process flow 1100 is a process in which the photographing condition management module 413 of the management server 103 calculates the photographing conditions of an image used for determining the growth state.
The photography condition management module 413 acquires the growth state management information 800 and/or related information (step 1110).
The imaging condition management module 413 calculates the imaging conditions based on the acquired information (step 1120).
The imaging condition management module 413 stores the calculated imaging conditions in the imaging condition management information 600 (step 1130).
The imaging condition management module 413 transmits the imaging condition management information 600 to the mobile terminal 102 (step 1140).

The imaging conditions to be calculated include, for example, the position coordinates where imaging is performed, such as the latitude, longitude, and height from the ground for a specific farm field. The position coordinates and number of multiple imaging positions are calculated based on at least one of the results of growth state determination obtained from previous imaging, topographical information, and soil information.
The topographical information is information on the topography of the field, such as the location of a water gate, the location of sunny or shady areas at a specific date and time, etc. This information can be calculated based on the map information 400 and the field position 511 and the field surrounding coordinates 512 in the field management information 500, and can also be confirmed in advance by surveying or the like. This topographical information is stored in the topographical information 514 in the field management information 500, and the photography condition management module 413 acquires this topographical information and uses it to calculate the photography conditions.

For example, since the growth conditions may change due to the inflow of water when a floodgate is opened, photographing positions can be set at regular intervals from the location of the floodgate.
In addition, since the shooting environment changes depending on whether it is in the sun or in the shade, it is possible to set the shooting position by specifying, for example, a position that is always in the sun or a position that is always in the shade when shooting.
The soil information is, for example, detection information of soil components, etc. This information is acquired, for example, based on detection of soil components in the field, and is stored in the soil information 515 of the field management information 500. The photography condition management module 413 acquires this soil information and uses it to calculate the photography conditions.

It is also possible to increase or decrease the number of times photography is taken based on changes in the growth state based on past photography.
For example, when first deciding the image capture positions, the number is determined from external information such as topographical information and soil conditions, and a relatively large number of image capture positions is set.
The second method of determination involves determining the photographing positions and number of plants based on the results of determining the growth state from the previously photographed image, in addition to external information.

The Nth number is determined based on the results of the assessment of the growth state from previous images, in addition to external information, but the number of shooting positions is reduced each time shooting is performed to save labor. For example, shooting positions where the trends in the assessment results or external information are the same or similar are reduced the number of shooting positions for the next shooting. When multiple fields are the target, the number of fields to be photographed can also be reduced. In other words, if the change in the growth state of the crop obtained from multiple past images taken from the same shooting position is below a predetermined value, information about the same shooting position is not output as a shooting condition.
Conversely, when symptoms of a crop disease appear, such as when disease spots appear, the number of photographing positions is increased, and when changes in growth occur, such as at important stages such as ear emergence, the number of photographing positions is also increased.

As other photographing conditions, the photographing direction is calculated based on at least one of, for example, the rows of the crop, the direction of the sunlight at the time of photographing, and the photographing direction of past photographs.
The direction of the crop rows is stored in the cultivated crop 516 of the field management information 500, the direction of sunlight (front-light or back-light direction) can be calculated from the map information 400 or the field management information 500, and the orientation of past photographs is stored in the photographing information management information 700, and the photographing condition management module 413 acquires this information and uses it to calculate the photographing conditions.
For example, as the photographing conditions, a photographing direction that is along the rows of the crop, in a direction where the light is coming from behind, and close to the direction used in a previous photograph taken at the same position, is calculated and set.

As other shooting conditions, for example, the shooting angle is calculated based on the past shooting angle or a predetermined angle of 5 degrees (±2 degrees). For example, the same shooting angle as the previous shooting is set.
As for other photographing conditions, the photographing procedure is calculated based on, for example, past photographing information and the results of the determination of the growth state. For example, if symptoms of disease were found in the previous photograph, the photographing procedure is calculated to photograph the base, stem, leaves, and tips in that order.

Other shooting conditions include the distance to the subject and distances related to the subject, such as the distance between the subject and the crop (distance to the crop) and the distance from the ground, calculated based on past shooting information and the results of determining the growth state.
As other photographing conditions, a photographing location indicating the location of the subject to be photographed is calculated based on past photographing information, the result of the determination of the growth state, the photographing time, and the crop growth time. For example, based on the photographing time, the photographing location can be set to the leaf area during the period when a disease is prevalent, and the ear tip area during the period when the ear emerges.
In addition, the photographing location (photographing location) for the crop to be photographed is determined or calculated based on the past growth history information of the crop or the date and time.

As other shooting conditions, the shooting time is calculated based on past shooting information, the results of the growth state judgment, the growth phase of the crop, etc. For example, the shooting time is set based on the period when a disease is prevalent or the past trend of changes in the growth state.
As other shooting conditions, shooting settings can be calculated based on past shooting information, growth state determination results, sunlight conditions, etc. For example, shooting settings of the same shutter speed, aperture value, and number of pixels as the previous shooting settings are calculated.
It is not necessary to set all of the various shooting conditions described above, and an appropriate combination of these is set as the shooting condition.

FIG. 12 shows an example of a display of imaging conditions and a flow of imaging processing 1200.
The imaging condition display and imaging process flow 1200 is a process of displaying the imaging condition management information 600 transmitted from the management server 103 on the display unit of the mobile terminal 102 and performing imaging.
The photography management module 213 of the mobile terminal 102 acquires the photography condition management information 600 (step 1210). That is, the photography management module 213 receives the photography condition management information 600 from the management server 103, or reads out the photography condition management information 600 stored in the auxiliary storage device 202 after receiving the information.
The photography management module 213 displays the acquired photography conditions on the output device 205 such as a touch panel (step 1220).

Here, examples of the displayed shooting conditions will be described with reference to FIGS.
FIG. 13 shows an example of a shooting schedule information display screen 1300 displayed on the mobile terminal 102 .
The photography schedule information display screen 1300 displays a photography schedule list 1310, and displays a list 1312 of field names to be photographed on each date 1311 based on the field names and photography periods 619 corresponding to the multiple photography locations 613 listed in the photography condition management information 600.

A plurality of farm fields 1302, 1303, and 1304 are displayed on map information 1301, and in this example, farm field A 1302 is highlighted as the photographing condition for May 10th.
An anchor 1305 is recorded for each field, and position information such as the latitude and longitude of the field is associated with the anchor 1305.
The photography schedule information display screen 1300 also has a compass 1361 indicating a direction, a button 1362 for displaying a wide area of the entire field, a current location button 1363, and a schedule display button 1370. When the schedule display button 1370 is tapped while viewing map information, the photography schedule list 1310 is displayed.

FIG. 14 shows an example of a shooting position list display screen 1400 displayed on the mobile terminal 102 .
A condition title 1421 and a list of photographing positions 1422 are displayed on the photographing position list screen 1400, and icons 1431 indicating the photographing positions are displayed at coordinate positions corresponding to the respective photographing positions in the photographing position list information 1422.
The photographing position list information 1422 displays the recommended photographing order together with the coordinate information of the photographing positions.
Furthermore, the display position coordinates 1442 display the position coordinates, which are the latitude and longitude of the center part of the screen, and when the current location button 1441 is tapped, the map information is updated so that the current location is at the center of the screen.

FIG. 15 shows an example of a shooting location details display screen 1500 displayed on the mobile terminal 102 .
14 for field A1510 indicated in condition title 1521, and then position coordinates No. 5 1531 are selected. A list 1522 of shooting conditions for shooting at position coordinates No. 5 is displayed on the shooting position detail display screen 1500.
As described above, the photography management module 213 of the mobile terminal 102 generates the display screens shown in FIGS. 13 to 15 based on various information stored in the acquired photography condition management information 600.

Returning again to the imaging condition display and imaging process flow 1200 in FIG. 12, the imaging management module 213 accepts the selection of imaging conditions for imaging (step 1230).
In the automatic shooting mode, if the shooting conditions match the shooting situation at the time of shooting (Yes in step 1240), the shooting management module 213 executes shooting (step 1260). Here, if the shooting conditions match the shooting situation at the time of shooting, a shooting instruction to shoot may be displayed or announced on the mobile terminal, and shooting may be executed when it is detected that the shooting button has been selected.
If the automatic shooting mode is not selected, step 1240 is not executed and the process proceeds to the next step 1250.
Even if the shooting conditions do not match, when the shooting button 1611 is tapped (Yes in step 1250), the shooting management module 213 executes shooting (step 1260).
When taking a photograph, the photographing conditions and the actual photographing situation at the time of photographing are stored in the photographing information management information 700 in association with the image (step 1260).

There are various variations in automatic photography, such as:
When it is determined that at least some of the shooting conditions are met, the camera automatically performs shooting.
When it is determined that at least some of the shooting conditions are met, a button that prompts the photographer to take a picture and a display of the shooting conditions, etc. are output.
- If it is determined that at least some of the shooting conditions are met, the camera automatically performs shooting, and if the shooting conditions are not met, a button to prompt the photographer to take a photo and a display of the shooting conditions, etc. are output.

After taking a picture, it is determined whether the actual shooting situation and the shooting conditions match, and if they match, the process is terminated assuming that the shooting end instruction has been accepted (step 1270). When a person takes a picture while holding a mobile terminal in his/her hand, the shooting position, shooting direction, shooting angle, shooting distance, etc. are likely to change during shooting (during sensor exposure), so it is determined whether the average value of these information in the actual shooting situation matches the shooting conditions. Furthermore, by also setting the amount of fluctuation (variation) that is acceptable as a shooting condition, it is possible to determine that the shooting conditions are not met if camera shake or the like occurs when taking a picture. If it is determined that the shooting situation does not meet the shooting conditions due to camera shake or the like, the process returns to step 1240 and takes the picture again.

In addition, in step 1240, if the shooting situation and shooting conditions match (step 1240 is Yes) and the shooting management module 213 executes shooting (step 1260), it is not necessary to determine in step 1270 whether the shooting situation and shooting conditions match.
Also, in step 1270, if an individual instruction to end shooting is received, such as when the photographer turns off the camera, the processing may end without determining whether the shooting situation and shooting conditions match.

FIG. 16 is an example of a screen 1600 displayed when photographing with the mobile terminal 102 .
When the photography management module 213 receives the selection of the photography conditions, it switches the display to the photography screen 1600. In this example, when the photographer comes close to the photography position coordinates 1531 of the field A 1302, the photography conditions for this position are selected.
Note that a configuration may be adopted in which, when the mobile terminal 102 approaches a shooting position using a GPS function or the like, the shooting conditions for that position are automatically selected and the shooting screen 1600 is displayed.

A photographing screen 1600 displays a crop 1610 of a subject captured by the camera unit 206 of the mobile terminal 102, and a condition title 1621 and a list of photographing conditions 1622 are displayed in front of the crop 1610.
The photography condition list 1622 has a judgment column 1630, in which the photography management module 213 judges whether the input information from the input device 204 such as the camera unit 206 of the mobile terminal 102 and various sensors satisfies various conditions and displays the result. By looking at this judgment column, the farm field manager can take pictures while checking to what extent the conditions are satisfied.

When the shooting button 1611 is selected, shooting is performed and an image is acquired. At this time, the shooting condition list 1622 and the judgment result in the judgment column 1630, as well as the actual shooting settings at the time of shooting, are stored in association with the image. In other words, the shooting button 1611 is an indication that executes shooting regardless of whether or not at least some of the shooting conditions are satisfied. Note that this is not limited to a button, and any indication that accepts an instruction to shoot may be used.
When the automatic shooting button 1612 is selected, if all the judgment results in the judgment column 1630 are circles, that is, when all the shooting conditions match the shooting situation, shooting is automatically performed and an image is acquired.
Note that the image does not have to be saved only when all of the conditions are met; it may be set so that the image is captured when a certain percentage, for example 80%, is met, or it may be set so that the image is captured when only a few specific items are marked with a circle.

Also, instead of acquiring and storing an image when a condition is satisfied, all images may be acquired and stored as a moving image or frame-by-frame images at a fixed time interval, and images that match the conditions may be flagged as having satisfied the conditions, so that they can be distinguished and used in later processing. Alternatively, information in the determination column 1630 indicating whether each condition in the shooting condition list 1622 is satisfied may be stored in association with each image in the moving image, and images that satisfy the conditions may be extracted in later processing.
In this way, the mobile terminal 102 can determine whether the shooting conditions calculated by the management server 103 are met at the time of shooting and acquire an image in which the shooting conditions are met, making it possible to shoot and acquire images that are more suitable for determining the crop growth status performed by the management server 103.

In addition, a series of work procedures can be displayed as a shooting procedure in the shooting condition list 1622, and it becomes possible to encourage the photographer to shoot according to this procedure. Examples of the shooting procedure include shooting the sky light first and then the crop, or shooting the base of the plant, the whole plant, and the tips in that order.
For example, if an early disease spot is detected as a result of the growth condition determination of the photographed image by the management server 103, the work procedure to be performed in response to this can be displayed as the photographing conditions, and a video or photograph can be taken to confirm that the work procedure was performed in order, thereby enabling confirmation of the work.
Furthermore, work instruction information other than that for photography may be displayed. For example, if it is determined that the growth state of a crop obtained by past photography shows signs of disease, work instruction information for treating the disease may be displayed.

In the above embodiment, the growth state management information 800 of the management server 103 analyzes the shooting information such as images received from the mobile terminal 102 to determine the growth state, and the shooting condition management module 413 calculates the shooting condition management information 600 based on the growth state determination result. However, the determination of the growth state and the calculation of the shooting conditions may be performed by the mobile terminal 102. In this case, the growth state management module 212 of the mobile terminal 102 determines the growth state, and the mobile terminal 102 may be provided with a shooting condition management module that calculates the shooting conditions.

The device that determines the growth state of the crop in this embodiment is sometimes called a growth management device. When the growth management device is a management server 103 located on the cloud, the output destination of information regarding the shooting conditions is a mobile terminal 102 connected via a network, and when the growth management device is a mobile terminal 102, the output destination of information regarding the shooting conditions is a display unit provided on the mobile terminal.

In the above embodiment, a plurality of photographing positions in a farm field are displayed on a map as in the photographing position list information 1422 in Fig. 14 to indicate the photographing order, and the farm field manager goes to the position and takes a photograph using the mobile terminal 102. At this time, the photographing management module 213 of the mobile terminal 102 may provide a navigation function that guides the farm field manager to the photographing positions. Also, instead of or in addition to the above navigation function, a photographing position where photographing has been completed may be displayed in a different color from a position where photographing has not been completed, thereby making it possible to distinguish between positions where photographing has been completed and positions where photographing has not been completed. This function allows the photographer to easily know which photographing position to go to next.
In another embodiment, the flight path management module of the management server 103 may calculate the flight path of the drone 101 based on the shooting position and shooting time calculated by the shooting condition management module 413, and the flight controller 501 of the drone 101 may take photographs while flying the drone based on the calculated flight path.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other.
The above-described embodiments disclose at least the configurations described in the claims.

101...drone, 102...mobile terminal, 103...management server, 110...base station, 211...photography information management module, 212...growth condition management module, 213...photography management module, 411...photography information management module, 412...growth condition management module, 413...photography condition management module, 400...map information, 500...field management information, 600...photography condition management information, 700...photography information management information, 800...growth condition management information

Claims (17)

A crop growth management device, comprising:
An acquisition unit that acquires image information captured by a camera;
a determination unit that determines a growth state of the crop based on the acquired image information;
an output unit that outputs information;
having
the output unit outputs a photographing condition for determining a growth state of the crop, the photographing condition indicating that the photographing situation of the camera matches at least one of a predetermined photographing position, a predetermined photographing orientation, a predetermined photographing angle, a predetermined photographing distance, a predetermined part of the crop to be photographed, an order in which each part of the crop is photographed, or a predetermined photographing time ;
A growth management device, characterized in that the camera executes photography when at least a part of the photography conditions are satisfied . 2. The growth management device according to claim 1, wherein the photographing positions are one or more positions on a map calculated based on at least one of the growth state of the crop obtained by past photographing, topographical information, and soil condition. The growth management device according to claim 2, which does not output information about the same photographing position as the photographing condition if the change in the growth state of the crop obtained by photographing the same photographing position multiple times in the past is equal to or less than a predetermined value. The growth management device according to any one of claims 1 to 3, wherein the shooting direction is a shooting direction calculated based on at least one of the shooting direction of a previous photograph, the row of the crop, and the direction of sunlight at the time of photographing. A growth management device as described in any one of claims 1 to 4, wherein the shooting angle is a shooting angle calculated based on at least one of a past shooting angle or a predetermined angle determined in advance for the crop to be photographed. 6. The growth management device according to claim 1, wherein the photographing distance is at least one of a distance from the camera to the crop to be photographed and a distance from the ground surface. The growth management device according to claim 1 , wherein the portion of the crop to be photographed is determined based on past growth history information of the crop or date and time. The growth management device according to claim 1, wherein the portion of the crop to be photographed is determined based on the growing season of the crop, and the growing season of the crop is at least one of a period when a disease is prevalent in the crop and a period when the crop produces ears. The growth management device according to any one of claims 1 to 8, wherein the photographing time is calculated based on at least one of the growth phase of the crop, the time when a disease is prevalent in the crop, and the trend of change in the growth condition of the crop obtained by past photographing. The growth management device according to any one of claims 1 to 9 , which outputs a display prompting the user to take a photograph when at least a part of the photographing conditions is satisfied. The growth management device according to any one of claims 1 to 10 , which outputs a display prompting the user to take a photograph in accordance with the photographing conditions when the photographing conditions are not satisfied. The growth management device according to any one of claims 1 to 11, wherein the output unit outputs an automatic shooting button that executes shooting when at least a portion of the shooting conditions are met, and a shooting button that executes shooting regardless of whether at least a portion of the shooting conditions are met. The growth management device according to any one of claims 1 to 12 , wherein the output unit outputs work instruction information when it is determined that the growth condition of the crop obtained by past photography shows signs of disease. The growth management device according to any one of claims 1 to 13 , wherein the growth management device is a server located on a cloud, and the output destination of the information regarding the shooting conditions is a mobile terminal connected via a network. The growth management device according to any one of claims 1 to 14 , wherein the growth management device is a mobile terminal, and the information regarding the photographing conditions is output to a display unit provided in the mobile terminal. A crop growth management program for causing a computer to function as the acquisition unit, the determination unit, and the output unit of the crop growth management device according to any one of claims 1 to 15 . A computer-implemented crop growth management method, comprising:
Acquire image information captured by the camera,
The growth status of crops is judged based on the acquired image information,
outputting a photographing condition for determining a growth state of the crop, the photographing condition indicating that the photographing situation of the camera matches at least one of a predetermined photographing position, a predetermined photographing orientation, a predetermined photographing angle, a predetermined photographing distance, a predetermined part of the crop to be photographed, an order in which each part of the crop is photographed, or a predetermined photographing time ;
A crop growth management method, comprising the steps of : photographing the crop using the camera when at least some of the photographing conditions are satisfied.

Images