JP7591628B2 - Diagnostic device, diagnostic method, and program - Google Patents

Description

The present disclosure relates to a diagnostic device, a diagnostic method , and a program.

A high level of specialized knowledge is required to diagnose the health of plants. Furthermore, it can be difficult for experts with such knowledge to directly observe plants. To solve this problem, Patent Document 1 discloses a learning model that inputs photographed images of a target plant and estimates the pests and diseases of the target plant.

JP 2022-94783 A

This learning model discloses a method for diagnosing pests and diseases in target plants by referring to the probability of pest and disease occurrence calculated from historical data on past pest and disease occurrence, in addition to the estimated results. However, since obtaining historical data on pest and disease occurrence is both time-consuming and laborious, it is desirable to be able to appropriately judge the health of plants without historical data.

The present disclosure aims to provide a diagnostic device, diagnostic method, learning model generation method, and program that can appropriately diagnose the health of a plant based on an image.

A diagnostic device according to one aspect of the present disclosure is a diagnostic device for diagnosing the health of a plant, the diagnostic device including a multi-task learner configured to output a plurality of prediction results associated with a plurality of tasks when an input image is input, the multi-task learner being trained with training data including a ground image group including a plurality of ground images of the plant covering the ground surface, and a wide-range image group including a plurality of wide-range images of the plant covering an area wider than the ground image, each of the ground image group and the wide-range image group including a plurality of asymptomatic images of the plant in a healthy state and a plurality of symptomatic images of the plant in an unhealthy state, the unhealthy state including a plurality of disease symptoms, the plurality of tasks including a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of disease symptoms or asymptomatic symptoms, and a binary classification task of predicting whether the input image corresponds to the ground image or the wide-range image.

A diagnostic method according to one aspect of the present disclosure is a method for diagnosing the health of a plant, the method including: acquiring an input image of the plant covering the ground surface; inputting the input image into a multitask learner; and outputting a plurality of prediction results associated with a plurality of tasks, the multitask learner being trained with training data including a ground image group including a plurality of ground images of the plant covering the ground surface; and a wide-range image group including a plurality of wide-range images of the plant covering an area wider than the ground image, each of the ground image group and the wide-range image group including a plurality of asymptomatic images of the plant in a healthy state and a plurality of symptomatic images of the plant in an unhealthy state, the unhealthy state including a plurality of disease symptoms, and the plurality of tasks including a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of disease symptoms or asymptomatic symptoms, and a binary classification task of predicting whether the input image corresponds to the ground image or the wide-range image.

In a method for generating a learning model according to one aspect of the present disclosure, a learning model for diagnosing the health condition of a plant is configured to output a plurality of prediction results associated with a plurality of tasks when an input image is input, and the generation method includes acquiring a group of images serving as training data and training the learning model using the training data, the training data including a group of ground images including a plurality of ground images depicting the plant covering the ground surface, and a group of wide-range images including a plurality of wide-range images depicting the plant in an area wider than the ground image, each of the group of ground images and the group of wide-range images including a plurality of asymptomatic images depicting the plant in a healthy state and a plurality of symptomatic images depicting the plant in an unhealthy state, the unhealthy state including a plurality of disease symptoms, and the plurality of tasks including a multi-value classification task for predicting whether the plant depicted in the input image corresponds to one of the plurality of disease symptoms or asymptomatic symptoms, and a binary classification task for predicting whether the input image corresponds to the ground image or the wide-range image.

A program according to one aspect of the present disclosure is a plant diagnosis program that causes one or more computers to execute the following: acquiring an input image of a plant covering the ground surface; inputting the acquired input image into a multitask learning device; and outputting a plurality of prediction results associated with a plurality of tasks, respectively. The multitask learning device is trained with training data including a ground image group including a plurality of ground images of the plant covering the ground surface, and a wide-range image group including a plurality of wide-range images of the plant covering an area wider than the ground image. Each of the ground image group and the wide-range image group includes a plurality of asymptomatic images of the plant in a healthy state and a plurality of symptomatic images of the plant in an unhealthy state, and the unhealthy state includes a plurality of disease symptoms. The plurality of tasks include a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of disease symptoms or asymptomatic symptoms, and a binary classification task of predicting whether the input image corresponds to the ground image or the wide-range image.

FIG. 1 is a schematic diagram for explaining a diagnostic device according to an embodiment. FIG. 2 is an example of a wide-range image input to the diagnostic device of FIG. FIG. 3 is an example of a ground image input to the diagnostic device of FIG. FIG. 4 is a table illustrating training data for a learning model according to an embodiment. FIG. 5 is a schematic diagram illustrating a learning model according to the embodiment. FIG. 6 is a graph showing the relationship between the loss function of the learning model in FIG. 5 and the prediction accuracy. FIG. 7 is a table showing the accuracy rate of a learning model including subtasks according to the embodiment and a learning model of a comparative example that does not include subtasks. FIG. 8 is a schematic diagram for explaining additional functions of the diagnostic device of FIG. FIG. 9 shows an example of an input image and a heat map. FIG. 10 is a flow diagram illustrating a diagnostic method according to an embodiment.

Examples of the diagnostic device, diagnostic method, learning model generation method, and program of the present disclosure are described below with reference to the drawings. The present invention is not limited to these examples, but is intended to include all modifications within the meaning and scope of the claims.

[Diagnostic Device]
FIG. 1 illustrates an example of a diagnostic device 30 configured to diagnose the health of a plant. In the present disclosure, grass is illustrated as an example of a plant. Hereinafter, grass growing to cover the ground surface is referred to as lawn 18. Plants that grow in clusters to cover the ground surface are also called ground cover plants, and are often evergreen perennials or evergreen woody plants with a height of about 50 cm or less, about 30 cm or less, or about 10 cm or less. Such plants, such as ground cover plants, are planted to densely cover the ground surface with a single species or multiple species for the purpose of adding color to the landscape, cushioning impacts, protecting the topsoil, suppressing temperature changes, or suppressing the growth of other weeds.

Lawns are generally cut short to keep the leaves short. Mowing the lawn drastically reduces the surface area available for photosynthesis. To compensate for this reduced efficiency of photosynthesis, the grass on the lawn increases the number of leaves so that they grow denser, forming a green carpet. Mown grass is less resistant to disease than unmown grass, and requires management such as spraying pesticides according to the symptoms of disease. The diagnostic device 30 can be used to determine whether or not management of plants such as lawns is required.

Regarding the health of the lawn, if it is growing evenly all over, it is asymptomatic and can be determined to be in a healthy state. In contrast, if the planted area of the lawn is peeling, withering, discolored, or patchy, it is symptomatic and can be determined to be in an unhealthy state.

In FIG. 1, the location where the grass 18 is planted is shown as a golf course 11, but it may also be a park, a kindergarten yard, a schoolyard, a horse racing track, or an athletic field such as a soccer field, an American football field, a baseball field, or a tennis court. In addition, the diagnostic device 30 may be used to diagnose the health of natural vegetation.

A photographer 20 operates a photographing device 21 to capture one or more images 10 of grass laid on a golf course 11. The photographing device 21 may be, for example, but is not limited to, a portable digital camera, a video camera, a smartphone, or a tablet. The images 10 may be captured by a photographing device mounted on an unmanned vehicle such as a drone or an autonomous vehicle, or may be captured by a fixed-point camera. The images 10 are captured periodically or irregularly. The images 10 may be captured at a fixed point, or may be captured at a different position each time.

The photographer 20 may photograph one or more predetermined shooting locations, for example one or more areas such as the green 12 that requires particular maintenance. The photographer 20 may take multiple images 10 for each location. The multiple images 10 may include, for example, one or more ground surface images 10A (see FIG. 3) that show vegetation (grass 18) covering the ground surface, and one or more wide-area images 10B (see FIG. 2) that show vegetation (grass 18) covering an area wider than the ground surface image 10A.

As shown in Figures 1 and 2, the wide-range image 10B may show natural or man-made objects in or around the lawn 18. An example of a natural object in the lawn 18 is the boundary line of the green 12, and an example of a natural object around the lawn 18 is a bunker 14, a pond 15, or a tree 16. An example of a man-made object in the lawn 18 is a cup hole 13 or a pin, and an example of a man-made object around the lawn 18 is a building such as a maintenance facility 17.

As shown in FIG. 3, the ground surface image 10A should be captured primarily to show the health of the lawn 18. In particular, if the photographer 20 visually observes an unhealthy condition in the lawn 18 that may be due to disease, such as peeling, withering, discoloration, or mottling, the photographer 20 should capture the ground surface image 10A to capture that condition. An unhealthy condition includes multiple symptoms of disease. If an unhealthy condition is confirmed, the photographer 20 should capture a wide-range image 10B in addition to the ground surface image 10A so that the distribution of symptoms can be seen.

Ground surface image 10A1 shown in FIG. 3 is an example of a symptomless image depicting a healthy lawn 18. Ground surface images 10A2 to 10A7 shown in FIG. 3 are examples of symptomatic images depicting unhealthy lawn 18. The types of disease may include, for example, one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, and snow mold. Ground surface images 10A2, 10A3, 10A4, 10A5, 10A6, and 10A7 depict lawns 18 exhibiting symptoms of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold, respectively.

The captured image 10 may be stored in the user terminal 22. The user terminal 22 may be located, for example, in the management facility 17 of the golf course 11. The user terminal 22 is, for example, a personal computer. The user terminal 22 may be configured to communicate with the diagnostic device 30 through the network 23. In this case, the image 10 stored in the user terminal 22 may be transmitted to the diagnostic device 30 through the network 23.

The network 23 includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, etc. When the imaging device 21 is a terminal having a communication function, such as a smartphone or a tablet, the captured image may be sent directly to the diagnostic device 30 without going through the user terminal 22. Alternatively, the user terminal 22 or a terminal having the imaging device 21 may be used as the diagnostic device 30.

The diagnostic device 30 is, for example, a server or computer including a processor 31, a memory 32, and a communication IF (interface) 33. The diagnostic device 30 may further include an input device 34, an output device 35, and a storage 36. The processor 31, the memory 32, the communication IF 33, the input device 34, the output device 35, and the storage 36 are connected to each other by a communication bus 37.

The processor 31 is, for example, a processing circuit configured to execute various software processes. The processing circuit may include a dedicated hardware circuit (e.g., an ASIC) that processes at least a part of the software processes. In other words, the software processes may be executed by a processing circuit that includes at least one of one or more software processing circuits and one or more dedicated hardware circuits.

The processor 31 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other computing device. In response to a signal provided thereto or in response to the establishment of a predetermined condition, the processor 31 executes a series of instructions contained in a program stored in the memory 32 or storage 36.

The memory 32 is, for example, a random access memory (RAM) or other volatile memory. The memory 32 is configured to temporarily store programs and data. The programs are read from, for example, the storage 36. The data stored in the memory 32 may include data received by the diagnostic device 30 and data generated by the processor 31.

The communication IF 33 is configured to connect to the network 23. The communication IF 33 is configured to communicate with other devices connected to the network 23. The communication IF 33 is realized, for example, as a LAN or other wired communication IF. The communication IF 33 can also be realized, for example, as Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless communication IF, but is not limited to these.

The input device 34 may be, for example, a keyboard and a mouse, and may also include buttons, keys, switches, a touchpad, or a microphone. The output device 35 may be, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor, or may be a touch screen having a touch panel that also serves as the input device 34. The output device 35 may be a speaker or a terminal for audio output.

The storage 36 is a type of memory and may be, for example, a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 36 may be a removable storage device such as a memory card. The storage 36 is configured to permanently store programs and data. The storage 36 may store a learning device 40, image data 41, one or more applications 43 for realizing various functions, and output data 44. The image data 41 includes one or more images 10 received from the user terminal 22. The output data 44 includes output results from the learning device 40 and the application 43.

The learning device 40 may be a multitask learning device configured to output a plurality of prediction results associated with a plurality of tasks, respectively, when the input image 10 is input. The learning device 40 may include a learning model 45 that uses a convolutional neural network. The learning model 45 included in the learning device 40 may be created as follows.

[How to generate a learning model]
1 is a computer including, for example, a processor 51, a memory 52, and a communication IF (interface) 53. The information processing device 50 may further include an input device 54, an output device 55, and a storage 56. The processor 51, the memory 52, the communication IF 53, the input device 54, the output device 55, and the storage 56 are connected to each other by a communication bus 57. The processor 51, the memory 52, the communication IF 53, the input device 54, the output device 55, and the storage 56 may have the same configuration as the processor 31, the memory 32, the communication IF 33, the input device 34, the output device 35, and the storage 36, respectively.

To generate a learning model, the processor 51 generates or acquires one or more learning programs 58 for machine learning and stores them in the storage 56. The processor 51 also generates or acquires training data 59 and stores them in the storage 56. The processor 51 then executes the learning program 58 to train, verify, and test the learning model using the training data 59 shown in FIG. 4. The learning model 45 generalized (generated) using the training data 59 is stored in the storage 56. The learning model 45 included in the learner 40 is a trained learning model 45 acquired from the information processing device 50 via the network 23.

The training data 59 includes a group of ground images and a group of wide-range images. The group of ground images includes a plurality of ground images capturing plants (lawn 18 in this example) covering the ground surface. The ground images can also be considered as zoomed-in images of plants captured by zooming in more than the wide-range images. The group of wide-range images includes a plurality of wide-range images capturing plants (lawn 18 in this example) over an area wider than the ground image. The wide-range images can also be considered as zoomed-out images of plants captured by zooming out more than the ground image. The ground images and the wide-range images may be captured at different positions.

A ground surface image and a wide-range image are generally distinguished by the objects contained within their shooting range. For example, a ground surface image may be an image in which the ground surface covered with vegetation occupies the entire shooting range. A wide-range image may be a landscape image that captures at least one of natural objects such as trees growing from the ground surface or the sky, or man-made objects including buildings, in addition to the ground surface covered with vegetation (grass 18).

The wide-range image may be an image captured from a higher position than the ground image, for example by an aircraft such as a drone. In this way, even if the ground surface occupies the entire shooting range of an image, an image captured at a high position higher than a person's eye level may be classified as a wide-range image. When a person takes a picture by holding the camera device 21 directly in their hand, the ground image may be an image captured at a high angle (indicated by the diagonally downward arrow in Figure 1), and the wide-range image may be an image captured at a lower angle than the ground image, for example a horizontal angle (indicated by the left-facing arrow in Figure 1) or a low angle (indicated by the diagonally upward arrow in Figure 1).

Each of the group of ground images and the group of wide-area images includes a plurality of asymptomatic images of healthy plants and a plurality of symptomatic images of unhealthy plants. Each of the images used as training data 59 is labeled with a first correct answer label indicating whether the image is a disease type or asymptomatic, and a second correct answer label indicating whether the image is a ground image or a wide-area image.

The symptomatic image group includes images of symptoms of multiple diseases. The diseases include, for example, one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, and snow mold. The types of diseases prepared as training data 59 become the types of diseases that can be diagnosed by the diagnostic device 30. Each of the ground surface image group and the wide-area image group may include multiple images of each symptom as training data 59.

As shown in FIG. 4, the training data 59 includes learning data, verification data, and test data. FIG. 4 shows an example of the number of each disease image used in training. The training data 59 may be collected, for example, by crawling data on the web using the name of the disease as a keyword. In this case, the name of the disease used as the keyword may be attached to the collected images as the first correct label. In addition, objects or feature points in the collected images may be extracted to determine whether they are surface images or wide-area images, and the result may be attached as the second correct label.

The amount of training data 59 may be increased by subjecting the collected images to one or more image processing operations, such as rotation, movement, enlargement, reduction, inversion, color change, noise addition, blurring, high frequency component emphasis, or distortion. By adding the processed images to the training data 59, it becomes possible to learn about images with different positions, orientations, sizes, brightness, or saturation.

As shown in FIG. 5, the learning model 45 uses a backbone of a convolutional neural network (CNN). Examples of convolutional neural networks are Resnet34, ResNet50, ResNet101, or EfficientNetB7, but in this example, ResNet50 is used. The learning model 45 includes a common layer 46, which is the backbone, and multiple (two in this example) output layers 47 (47A, 47B).

The common layer 46 includes an input layer to which the input image 10 is input, and multiple intermediate layers trained with training data 59. The multiple (two in this example) output layers 47A, 47B are associated with multiple (two in this example) different tasks. The learning model 45 includes a program that causes the computer to input the input image 10, which depicts vegetation covering the ground surface, into the input layer (common layer 46), and output multiple prediction results from the multiple output layers 47, each associated with a multiple task.

The multiple tasks include a multi-value classification task for predicting whether the plant (grass 18) depicted in the input image 10 corresponds to a disease type or to asymptomatic, and a binary classification task for predicting whether the input image 10 corresponds to the ground surface image 10A or the wide-range image 10B. The multi-value classification task is also called the first task or main task. The binary classification task is also called the second task or subtask. The multi-value classification task is for making predictions related to plant diseases. The binary classification task is for predicting the zoom level of the input image 10. The learning model 45 is also a classifier or neural network system for simultaneously performing classification related to plant diseases and classification based on the zoom level of the input image 10.

As shown in FIG. 5, the first output layer 47A is configured to output a prediction result 48A of the first task, and the second output layer 47B is configured to output a prediction result 48B of the second task. In this way, the learning model 45 is configured to output multiple (two) prediction results 48 (48A, 48B) for one input image 10.

The multi-value classification task may be configured to output probabilities corresponding to multiple diseases or asymptomatic as prediction results. The output probabilities are not limited to percentages, and may be displayed so that the sum of the probabilities is 1. For example, the multi-value classification task may output the probabilities corresponding to each of the seven classifications (also called classes) of six types of diseases and asymptomatic, or may output the highest class with the highest probability and its probability, for example, "asymptomatic: 96.4%". Alternatively, the multi-value classification task may output the top two or three classes with the highest probability among the seven classes and numerical values corresponding to the respective probabilities, for example, "asymptomatic: 0.964, dry spot: 0.015, brown patch: 0.008". The binary classification task may output whether the input image 10 corresponds to the earth surface image 10A or the wide-range image 10B, and the probability of the corresponding condition.

In the learning model 45, the cross entropy loss (hereafter, CELoss) is set as the loss for the main task, and the binary cross entropy loss (hereafter, BCELoss) is set as the loss for the subtask. The loss function of the learning model 45 for the two tasks is integrated as (1-λ)CELoss+λBCELoss.

Figure 6 shows the results of verifying the classification accuracy of the main task and subtasks according to λ. As shown in Figure 6, the accuracy of the main task is highest when λ = 0.05, but the prediction accuracy of the subtasks is low at this time. The main purpose of the learning model 45 is to diagnose the health of the plant. Therefore, λ = 0.05 was set so that the prediction accuracy of the main task would be high. In other words, the loss function of the learning model 45 is set so that the prediction performance of the main task is higher than the prediction performance of the subtasks.

Figure 7 shows the results of comparing the accuracy of the learning model 45 of this example with subtasks and a single-task learning model without subtasks. The top-1 accuracy is the probability (accuracy rate) that the most probable prediction result of the multi-value classification task matches the first correct label. In addition, the accuracy rate of whether the prediction result of the multi-value classification task is symptomatic or asymptomatic is shown as the "symptomatic vs. asymptomatic accuracy rate" (prediction of the presence or absence of disease).

As shown in Figure 7, the accuracy rate for symptomatic vs. asymptomatic does not differ with or without subtasks, but the first place accuracy rate is higher for learning model 45 with subtasks. Therefore, it can be said that adding a subtask for predicting zoom level to learning model 45 for diagnosing the health of plants increases the prediction accuracy. In this way, subtasks improve the generalization performance of the main task.

[Additional Functions of the Diagnostic Device]
The diagnostic device 30 has a basic function of predicting plant diseases using a learning model 45, but additionally has an application 43 that has a function of outputting diagnostic results based on the prediction results of the learning model 45.

As shown in FIG. 8, the application 43 may include one or more of the following optional additional programs: a diagnosis output unit 71, a heat map generation unit 72, a shooting position acquisition unit 73, a position information creation unit 74, a shooting range estimation unit 75, a three-dimensional data generation unit 76, and a planar image generation unit 77. When the processor 31 executes the programs 71 to 77, the diagnostic device 30 operates as the functional units described below. In other words, the diagnostic device 30 includes a diagnosis output unit 71, a heat map generation unit 72, a shooting position acquisition unit 73, a position information creation unit 74, a shooting range estimation unit 75, a three-dimensional data generation unit 76, and a planar image generation unit 77.

The diagnosis output unit 71 is configured to be executed by the processor 31 to output a diagnosis result regarding whether or not the plant needs care, based on the prediction result of the learning device 40. If the prediction result of the multi-value classification task is asymptomatic, the diagnosis output unit 71 outputs a diagnosis result that no care is required.

The diagnosis output unit 71 may output a diagnosis result indicating that care is required if the predicted result of the multi-value classification task is any of a plurality of diseases. In this case, the output diagnosis result may include the details of care, such as spraying a chemical according to the disease. In one example, the diagnosis output unit 71 is configured to output a diagnosis result indicating that care is required for the plant if the probability of any of the diseases output by the multi-value classification task exceeds a symptom threshold (e.g., 70%). Alternatively, the diagnosis output unit 71 may output a diagnosis result indicating that care is required for the plant if the probability of the disease exceeds a first symptom threshold (e.g., 90%), and output a diagnosis result indicating that observation is required if the probability of the disease is less than the first symptom threshold (e.g., 90%) and a second symptom threshold (e.g., 70%).

The diagnosis output unit 71 may output a diagnosis result regarding the need for plant care after taking into account the prediction result of the binary classification task to the prediction result of the multi-value classification task. For example, the diagnosis output unit 71 may be configured to output a diagnosis result regarding the need for plant care when the prediction result of the binary classification task corresponds to the wide-range image 10B and the prediction result of the multi-value classification task is any of a plurality of diseases. In this case, the diagnosis output unit 71 may output a diagnosis result that the plant needs care based on the wide-range image 10B. In this case, it is preferable to capture a set of the ground surface image 10A and the wide-range image 10B at each shooting location. More specifically, when the distribution value related to the spread of the disease in the wide-range image 10B exceeds a threshold value, the diagnosis output unit 71 may output a diagnosis result that the plant needs care.

After obtaining the prediction result of the multi-value classification task, the diagnosis output unit 71 may perform additional diagnosis and then output a diagnosis result indicating that maintenance is required. The additional diagnosis may be performed based on one or more of the type, scope, degree, or timing of the disease, for example. The additional diagnosis may be realized as an additional function of the learning device 40, or may be realized by executing a program included in the application 43. The prediction result output by the diagnosis output unit 71 is associated with the additional diagnosis and related data as necessary and stored in the storage 36.

The heat map generating unit 72 is configured to be executed by the processor 31 to generate a heat map 65 in which the prediction basis of the multi-value classification task is superimposed on the input image 10 as shown in FIG. 9. The heat map generating unit 72 can use, for example, Grad-CAM. In this case, the heat map 65 is a feature map of the final output of the multi-value classification task, with the areas that form the basis of the prediction shown in darker colors. The generated heat map 65 is associated with the input image 10 and the prediction result and stored in the storage 36.

By referring to the heat map 65, it is possible to visualize which part of the image 10 is the basis for the judgment. This allows the user to confirm that the judgment has been made on the correct basis if the judgment is based on dead parts of the diseased grass in the image 10, and can infer that the prediction is inaccurate if, for example, an artificial object other than grass is the basis for the judgment.

The shooting position acquisition unit 73 is configured to be executed by the processor 31 to acquire position information related to the shooting position 60 of the input image 10. The position information may be attached to the image 10 by the shooting device 21, for example. The position information acquired by the shooting position acquisition unit 73 is stored in the storage 36. The shooting position 60 may be output or displayed together with the diagnosis result output by the diagnosis output unit 71.

When the application 43 includes a location information creation unit 74, the storage 36 stores map data 61 including the location where the input image 10 was taken, for example, a golf course 11. The map data 61 may include a map including latitude and longitude, or in the case of a golf course 11, may include a layout diagram unique to the facility, such as a course layout. The location information creation unit 74 is configured to be executed by the processor 31 to create location information that plots the shooting location 60 on the map data 61.

When using the three-dimensional data generation unit 76, the photographer 20 acquires multiple landscape images taken from different viewpoints so as to include the shooting range. The multiple landscape images may be replaced with a video. A landscape image group including the multiple landscape images is transmitted from the shooting device 21 to the diagnosis device 30 via the network 23 and stored in the storage 36. The three-dimensional data generation unit 76 is configured to generate three-dimensional data 62 of the target area (e.g., golf course 11) from the landscape image group or the video.

The planar image generating unit 77 is configured to generate a planar image 63 from the three-dimensional data 62. Specifically, the planar image generating unit 77 generates a planar image 63 that is a plan view of the three-dimensional data 62 viewed from above. The position information creating unit 74 may create position information in which the shooting position 60 is plotted on the planar image 63 created by the planar image generating unit 77. The position information creating unit 74 may overlay the planar image 63 with map data 61 including latitude and longitude, and then create position information in which the shooting position 60 is plotted. The planar image 63 including the shooting position 60 of the input image 10 thus generated is stored in the storage 36.

The shooting range estimation unit 75 is configured to be executed by the processor 31 to estimate the shooting range 64 of the input image 10. The shooting range estimation unit 75 may be configured to extract characteristic objects or feature points from the image 10 and compare them with the three-dimensional data 62. The shooting range estimation unit 75 may then be configured to estimate the shooting position 60 and the shooting range 64 by determining corresponding objects or feature points in the image 10 and the three-dimensional data 62.

When the image 10 is predicted to be a ground surface image 10A, the shooting range estimation unit 75 may estimate the shooting position 60 (point data) or a specified area including the shooting position 60 (e.g., an area of 30 to 50 cm square) as the shooting range 64 without extracting objects or feature points. The shooting range estimation unit 75 may be configured to overlay the shooting position 60 and the shooting range 64 on the planar image 63.

The diagnostic output unit 71 may output one or more of the heat map 65, the three-dimensional data 62, the planar image 63, the photographing position 60, or the photographing range 64 together with the prediction result as the diagnostic result. For example, when the prediction result of the main task corresponds to any one of a plurality of diseases, the diagnostic output unit 71 may add information indicating that it is the predicted disease to the photographing range 64 of the planar image 63 and output it. The application 43 may include a program for displaying the diagnostic result output by the diagnostic output unit 71 by combining it into one screen or by dividing it into a plurality of switchable screens.

[Method for diagnosing the health of a plant]
A method for diagnosing the health condition of a plant will be described with reference to Fig. 10. Steps S10 to S14 and S17 are steps relating to displaying the diagnosis results, and steps S8 and S15 to S16 are steps relating to predicting disease damage by the learning device 40. Steps S10 to S14 may be performed after step S16.

First, in step S8, the processor 31 acquires an input image 10 that shows vegetation covering the ground surface. The following steps S10 to S12 are processes related to the creation of a background map for displaying the prediction results. First, in step S10, the processor 31 acquires a group of landscape images. In step S11, the three-dimensional data generation unit 76 generates three-dimensional data 62 based on the group of landscape images. Then, in step S12, the planar image generation unit 77 generates a planar image 63 based on the three-dimensional data 62. This results in the planar image 63 that serves as the background image.

Steps S13 and S14 are preparation steps for linking the shooting position 60 and the shooting range 64 with the background image. In step S13, the shooting position acquisition unit 73 acquires the shooting position 60. In the following step S14, the shooting range estimation unit 75 estimates the shooting range 64. The estimated shooting position 60 and shooting range 64 may be overlaid on the planar image 63.

In step S15, the processor 31 inputs the image 10 to the learning device 40. The image 10 input at this time is the same as the image 10 for which the shooting position 60 was acquired and the shooting range 64 was estimated in steps S13 and S14. Next, in step S16, the learning device 40 outputs the prediction results of the main task and subtask.

In step S17, the heat map generator 72 generates a heat map 65 to display the prediction basis of the prediction result in step S16. The generated heat map 65 may be synthesized so as to be aligned with the original image 10.

In step S18, the diagnosis output unit 71 outputs the diagnosis result. The diagnosis result may include the prediction result of the main task output in step S16 and a diagnosis of whether or not the plant needs to be cared for, and may not include the prediction result of the subtask. The prediction result of the main task may be output in association with the shooting position 60 and shooting range 64 superimposed on the planar image 63, the original image 10, and the heat map 65.

Finally, in step S19, the processor 31 transmits the diagnosis result to the user terminal 22 via the network 23. If the diagnostic device 30 does not include a diagnostic output unit 71, the processor 31 may transmit the prediction result of step S16 to the user terminal 22 instead of the diagnosis result of step S18.

[Action of the Present Disclosure]
In the case of agricultural or ornamental plants, disease and pests are often judged from the condition of the leaves. However, grasses such as grass have thin leaves, and the condition of the leaves is difficult to see when the grass is cut short. Therefore, when a lawn is infected with disease, the disease is often discovered when the disease has spread to a certain extent and the green area covering the ground becomes partially peeled, withered, discolored, or mottled.

To diagnose the symptoms of disease in grass based on images, it is necessary to capture a fairly wide area. However, to determine the type of disease, it is necessary to get close to the grass and capture the images so that the condition of the grass blades can be seen.

In this regard, since the learning device 40 is trained with training data 59 that includes both ground images and wide-range images, it is possible to diagnose the health of the plants depicted in the image 10 regardless of whether the input image 10 is a ground image 10A or a wide-range image 10B. For the photographer 20, management is also easier because the image 10 can be acquired without being restricted by the shooting range 64.

The learning model 45 of the learning device 40 includes subtasks in addition to the multi-value classification task related to disease diagnosis, and therefore has higher prediction accuracy for the multi-value classification task than a learning model that does not include subtasks. This makes it possible to properly diagnose the health of the lawn based on the image 10.

[Effects of the present disclosure]
According to the present disclosure, the following effects can be achieved.
(1) By adding a binary classification task to a multi-value classification task, the prediction accuracy of the multi-value classification task can be improved.

(2) The diagnosis output unit 71 can output a diagnosis result regarding whether or not the plant needs care based on the prediction result of the learning device 40.
(3) By outputting the prediction results of a multi-class classification task as a probability, it is possible to show the user an indicator of the reliability of the prediction results.

(4) When the prediction result of the binary classification task corresponds to the wide-range image 10B and the prediction result of the multi-value classification task is one of a plurality of diseases, it becomes possible to determine the extent of the disease based on the wide-range image 10B that is the basis of the prediction. Therefore, it is possible to diagnose that the plant needs care based on the spread of the disease ascertained from the wide-range image 10B.

(5) By comparing the heat map 65 with the input image 10, the user can confirm which part of the image 10 was the basis for the output of the prediction result.
(6) By acquiring location information related to the shooting location 60 of the image 10, the shooting location can be associated with the prediction result.

(7) By creating location information by plotting the photography location 60 on map data, the user can easily manage plants based on the diagnosis results.
(8) By generating a planar image 63 based on the three-dimensional data 62, the environment in which the plant grows can be illustrated.

(9) The area related to diagnosis can be illustrated by superimposing the shooting range 64 of the wide-range image 10B on the planar image 63. This makes it easier for the user to manage the plant.
[Example of change]
This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.

[Modification 1] One or more ground surface images 10A and one or more wide-range images 10B taken at the same photographing position 60 may be input as a set to the learning device 40. In this case, the type and presence or absence of disease may be determined based on the prediction result of the multi-value classification task when the ground surface image 10A is input, and the diagnosis output unit 71 may output a diagnosis result regarding the need for plant care based on the wide-range image 10B.

[Modification 2] The subtask may be changed from a binary classification task to a task that performs classification into three or more classes. For example, three or more zoom levels may be set, or a three-class classification may be performed by adding aerial images taken from the sky by an aircraft such as a drone to ground surface images and wide-area images taken from different angles from the ground.

[Modification 3] The learner 40 may be a single-task learner. The single-task learner may be a multi-modal model that inputs an image and other information. For example, along with the input image 10, whether the image 10 is a surface image or a wide-range image may also be input to solve a multi-value classification task, which is a single task. Other examples of information to be added to the image include the conditions at the time of shooting, such as temperature, weather, location, and season.

[Modification 4] The diagnostic device 30 may include multiple single-task learners instead of the multi-task learner 40. For example, the diagnostic device 30 may include a first learner trained on a group of ground surface images that predicts plant disease, and a second learner trained on a group of wide-area images that predicts plant disease. Furthermore, the diagnostic device 30 may include a third learner that predicts the zoom level of the image. In this case, after the third learner predicts whether the input image corresponds to a ground surface image or a wide-area image, if the image is a ground surface image, it may be input to the first learner, and if the image is a wide-area image, it may be input to the second learner to obtain a prediction result related to plant disease.

[Modification 5] The learning device 40 may be a multitask learning device including three or more tasks. The additional tasks may be, for example, prediction of the type of plant, prediction of the management status of the plant (e.g., whether or not the lawn has been mowed), prediction of diseased areas, prediction of the ratio of diseased areas to non-dangerous areas, and prediction of the season when the image was taken. When predicting the type of plant, a single diagnostic device 30 can diagnose multiple plants. Examples of plants other than grass include ground cover plants such as Daichondra, Iwadatasou, Clavia, and White clover. Alternatively, different types of grass, such as wild grass, Korean grass, and Western grass, may be diagnosed separately, or they may be classified according to growth type, such as summer grass and winter grass.

[Modification 6] A portion of a landscape image may be extracted to generate one or more ground images and one or more wide-range images for input or training.
[Variation 7] In addition to disease, the unhealthy state may include one or more of the following: symptoms of insect damage, withering, and physical damage (such as hollows in the ground).

[Modification 8] The diagnostic device 30 may also function as the information processing device 50. In other words, the computer that is the diagnostic device 30 may generate (train) the learning model 45.
[Modification 9] The learning device 40 may be provided as a diagnostic program for diagnosing the health condition of a plant together with at least a part of the application 43. The diagnostic program causes one or more computers to acquire an input image 10 depicting a plant covering the ground surface, input the acquired input image 10 to the learning device 40, and output a plurality of prediction results associated with a plurality of tasks, respectively. The diagnostic program may be downloaded to a personal computer, tablet, or smartphone such as the user terminal 22 and used as a diagnostic application. The diagnostic program may be used as a cloud application provided by the diagnostic device 30 via the network 23.

[Variation 10] In addition to the group of ground images and the group of wide-area images, the training data 59 may also include attribute information related to the user or imaging device that captured the ground images (e.g., the camera or unmanned aerial vehicle such as a drone used for capturing the images). This attribute information may include at least one of the following pieces of information 1) to 4).

1) Photographer identification information indicating whether the subject who photographed the ground image is a human or not;
2) Photographer information including at least one of the gender and age group of the person who photographed the ground surface image;
3) biographical information including at least one of a medical history and a golfing history of the person who captured the ground image;
4) Equipment information including at least one of the year and model number of the imaging device (e.g., a drone) that captured the ground image.

The learning device 40 may train the learning model 45 based on training data 59 that includes such attribute information. In this case, the diagnostic device 30 may obtain a diagnostic result by inputting the attribute information in addition to the input image 10 to the input layer of the learning model 45.

Aspects grasped based on the above embodiment and modified examples are listed below.
[1] A diagnostic device for diagnosing a health condition of a plant, the diagnostic device comprising:
a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks,
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The ill health condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
Diagnostic equipment.

[2] A diagnosis output unit configured to output a diagnosis result regarding whether or not the plant needs care based on a prediction result of the multi-task learning device.
The diagnostic device according to [1] above.

[3] The multi-value classification task is configured to output probabilities corresponding to the plurality of diseases or asymptomatic diseases as a prediction result;
The diagnosis output unit is configured to output a diagnosis result indicating that care of the plant is required when the probability of any of the diseases output by the multi-value classification task exceeds a symptom threshold.
The diagnostic device according to [2] above.

[4] The diagnosis output unit is configured to output, when a prediction result of the binary classification task corresponds to the wide-range image and a prediction result of the multi-value classification task is any one of the plurality of diseases, a diagnosis result that the plant needs care based on the wide-range image.
The diagnostic device according to any one of claims 2 to 3.

[5] A heat map generating unit configured to generate a heat map in which a prediction basis of the multi-value classification task is superimposed on the input image.
The diagnostic device according to any one of the above [1] to [4].

[6] A photographing position acquisition unit configured to acquire position information relating to a photographing position of the input image;
A location information generating unit;
one or more memories;
The memory stores map data including the photographing position of the input image,
the location information creation unit is configured to create location information by plotting the shooting location on the map data.
The diagnostic device according to any one of the above [1] to [5].

[7] A three-dimensional data generating unit configured to generate three-dimensional data from a landscape image group including a plurality of landscape images captured from different viewpoints;
a planar image generator configured to generate a planar image from the three-dimensional data;
The diagnostic device according to any one of the above items [1] to [6].

[8] A shooting range estimation unit configured to estimate a shooting range of the input image.
The diagnostic device according to any one of the above [1] to [7].

[9] The multi-task learner includes a learning model using a convolutional neural network;
a loss function of the learning model is set so that the predictive performance of the multi-value classification task is higher than the predictive performance of the binary classification task.
The diagnostic device according to any one of the above [1] to [8].

[10] The plant is a ground cover plant that grows in clusters to cover the ground surface.
The diagnostic device according to any one of the above [1] to [9].

[11] The plant is grass,
The plurality of diseases include one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold;
The diagnostic device according to any one of the above [1] to [10].

[12] The ground surface image is an image in which the ground surface covered with the vegetation occupies an entire shooting range,
The wide-range image is a landscape image that includes at least one of trees growing on the ground surface and artificial objects including buildings in addition to the ground surface covered with vegetation.
The diagnostic device according to any one of the above [1] to [11].

[13] A method for diagnosing the health of a plant, the method comprising:
acquiring an input image showing the vegetation covering the ground surface;
inputting the input image into a multitask learner;
the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks;
Including,
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The ill health condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
Diagnostic methods.

[14] A method for generating a learning model for diagnosing a health condition of a plant, the learning model being configured to output a plurality of prediction results associated with a plurality of tasks when an input image is input;
The method for producing the same comprises the steps of:
Obtaining a group of images that serve as training data;
training the learning model using the training data;
The training data is
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image;
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
How to generate a learning model.

[15] one or more computers,
acquiring an input image showing vegetation covering a ground surface;
inputting the acquired input image into a multitask learning device;
the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks;
A program for plant diagnosis,
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
program.

[16] A diagnostic device for diagnosing a health condition of a plant, the diagnostic device comprising:
One or more processors and one or more memories;
The memory stores a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks,
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
Diagnostic equipment.

[17] The processor is configured to perform one or more of a plurality of operations;
The plurality of processes include:
A process of inputting the input image to the multi-task learner and outputting the plurality of prediction results;
outputting a diagnosis result regarding whether or not the plant needs care based on the prediction result of the multi-task learner;
A process of generating a heat map in which prediction grounds for the multi-class classification task are superimposed on the input image;
A process of acquiring position information relating to a photographing position of the input image;
The diagnostic device according to [16] above,

[18] An input layer to which an input image is input;
A hidden layer trained with the training data;
a plurality of output layers respectively associated with a plurality of tasks;
A learning model for causing a computer to function in such a way that an input image showing plants covering a ground surface is input to the input layer, and a plurality of prediction results respectively associated with the plurality of tasks are output from the plurality of output layers,
The training data is
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image;
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
Learning model.

10...Image (input image), 10A (10A1 to 10A7)...Surface image, 10B...Wide-range image, 11...Golf course, 12...Green, 13...Cup hole, 14...Bunker, 15...Pond, 16...Trees, 17...Management facility, 18...Lawn, 20...Photographer, 21...Photographing device, 22...User terminal, 23...Network, 30...Diagnosis device, 31...Processor, 32...Memory, 33...Communication IF, 34...Input device, 35...Output device, 36...Storage, 37...Communication bus, 40...Learning device, 41...Image data, 43...Application, 44...Output data, 45...Learning model, 46...Common layer , 47 (47A, 47B)...output layer, 48 (48A, 48B)...prediction result, 50...information processing device, 51...processor, 52...memory, 53...communication IF, 54...input device, 55...output device, 56...storage, 57...communication bus, 58...learning program, 59...training data, 60...shooting position, 61...map data, 62...three-dimensional data, 63...planar image, 64...shooting range, 65...heat map, 71...diagnosis output unit, 72...heat map generation unit, 73...shooting position acquisition unit, 74...position information creation unit, 75...shooting range estimation unit, 76...three-dimensional data generation unit, 77...planar image generation unit.

Claims (14)

A diagnostic device for diagnosing a health condition of a plant, the diagnostic device comprising:
a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks ;
a capturing range estimating unit configured to estimate a capturing range of the input image;
Equipped with
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
The shooting range estimation unit is
when a prediction result of the binary classification task is the wide-range image, a characteristic object or feature point is extracted from the input image, the extracted characteristic object or feature point is compared with three-dimensional data generated from a plurality of landscape images taken from different viewpoints in the region, and corresponding objects or feature points between the input image and the three-dimensional data are identified, thereby estimating the shooting range;
When a prediction result of the binary classification task is the ground image, a specified area including a shooting position of the input image is estimated as the shooting range without extracting the characteristic object or feature point from the input image.
Diagnostic equipment. a diagnosis output unit configured to output a diagnosis result regarding whether or not the plant needs care based on a prediction result of the multi-task learning device;
The diagnostic device of claim 1 . The multi-value classification task is configured to output, as a prediction result, probabilities corresponding to the plurality of diseases or asymptomatic diseases;
The diagnosis output unit is configured to output a diagnosis result indicating that care of the plant is required when the probability of any of the diseases output by the multi-value classification task exceeds a symptom threshold.
The diagnostic device of claim 2. the diagnosis output unit is configured to output, when a prediction result of the binary classification task corresponds to the wide-range image and a prediction result of the multi-value classification task is any one of the plurality of diseases, a diagnosis result that the plant needs care based on the wide-range image.
The diagnostic device of claim 2. A heat map generator configured to generate a heat map in which a prediction basis of the multi-class classification task is superimposed on the input image.
The diagnostic device according to any one of claims 1 to 4. a photographing position acquisition unit configured to acquire position information relating to the photographing position of the input image;
A location information generating unit;
one or more memories;
The memory stores map data including the photographing position of the input image,
the location information creation unit is configured to create location information by plotting the shooting location on the map data.
The diagnostic device according to any one of claims 1 to 4. a three-dimensional data generating unit configured to generate the three -dimensional data from a landscape image group including the plurality of landscape images;
a planar image generator configured to generate a planar image from the three-dimensional data;
The diagnostic device according to any one of claims 1 to 4, comprising: the input image is an image of a lawn laid on a golf course,
the ground surface image is an image of the ground surface covered with grass,
the wide-range image is a landscape image that captures at least one of a natural object or an artificial object in or around the lawn in addition to the ground surface covered with the lawn,
the natural object is a boundary line of a green, a bunker, a pond, or trees growing around the green;
The artifact is a cup hole or a pin.
A diagnostic device according to any one of claims 1 to 4. The multi-task learner includes a learning model using a convolutional neural network;
a loss function of the learning model is set so that the predictive performance of the multi-value classification task is higher than the predictive performance of the binary classification task.
The diagnostic device according to any one of claims 1 to 4. The plant is a ground cover plant that grows in clusters to cover the ground surface.
The diagnostic device according to any one of claims 1 to 4. the plant is grass;
The plurality of diseases include one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold;
The diagnostic device according to any one of claims 1 to 4. the ground surface image is an image in which the ground surface covered with the vegetation occupies an entire shooting range,
The wide-range image is a landscape image that includes at least one of trees growing on the ground surface and artificial objects including buildings in addition to the ground surface covered with vegetation.
A diagnostic device according to any one of claims 1 to 4. A method for diagnosing a plant's health, the method comprising:
acquiring an input image showing the vegetation covering the ground surface;
inputting the input image into a multitask learner;
the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks;
Estimating a capture range of the input image;
Including,
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image;
The estimating of the imaging range includes:
When the predicted result of the binary classification task is the wide-range image, extracting characteristic objects or feature points from the input image, comparing the extracted characteristic objects or feature points with three-dimensional data generated from a plurality of landscape images taken in the region from different viewpoints, and determining corresponding objects or feature points between the input image and the three-dimensional data, thereby estimating the shooting range;
When a prediction result of the binary classification task is the ground image, estimating a specified area including a shooting position of the input image as the shooting range without extracting the characteristic object or feature point from the input image.
Diagnostic methods. On one or more computers,
acquiring an input image showing vegetation covering a ground surface;
inputting the acquired input image into a multitask learning device;
the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks;
Estimating a capture range of the input image;
A program for plant diagnosis that executes the following:
The multi-task learner comprises:
a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface;
and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image,
Each of the ground surface image group and the wide range image group is
A plurality of asymptomatic images of the plant in a healthy state; and
a plurality of symptomatic images of the plant in an unhealthy state;
The unhealthy condition includes symptoms of multiple diseases;
The plurality of tasks include:
A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases;
a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image ;
The estimating of the imaging range includes:
When the predicted result of the binary classification task is the wide-range image, extracting characteristic objects or feature points from the input image, comparing the extracted characteristic objects or feature points with three-dimensional data generated from a plurality of landscape images taken in the region from different viewpoints, and determining corresponding objects or feature points between the input image and the three-dimensional data, thereby estimating the shooting range;
When a prediction result of the binary classification task is the ground image, estimating a specified area including a shooting position of the input image as the shooting range without extracting the characteristic object or feature point from the input image.
program.

Images