JP7627939B2 - Information processing device, information processing system, information processing method, control program, and recording medium - Google Patents

Description

The present invention relates to an information processing device, an information processing system, an information processing method, a control program, and a recording medium.

In the case of determining whether or not there are abnormalities in a plant strain in an image, an information processing device that extracts an area showing the target plant strain is known as prior art. Non-Patent Document 1 discloses a technology for processing plant images in which only areas of leaves and fruits are extracted from an image and areas that are not extracted are removed as background. Non-Patent Document 2 discloses a technology for recognizing leaf areas in an image that includes multiple leaves in a greenhouse.

Yatomi, 2019 Trends and issues in automatic plant disease diagnosis technology Journal of the Japanese Society for Neural Networks, Vol. 26, No. 4, pp. 123-134 Suwa et al., 2019 Intrinsic difficulties of plant diagnosis from wide-angle image, IDDD Proc.Big Data 2019 workshop, URL(https://arxiv.org/pdf/1910.11506.pdf)

However, the above-mentioned conventional technology has a problem in that the color of the plant in the image changes depending on the lighting environment when the image is captured, so the accuracy of extracting the leaves of the plant is easily affected by weather, etc.

One aspect of the present invention has been made in consideration of the above problems, and aims to improve the accuracy of extracting plant leaves in an image, regardless of the lighting environment when the image was captured.

In order to solve the above problem, an information processing device according to one aspect of the present invention includes a first extraction unit that receives an image including a plant stock as an input and extracts edge information indicating the contour of an object included in the image and brightness information indicating the brightness of the object, and a second extraction unit that receives the edge information and the brightness information as an input and extracts an area indicating plant leaves in the image using a first trained model that extracts an area indicating plant leaves.

In order to solve the above problem, an information processing device according to one aspect of the present invention includes an acquisition unit that acquires edge information indicating the outline of an object included in an image including a plant stock, brightness information indicating the brightness of the object, and information indicating an area indicating the plant leaves, and a learning unit that uses the edge information and the brightness information as input and trains a learning model that extracts areas indicating plant leaves by using a set of the edge information and the brightness information acquired by the acquisition unit and the information indicating the area indicating the plant leaves as training data.

In order to solve the above problems, an information processing method according to one aspect of the present invention is an information processing method executed by an information processing device, and includes a first extraction step of taking an image including a plant stock as input and extracting edge information indicating the contour of an object included in the image and brightness information indicating the brightness of the object, and a second extraction step of taking the edge information and the brightness information as input and extracting an area indicating plant leaves in the image using a trained model that extracts an area indicating plant leaves.

According to one aspect of the present invention, it is possible to improve the accuracy of extracting plant leaves in an image, regardless of the lighting environment when the image was captured.

1 is a functional block diagram of an information processing system according to a first embodiment. 1 is a flowchart showing a process flow according to a first embodiment; FIG. 1 shows the distribution of pixel values corresponding to objects contained in an image in Lab color space showing a field of plant lines; The distribution of the number of pixels of pixel values corresponding to plant stocks and soil is shown on the ab plane of the Lab color space. 1 is an image showing a field of plant strains. FIG. 11 is a functional block diagram of an information processing system according to a second embodiment. 1 is an image showing a field of plant strains. 10 is a flowchart showing a process flow according to a second embodiment. 1 is an image showing a field of plant strains. FIG. 11 is a functional block diagram of an information processing system according to a third embodiment. 1 shows information relating to a first extraction unit. 10 is a flowchart showing a process flow according to a third embodiment;

[Embodiment 1]
An embodiment of the present invention will be described in detail below. In this embodiment, an example of an information processing device capable of classifying a plant stock portion and a soil portion with high accuracy even when an image of the plant stock captured on a sunny day is input will be described.

[1. Example of information processing system configuration]
An information processing system 1 according to this embodiment will be described. FIG. 1 is a functional block diagram of the information processing system 1 according to this embodiment. As shown in FIG. 1, the information processing system 1 includes an information processing device 10, a display device 38, and a mobile object 42. Note that each component included in the information processing system 1 is not limited to being singular, and each may be multiple. Also, the function of a single component included in the information processing system 1 may be realized by multiple other components, and the functions of multiple components included in the information processing system 1 may be realized by a single other component.

The information processing device 10 includes a control unit 12, a storage unit 30, and a communication unit 34. The control unit 12 is a control device that controls the entire information processing device 10, and includes an acquisition unit 14, a preprocessing unit 16, an extraction unit 18, a classification unit 20, a learning unit 22, and a presentation unit 24.

The acquisition unit 14 acquires an image including a plant strain. The image may be one that has been converted by the preprocessing unit 16 into an image in Lab color space. Here, an image in Lab color space is an image in which pixel values related to color are defined by the value of L corresponding to lightness and the values of a and b corresponding to complementary colors. The acquisition unit 14 also acquires information input from a mobile object 42, etc., and information related to the presence or absence of abnormalities in the plant strain included in the image. Here, the information related to the presence or absence of abnormalities may include, for example, information indicating the presence or absence of abnormalities, or information indicating what type of abnormality exists.

The pre-processing unit 16 performs a predetermined process on the image acquired by the acquisition unit 14. For example, the pre-processing unit 16 performs a process of converting the image into an image in the Lab color space, and a process of removing noise by applying a Gaussian filter to the image. Note that the configuration may also be such that the moving object 42 captures a moving image, and the pre-processing unit 16 extracts and uses a still image from the moving image.

The extraction unit 18 extracts an area in the image that represents a plant stock by comparing a value corresponding to both the a-value and the b-value in the pixel values of the image acquired by the acquisition unit 14 with a predetermined threshold. In other words, the extraction unit 18 assigns a threshold to a feature amount related to both the a-value and the b-value in the pixel values of the image, and determines whether the target pixel represents a plant stock or an object other than a plant stock, such as soil, depending on the comparison result with the threshold.

As will be described later with reference to FIG. 4, for example, the extraction unit 18 extracts an area indicating a plant strain in accordance with the positional relationship between the position to which the pixel value of the target pixel corresponds and the line or curve indicating the threshold value on the a-b plane in the Lab color space. From another perspective, the extraction unit 18 determines whether the target pixel indicates a plant strain or soil depending on which area defined by the line or curve the pixel value corresponds to.

The classification unit 20 receives an image including a region indicating a plant strain extracted by the extraction unit 18 as input, and classifies the plant strain as healthy or abnormal using a trained model that outputs a determination result regarding the presence or absence of an abnormality in the plant strain. Here, a trained model means a learning model that has been trained a sufficient number of times.

The learning unit 22 uses an image including a plant stock as input, and trains a learning model that outputs a determination result regarding the presence or absence of abnormalities in the plant stock, using a pair of an image including the plant stock and information regarding the presence or absence of abnormalities in the plant stock as teacher data. The learning unit 22 executes a process of training the learning model, for example, by performing the following procedure a sufficient number of times. The trained model trained by the following process is an example of a second trained model in the present disclosure. Note that when the second trained model targets plant leaves, "plant stock" in (1) below is read as "plant leaves".

(1) The acquisition unit 14 acquires training data including a plurality of pairs of images including a plant strain and information regarding the presence or absence of abnormalities in the plant strain included in the images from the storage unit 30, etc.

(2) The learning unit 22 inputs the image into the learning model.

(3) The learning unit 22 updates the values of the parameter set that defines the learning model in a gradient direction that reduces the loss function defined by the information and the output value of the learning model.

An example of the first trained model in this disclosure will be described later in embodiment 3.

The classification method performed by the classification unit 20 and the machine learning method performed by the learning unit 22 are not limited to a specific method, and may be, for example, a method using a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). Either a classification or regression model may be used. When using a neural network such as a CNN or an RNN, the input data may be processed in advance for input to the neural network. In addition to two-dimensional or multidimensional arrangement of data, such processing may include, for example, various data augmentation methods, adjustments of brightness, color tone, image quality, and angle of the object, object detection for extracting the object area, and segmentation.

When using CNN, one or more layers included in the neural network may be a convolutional layer that performs convolutional operations, and a filter operation (product-sum operation) may be performed on the input data input to the layer. When performing the filter operation, padding or other processing may be used in combination, or an appropriately set stride width may be adopted.

Other configurations may also be used, such as the following machine learning techniques or a combination of them:

・Support Vector Machine (SVM)
・Clustering
・Inductive Logic Programming (ILP)
・Genetic Algorithm (GP: Genetic Programming)
・Bayesian Network (BN)
・Autoencoder
The presentation unit 24 presents information indicating the position of the plant strain classified as an abnormal strain by the classification unit 20. For example, the presentation unit 24 displays an image indicating the position of the plant strain on the screen of the display device 38. The presentation unit 24 may be configured to present the information indicating the position of the plant strain by sound. In this configuration, the presentation unit 24 may present the information by, for example, outputting sound indicating the coordinates of the plant strain from a speaker (not shown) provided in the information processing device 10.

In one embodiment, the presentation unit 24 may present information indicating the location of a plant strain classified as an abnormal strain in real time in parallel with the movement of the mobile object 42. For example, in the above configuration, the presentation unit 24 may be configured to sound a notification sound while displaying information indicating the location of the abnormal strain on the screen of the display device 38 in response to a plant strain being classified as an abnormal strain. In this way, even if the user is not looking at the screen, the user can easily understand that an abnormal strain has been found near the mobile object 42 when the notification sound is sounded, and can check the screen.

The memory unit 30 is a storage device that stores various information, such as a parameter set that defines a learning model. The communication unit 34 performs communication processing with external devices such as a mobile object 42.

The display device 38 is a display that displays moving images, text, etc. For example, the display device 38 displays information instructed by the presentation unit 24 on the screen.

The moving body 42 is realized as a flying object such as a vehicle or a drone, and includes an imaging unit 44 and a communication unit 46. The imaging unit 44 is a camera that captures images of the surroundings of the moving body 42. The communication unit 46 performs communication processing with an external device such as an information processing device 10. Note that the present disclosure also includes an embodiment in which the moving body 42 is realized by a person or an animal carrying a camera. The moving body 42 may also be capable of carrying a person or an information processing device 10. The moving body 42 transmits an image captured by the imaging unit 44 and information indicating the position of the moving body 42 when the image was captured to the information processing device 10, for example, by wireless communication. The configuration in which the moving body 42 acquires its own position may be realized by, for example, mounting a position information terminal using RTK (Real-Time Kinematic) on the moving body 42.

[2. Processing example of information processing system]
Next, an example of a process in which the information processing system 1 presents the position of an abnormal plant will be described. In this example, the information processing device 10 processes an image of a field in which a plant plant is planted as an image including the plant plant. This may also be the case in the following embodiments. Fig. 2 is a flowchart showing the flow of the process in this example.

In step S101, while the mobile body 42 is moving over the field of plant stocks, the imaging unit 44 captures one or more images including the plant stocks in the vicinity of the mobile body 42.

In step S102, the moving object 42 transmits the image captured by the imaging unit 44 and information indicating the position of the moving object 42 when the image was captured to the information processing device 10. The acquisition unit 14 of the information processing device 10 acquires each piece of information transmitted from the moving object 42 via the communication unit 34. The information processing device 10 associates the image with the information indicating the position of the moving object 42 and stores them in the storage unit 30. The process of step S102 is not limited to a configuration in which it is started after the imaging of the entire target field is completed, and may be a configuration in which it is executed whenever the imaging unit 44 captures an image.

In step S103, the preprocessing unit 16 converts the image acquired by the acquisition unit 14 into an image in the Lab color space, and performs processing to remove noise in the image, for example, by applying a Gaussian filter. The acquisition unit 14 also acquires the image that has been subjected to the processing.

In step S104, the extraction unit 18 extracts an area representing a plant strain from the image acquired by the acquisition unit 14 and processed by the pre-processing unit 16.

Figure 3 shows an image in Lab color space that illustrates the distribution of pixel values corresponding to objects contained in an image showing a field of plant stocks. In Figure 3, region 50 illustrates the union of the distribution of pixel values corresponding to plant stocks imaged on a cloudy day and the distribution of pixel values corresponding to plant stocks imaged on a sunny day. Region 52 illustrates the distribution of pixel values corresponding to soil imaged on a cloudy day. Region 54 illustrates the distribution of pixel values corresponding to soil imaged on a sunny day.

The explanatory diagram 58 in FIG. 4 shows the distribution of the number of pixels of pixel values corresponding to plant stocks in the a-b plane when the horizontal axis is the a-axis and the vertical axis is the b-axis in the Lab color space. Here, the distribution of the number of pixels corresponds to the number of points that overlap in the L direction at each coordinate of the a-b plane when the graph in FIG. 3 is viewed from directly above. The explanatory diagram 60 shows the distribution of the number of pixels of pixel values corresponding to soil captured on a cloudy day in the a-b plane. The explanatory diagram 62 shows the distribution of the number of pixels of pixel values corresponding to soil captured on a sunny day in the a-b plane. The soil also includes soil parts that are in the shadow of plant stocks, etc. In each explanatory diagram, the distribution is expressed in exponential terms, and since the number of pixels in the peripheral parts of the distribution is very small compared to the central part of the distribution, there is almost no impact even if it is not taken into consideration. In each explanatory diagram, the straight line 66 is a straight line indicating a threshold value that is compared with only the a value, and the straight line 68 is a straight line indicating a threshold value that is compared with values corresponding to both the a value and the b value.

As shown in diagrams 58 and 60, in an image captured on a cloudy day, it is possible to appropriately classify pixel values corresponding to plant stocks and pixel values corresponding to soil using line 66 as the boundary. However, as shown in diagrams 58 and 62, in an image captured on a sunny day, it is not possible to appropriately classify pixel values corresponding to plant stocks and pixel values corresponding to soil using line 66 as the boundary.

Therefore, by using the straight line 68 as the boundary, it becomes possible to appropriately classify pixel values corresponding to plant stocks and pixel values corresponding to soil in an image captured on a sunny day. That is, in the example shown in each diagram in FIG. 4, if the pixel value of the pixel to be processed is located in the upper left area of the straight line 68 on the a-b plane, the extraction unit 18 determines that the pixel corresponds to a plant stock. Also, if the pixel value of the pixel to be processed is located in the lower right area of the straight line 68 on the a-b plane, the extraction unit 18 determines that the pixel corresponds to an object other than a plant stock, such as soil.

The value of b/a, which indicates the slope of the line 68, is not limited to a specific value and may be 1.15, 1.5, or the like. From another perspective, generalizing the line 68 is equivalent to giving a threshold value to the value of (b-β)/a, where β is any real number. At least one of the threshold values may be represented by a curve. At least one of the lines or curves indicating the threshold values in the a-b plane of the Lab color space may have a b-intercept value different from 0, like the line 68.

The equation of the line or curve indicating the threshold value may be configured to be updateable by the learning unit 22 through machine learning using a pair of an image including soil and plant stocks and information indicating the area of the plant stocks as training data.

Furthermore, following or in parallel with the process of extracting the area showing the plant stock in the image, the extraction unit 18 performs a process of masking the background part of the area. That is, the extraction unit 18 performs a process of deleting color information or filling in the area other than the area showing the plant stock in the image with a single color such as black.

Image 72 in FIG. 5 is an image of a field of plant stocks on a sunny day, captured by a drone, which is a mobile object 42. Image 74 shows the detection result of a plant stock when pixel values corresponding to the plant stocks and pixel values corresponding to the soil are classified using a threshold value that is compared only with the a value. In image 74, the soil part on the right side is erroneously detected as a plant stock. Image 76 shows the detection result of a plant stock when pixel values corresponding to the plant stocks and pixel values corresponding to the soil are classified by comparing values corresponding to both the a value and the b value in image 72 with a predetermined threshold value. In image 76, the area corresponding to the upper left of the above-mentioned straight line 68 is extracted as the area indicating the plant stock. The detection result of the plant stock shown in image 76 has higher detection accuracy than the detection result shown in image 74.

In step S105, the extraction unit 18 divides the image of the field into areas for each plant strain so as to include an area showing a single plant strain. The control unit 12 also calculates the position of each plant strain in the field by referring to the image of the field and information showing the position of the mobile object 42 at the time the image was captured.

In step S106, the classification unit 20 inputs an image including an area showing a single plant strain, and classifies the plant strain as healthy or abnormal using a trained model that outputs a determination result regarding the presence or absence of abnormalities in the plant strain.

In step S107, the control unit 12 performs branching for the target plant strain depending on whether the classification result by the classification unit 20 is a healthy strain or an abnormal strain. If the plant strain is a healthy strain, the process of step 109 is subsequently executed, and if the plant strain is an abnormal strain, the process of step S108 is subsequently executed.

In step S108, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the location of the plant strain that the classification unit 20 has classified as an abnormal strain.

In step S109, the control unit 12 determines whether the processing of steps S106 to S108 has been completed for all target plant strains. If the control unit 12 determines that the processing has been completed for all plant strains, the processing based on the flowchart in FIG. 2 ends. If the control unit 12 determines that the processing has not been completed for any plant strain, the processing from S106 is executed for the target plant strain.

As described above, the information processing method executed by the information processing device 10 includes an acquisition step and an extraction step. In the acquisition step, an image in Lab color space that includes a plant stock is acquired. In the extraction step, a region showing the plant stock in the image is extracted by comparing a value corresponding to both the a-value and the b-value in the pixel values of the image acquired in the acquisition step with a predetermined threshold value.

According to the configuration of this example, by comparing the values corresponding to both the a-value and the b-value with a predetermined threshold value, it is possible to improve the accuracy of extracting plant stocks in an image regardless of the lighting environment when the image was captured. In addition, since the location of abnormal stocks can be grasped without an expert having to visually check, it is possible to reduce the labor required for, for example, removing abnormal stocks.

[Embodiment 2]
The second embodiment of the present invention will be described below. For convenience of explanation, the same reference numerals are attached to components having the same functions as the components described in the above embodiment, and the overlapping explanations will not be repeated. The same applies to the following embodiments. In this embodiment, an example of an information processing system capable of detecting a plant strain having a predetermined difference compared to other plant strains will be described. Here, the plant strain having a predetermined difference may include a plant strain that is relatively small in size, a wilted plant strain, or a plant strain with a different color compared to nearby plant strains. In addition, the above-mentioned nearby plant strains may include adjacent plant strains, or one or more plant strains among a predetermined number of plant strains within a predetermined distance or adjacent to a certain plant strain.

[1. Example of information processing system configuration]
An information processing system 1a according to the present embodiment will be described. Fig. 6 is a functional block diagram of an information processing device 10a according to the present embodiment. The information processing device 10a is configured to include a detection unit 21 instead of the classification unit 20, as compared with the information processing device 10 shown in Fig. 1.

The detection unit 21 receives an image including multiple plant strains as input, and detects a plant strain in a target image that has a predetermined difference compared to nearby plant strains using a trained model that detects plant strains in the image that have a predetermined difference compared to nearby plant strains. Here, the trained model is a trained model trained using a set of an image including multiple plant strains and information regarding a difference between any one of the multiple plant strains and other plant strains as training data. Here, the information regarding the difference may include, for example, information indicating the presence or absence of the difference, or information indicating what type of difference there is.

The acquisition unit 14 according to this embodiment acquires an image including multiple plant strains. The acquisition unit 14 also acquires information regarding the difference between one of the multiple plant strains included in the image and other plant strains.

Here, the information on the difference between a certain plant strain and another plant strain includes:
This may include information regarding the difference between the size of one plant strain and another, information regarding the difference between the color of one plant strain and another, information regarding the difference between the shape of one plant strain and another, etc.

The learning unit 22 according to this embodiment also learns a learning model that receives an image containing multiple plant strains as input and detects plant strains that have a predetermined difference compared to nearby plant strains in the image, using a pair of the image acquired by the acquisition unit 14 and information about the difference as training data.

The detection method performed by the detection unit 21 and the machine learning method performed by the learning unit 22 may be, for example, a method using Faster R-CNN (Regions with Convolutional Neural Networks).

Each image in FIG. 7 shows a field of plant stocks. The portion of the image 80 enclosed in a frame 82 corresponds to an image that would be training data for a learning model in a configuration other than that of this embodiment. As shown in frame 82, the image contains a single plant stock.

On the other hand, the image serving as the training data in this embodiment includes multiple plant strains. As an example, the image portion surrounded by a frame 86 in image 84 corresponds to the image serving as the training data for the learning model in this embodiment. Also, image 88 in FIG. 7 shows an example of a search result by the detection unit 21.

In addition, the other components in the information processing system 1a have the function of performing the processing described below.

[2. Processing example of information processing system]
Next, an example of a process in which the information processing system 1a presents the position of a plant strain that has a predetermined difference compared to other plant strains will be described. Fig. 8 is a flowchart showing the flow of the process in this example.

In step S201, while the mobile object 42 is moving over the field of plant stocks, the imaging unit 44 captures one or more images including multiple plant stocks in the vicinity of the mobile object 42.

In step S102, the same processing as in embodiment 1 is performed.

In step S203, the preprocessing unit 16 performs a process to remove noise in the image acquired by the acquisition unit 14, for example by applying a Gaussian filter. The acquisition unit 14 also acquires the image that has been subjected to the process.

In step S204, the extraction unit 18 performs a process of extracting a region showing a plant strain in the image acquired by the acquisition unit 14 and processed by the pre-processing unit 16. In addition, following or in parallel with the process of extracting the region showing a plant strain in the image, the extraction unit 18 performs a process of masking the background portion of the region.

In step S205, the detection unit 21 uses the trained model described above to detect plant strains that have a predetermined difference compared to nearby plant strains in the image processed by the extraction unit 18.

Images 94 and 98 in FIG. 9 are examples of images that include multiple plant strains. Images 96 and 100 respectively show the detection results when images 94 and 98 are input into a trained model. As exemplified by image 96 and image 88 in FIG. 7, when multiple plant strains are adjacent to a certain plant strain, some of the detection positions may overlap. Furthermore, when multiple plant strains with a specified difference are adjacent to each other, a configuration may be used to detect a range that includes the multiple plant strains and plant strains that do not have the specified difference.

In step S206, the control unit 12a refers to the image processed by the extraction unit 18 and information indicating the position of the moving body 42 at the time the image was captured, and calculates the positions of the plant stocks that have a predetermined difference.

In step S207, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the location of the plant stock that has a predetermined difference.

In step S208, the control unit 12a determines whether the processing of steps S206 and S207 has been completed for all of the detected adjacent plant strains. If the control unit 12a determines that the processing has been completed for all of the plant strains, the processing based on the flowchart in FIG. 8 ends. If the control unit 12a determines that the processing has not been completed for any of the plant strains, the processing from S206 is executed for the target plant strain.

As described above, the information processing method executed by the information processing device 10a includes a detection step and the like. In the detection step, an image including multiple plant strains is input, and a trained model for detecting plant strains that have a predetermined difference in the image compared to nearby plant strains is used to detect plant strains that have the difference in the target image compared to nearby plant strains. In addition, the trained model is trained using a combination of an image including multiple plant strains and information regarding the difference between any one of the multiple plant strains and other plant strains as training data.

The process of training the learning model according to this embodiment includes an acquisition step and a learning step. In the acquisition step, an image including multiple plant strains and information regarding the differences between any of the multiple plant strains and other plant strains are acquired. In the learning step, an image including multiple plant strains is input, and a learning model that detects plant strains that have a predetermined difference in the image compared with nearby plant strains is trained using the pair of the image acquired in the acquisition step and the information regarding the difference as training data.

According to the configuration of this example, by using images containing multiple plant strains as training data, it becomes possible to detect plant strains that have a certain difference compared to other plant strains.

In addition, in steps S203 and S204 described above, similar to embodiment 1, the extraction unit 18 may extract an area indicating a plant stock by comparing a value corresponding to both the a-value and the b-value in the pixel values of the image converted by the preprocessing unit 16 into an image in the Lab color space with a predetermined threshold value.

[Embodiment 3]
The third embodiment of the present invention will be described below. In this embodiment, an example of an information processing system capable of extracting a region showing a plant leaf using edge information and brightness information extracted from an image including a plant stock will be described. Here, edge information means information showing the outline of an object included in an image. Furthermore, brightness information means information showing the brightness of an object included in an image. The brightness information corresponds to a pixel value showing the brightness of a pixel showing an object included in an image. For example, the brightness information corresponds to the value of L in the above-mentioned Lab color space, or the value of Y in the YCbCr color space represented by the luminance signal Y and the color difference signals Cb and Cr. Furthermore, as will be described later in FIG. 11, the edge information and brightness information can be expressed in the form of an image.

[1. Example of information processing system configuration]
An information processing system 1b according to the present embodiment will be described. Fig. 10 is a functional block diagram of an information processing device 10b according to the present embodiment. The information processing device 10b is configured such that the extraction unit 18 includes a first extraction unit 23 and a second extraction unit 25, as compared with the information processing device 10 shown in Fig. 1.

The first extraction unit 23 receives an image including a plant stock as input and extracts edge information and brightness information of objects included in the image. The first extraction unit 23 may be configured to extract edge information using, for example, a second-order differential filter such as a Laplacian filter, a Sobel filter, or a Canny filter as an edge filter.

The second extraction unit 25 receives edge information and brightness information as input, and extracts areas representing plant leaves in the image using a first trained model that extracts areas representing plant leaves.

Each image in FIG. 11 shows information related to the first extraction unit 23. Image 104 shows an image including a plant stock, which is input to the first extraction unit 23. Images 106 and 108 show the brightness information and edge information, respectively, extracted by the first extraction unit 23. Image 110 shows a composite image in which image 106 showing the brightness information and image 108 showing the edge information are combined into a single image.

The first trained model may be configured to receive a synthetic image such as the example shown in image 110. The process of generating the synthetic image may be performed by the first extraction unit 23 or the second extraction unit 25.

The acquisition unit 14 according to this embodiment also acquires, from the storage unit 30, teacher data including multiple pairs of edge information indicating the contour of an object included in an image including plant leaves, brightness information indicating the brightness of the object, and information indicating an area showing the plant leaves.

The learning unit 22 according to this embodiment receives edge information and brightness information as input, and performs a process of learning a learning model that extracts areas showing plant leaves, using a set of edge information and brightness information acquired by the acquisition unit 14 and information showing the areas showing plant leaves as training data. The trained model trained by the process is an example of a first trained model in the present disclosure.

In addition, the other components in information processing system 1b have the function of performing the processing described below.

2. Examples of processing by the information processing system
Next, an example of a process in which the information processing system 1b presents the position of an abnormal leaf will be described. FIG 12 is a flowchart showing the flow of the process in this example.

In steps S101 and S102, the same processing as in embodiment 1 is performed.

In steps S203 and S204, the same processing as in embodiment 2 is performed.

In step S304, the first extraction unit 23 extracts edge information and brightness information of objects contained in the image acquired by the acquisition unit 14 and processed by the pre-processing unit 16.

In step S305, the second extraction unit 25 inputs the edge information and brightness information extracted by the first extraction unit 23 into the first trained model, and extracts an area showing plant leaves in the image that has been processed by the pre-processing unit 16.

In step S306, the extraction unit 18 divides the image of the field into areas for each plant leaf so as to include an area showing a single plant leaf. The control unit 12b also calculates the position of each plant leaf in the field by referring to the image of the field and information showing the position of the moving body 42 when the image was captured.

In step S307, the classification unit 20 inputs an image including a region showing a plant leaf extracted by the second extraction unit 25, the image including a region showing a single plant leaf, into a second trained model targeting plant leaves, thereby classifying the plant leaf as being healthy or abnormal.

In step S308, the control unit 12b performs branching for the target plant leaf depending on whether the classification result by the classification unit 20 is a healthy leaf or an abnormal leaf. If the plant leaf is a healthy leaf, the process of step S310 is subsequently executed, and if the plant leaf is an abnormal leaf, the process of step S309 is subsequently executed.

In step S309, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the position of the plant leaf that the classification unit 20 has classified as an abnormal leaf.

In step S310, the control unit 12b determines whether the processing of steps S307 to S309 has been completed for all target plant leaves. If the control unit 12b determines that the processing has been completed for all plant leaves, the processing based on the flowchart in FIG. 12 ends. If the control unit 12b determines that the processing has not been completed for any plant leaves, the processing from S307 is executed for the target plant leaves.

As described above, the information processing method executed by the information processing device 10b includes a first extraction step and a second extraction step. In the first extraction step, an image including a plant stock is input, and edge information indicating the contour of an object included in the image and brightness information indicating the brightness of the object are extracted. In addition, in the second extraction step, edge information and brightness information are input, and a trained model that extracts areas showing plant leaves is used to extract areas showing plant leaves in the image.

According to the configuration of this example, by extracting areas showing plant stocks and leaves using the extracted edge information and brightness information, it is possible to improve the accuracy of extracting plant stocks and leaves in an image regardless of the lighting environment when the image was captured.

The present disclosure also includes a configuration in which the extraction unit 18b does not include the first extraction unit 23, and the second extraction unit 25 acquires edge information and brightness information extracted by an external device different from the information processing system 1b and inputs the information to the first trained model to extract a region showing plant leaves.

[Modifications of the First to Third Embodiments]
The configurations described above in the first to third embodiments are not limited to targeting plant stocks and leaves. In the first and third embodiments, for example, trash on the road, animals, or traffic signs may be the targets. In addition, for example, a person carrying a terminal device having a camera and a GPS function may act as the mobile body 42, and the location of the trash or the like may be displayed on a map in real time on a display device 38 in a remote location. In the second embodiment, for example, detection may be performed on industrial products and other products.

[Software implementation example]
The functions of the information processing device (10, 10a, 10b) (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control unit (12, 12a, 12b)).

In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing each of the functions described in each of the above embodiments.

The program may be recorded on one or more computer-readable recording media, not on a temporary basis. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

In addition, some or all of the functions of each of the above control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the above control blocks is formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

The processes described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may run on the control device, or on another device (such as an edge computer or a cloud server).

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments.

Reference Signs List 1, 1a, 1b Information processing system 10, 10a, 10b Information processing device 12, 12a, 12b Control unit 14 Acquisition unit 16 Preprocessing unit 18 Extraction unit 20 Classification unit 21 Detection unit 22 Learning unit 23 First extraction unit 24 Presentation unit 25 Second extraction unit 30 Storage unit 34, 46 Communication unit 38 Display device 42 Mobile object 44 Imaging unit

Claims (8)

a first extraction unit that receives an image including a plant strain as input and extracts edge information indicating an outline of an object included in the image and brightness information indicating a brightness of the object;
a second extraction unit that extracts a region indicating a plant leaf from the image using a first trained model that receives the edge information and the brightness information as input and extracts a region indicating a plant leaf ;
a classification unit that classifies the plant leaf as a healthy leaf or an abnormal leaf using a second trained model that receives an image including a region indicating the plant leaf extracted by the second extraction unit and outputs a determination result regarding the presence or absence of an abnormality in the plant leaf;
a presentation unit that presents information indicating the positions of plant leaves that have been classified as abnormal leaves by the classification unit;
An information processing device comprising: The information processing device according to claim 1, further comprising a learning unit that uses a pair of an image including a plant leaf and information regarding the presence or absence of abnormalities in the plant leaf as teacher data to train a learning model that receives an image including a plant leaf and outputs a determination result regarding the presence or absence of abnormalities in the plant leaf . The first extraction unit is
3. The information processing apparatus according to claim 1, wherein the edge information is extracted using a second-order differential filter as an edge filter. an acquisition unit that acquires edge information indicating an outline of an object included in an image including a plant leaf, brightness information indicating the brightness of the object, and information indicating an area indicating the plant leaf;
The information processing device according to claim 1, further comprising a learning unit that uses the edge information and the brightness information as input and learns a learning model that extracts areas representing plant leaves by using a set of the edge information and the brightness information acquired by the acquisition unit and information indicating the areas representing plant leaves as teacher data. A moving object including an imaging unit;
The information processing device according to claim 1 , wherein the image captured by the imaging unit is input to the first extraction unit.
An information processing system comprising : An information processing method executed by an information processing device,
a first extraction step of extracting edge information indicating an outline of an object included in the image and brightness information indicating a brightness of the object using an image including a plant strain as an input;
a second extraction step of extracting a region indicating a plant leaf from the image using a first trained model that receives the edge information and the brightness information as input and extracts a region indicating a plant leaf ;
a classification step of classifying the plant leaf as a healthy leaf or an abnormal leaf using a second trained model that receives an input of an image including a region indicating the plant leaf extracted in the second extraction step and outputs a determination result regarding the presence or absence of an abnormality in the plant leaf;
a display step of displaying information indicating the positions of the plant leaves classified as abnormal leaves in the classification step;
13. An information processing method comprising: A control program for causing a computer to function as the information processing device according to claim 1 , the control program causing a computer to function as the first extraction unit , the second extraction unit , the classification unit, and the presentation unit . A computer-readable recording medium having the control program according to claim 7 recorded thereon.

Images