JP7797003B2 - Class inference system, tree species map generation system, class inference method, and tree species map generation method - Google Patents

Description

The present invention relates to technology for classifying objects such as trees in a forest.

As described in Non-Patent Document 1, methods have been proposed in the past to acquire forest information using specialized hardware such as hyperspectral cameras, multispectral cameras, or LiDAR (Light Detection and Ranging) sensors, and to identify the types of trees growing in the forest based on the acquired information.

However, this hardware is expensive, so this method increases costs. Therefore, a method has been proposed that uses AI (Artificial Intelligence) to infer tree species.

For example, according to the method described in Patent Document 1, the contours of each of multiple objects present in a test area are identified based on a photograph of the test area taken from above or a first distribution that is the distribution of heights at each point in the test area and a second distribution that is the distribution of gradients at each point. For each identified contour, a label corresponding to the type of object present in the contour from among multiple predetermined types is assigned, and training data is generated for each contour, showing partial images of the part of the photograph surrounded by each contour and the labels assigned to each contour. A trained model is generated using the partial images and labels shown in each training data as input and correct answer, respectively. The type of tree to be inferred is then inferred based on the generated trained model.

Japanese Patent Application Laid-Open No. 2020-91640

"Review of studies on tree species classification from remotely sensed data." Remote Sensing of Environment Sensing of Environment Sensing of Environment Sensing of Environment Sensing of Environment , 186 , 64 -87., Fassnacht, F. E., Latifi, H., Sterenczak, K., Modzelewska, A., Lefsky, M., Waser, L. T., ... & Ghosh, A. 2016

However, conventional AI methods such as those described in Patent Document 1 are unable to distinguish between classes (types) that have not been trained. Therefore, an existing trained model generated based on training data acquired under a certain type of vegetation cannot be used in areas with different vegetation.

One possible solution would be to generate a new trained model based on the vegetation of the area, but this would require acquiring a huge amount of new training data. Furthermore, a high-performance computer would be required for the machine learning calculations.

In light of these problems, the present invention aims to make it easier than ever to build AI that can infer classes such as trees.

A class inference system according to one embodiment of the present invention comprises a first acquisition means for acquiring a first feature vector representing the features of an image of the crown of a tree belonging to each of a plurality of classes that grows in a specific region by inputting the image into a trained model; a second acquisition means for acquiring a second feature vector representing the features of an image of the crown of an inference target tree that grows in the specific region and is the subject of inference by inputting the image into the trained model; and an inference means for inferring the class to which the inference target tree belongs among the plurality of classes based on the first feature vector of each of the plurality of classes and the second feature vector of the inference target tree, wherein the trained model is a network that calculates the feature vector of the input image by convolution processing within a convolutional neural network (CNN) generated by deep learning using the class and the image of the crown of each of a plurality of trees as the objective variable and the explanatory variable, respectively.
A tree species map generation system according to one embodiment of the present invention is a tree species map generation system that generates a tree species map of an area having multiple regions and in which trees belonging to each of multiple classes grow, and includes a first acquisition means that acquires, for each of the multiple regions, a first feature vector representing the characteristics of an image of the crown of a tree belonging to each of the multiple classes that grows in the area by inputting the image into a trained model; a second acquisition means that acquires a second feature vector representing the characteristics of an image of the crown of each of multiple inference target trees that grow in the area by inputting the image into the trained model; an inference means that infers the class to which each of the multiple inference target trees belongs based on the first feature vector of each of the multiple classes in the area in which the inference target tree grows among the multiple regions and the second feature vector of the inference target tree; and a map generation means that generates the tree species map based on location information of each of the multiple inference target trees and the inference results by the inference means.

This invention makes it easier than ever to build AI that can infer classes such as trees.

FIG. 1 is a diagram illustrating an example of the overall configuration of a tree classification system. FIG. 1 illustrates an example of a hardware configuration of a computer. FIG. 2 is a diagram illustrating an example of a functional configuration of a computer. FIG. 1 illustrates an example of a tree classification network. 1A and 1B are diagrams showing examples of an aerial photograph and a canopy map, respectively. FIG. 10 is a diagram illustrating an example of object data. FIG. 10 is a diagram illustrating an example of an inference method. FIG. 10 is a diagram illustrating an example of the distribution of feature points in a 2640-dimensional space. FIG. 10 is a diagram illustrating an example of a tree classification map. 10 is a flowchart illustrating an example of the overall processing flow of a tree classification program. FIG. 10 is a diagram showing an example of a correct/incorrect result. FIG. 1 shows an example of a method for generalizing a feature extraction network and creating a national tree map.

[Overall structure]
Fig. 1 is a diagram showing an example of the overall configuration of a tree classification system 3. Fig. 2 is a diagram showing an example of the hardware configuration of a computer 1. Fig. 3 is a diagram showing an example of the functional configuration of the computer 1.

As shown in Figure 1, the tree classification system 3 is composed of a computer 1, a drone 2, and other components. The tree classification system 3 provides a service that uses AI (artificial intelligence) to classify trees that appear in aerial photographs.

Drone 2 is a UAV (Unmanned Aerial Vehicle) equipped with a digital camera, and is used to photograph the forest from above to obtain color photographs that will serve as the basis for learning data. Drone 2 can be a commercially available model; for example, a DJI PHANTOM 4 is used.

As shown in FIG. 2, the computer 1 is composed of a processor 10, RAM (Random Access Memory) 11, ROM (Read Only Memory) 12, auxiliary storage device 13, network adapter 14, keyboard 15, pointing device 16, input/output board 17, touch panel display 18, and audio output unit 19.

In addition to the operating system, various other programs are installed in the ROM 12 or auxiliary storage device 13. In particular, in this embodiment, the tree classification program 4 (see Figure 3) is installed. The auxiliary storage device 13 may be an SSD (Solid State Drive) or a hard disk.

RAM 11 is the main memory of computer 1. In addition to the operating system, programs such as tree classification program 4 are loaded into RAM 11.

The processor 10 executes programs loaded into the RAM 11. The processor 10 may be a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit).

The network adapter 14 is a device for communicating with other devices such as the drone 2 using protocols such as TCP/IP (Transmission Control Protocol/Internet Protocol).

The keyboard 15 and pointing device 16 are input devices that allow the operator to input commands, data, etc.

The input/output board 17 communicates with the drone 2 via wired or wireless communication. For example, an input/output board compliant with USB (Universal Serial Bus) or Bluetooth is used as the input/output board 17.

The touch panel display 18 displays a screen for inputting commands or data, or a map generated by the processor 10.

The audio output unit 19 is composed of an audio board and speaker, and outputs sounds such as warning sounds.

The tree classification program 4 is a computer program for realizing the canopy map generation unit 41, label assignment unit 42, feature vector calculation unit 43, training data storage unit 44, canopy detection unit 45, feature calculation unit 46, class inference unit 47, tree classification map generation unit 48, and tree classification map output unit 49 shown in Figure 3. The tree classification program 4 can generate a classifier for inferring the class (type) of a tree, and infer the class of a target tree based on this classifier.

Below, we will explain the processing of the drone 2 and each part shown in Figure 3, broadly dividing it into a learning phase and an inference phase, using an example in which the tree classification system 3 is tuned and used to classify trees in a forest 80 in a certain region (site).

[Learning Phase]
(1) Network and Data Fig. 4 is a diagram showing an example of a tree classification network 5. Fig. 5 is a diagram showing examples of an aerial photograph 60 and a canopy map 61. Fig. 6 is a diagram showing examples of object data 62.

A feature extraction network 51 is prepared in advance in the computer 1. The feature extraction network 51 is part of the tree classification network 5 shown in Figure 4.

The tree classification network 5 is a convolutional neural network (CNN) that infers which of N classes a target tree belongs to based on an image of the tree's canopy, and is primarily composed of a feature extraction network 51 and a probability calculation network 52.

The feature extraction network 51 is a network having an input layer, a convolutional layer, a pooling layer, and an output layer, and calculates a feature vector F representing the features of the input image. The convolutional layers and pooling layers may be arranged alternately. In this embodiment, a 224 x 224 pixel RGB color image is input to the input layer, and through calculations in each layer, a 2560-dimensional vector is output from the output layer as a feature vector.

The probability calculation network 52 is composed of an input layer, one or more fully connected layers, and an output layer. Softmax is used as the activation function for the output layer. When a feature vector F is input to the input layer, the output layer outputs the probabilities _p1 , _p2 , ..., _pN that the target tree belongs to the first, second, ..., Nth class, respectively, through calculations in each layer.

The tree classification network 5 may be an existing network (trained model) trained based on images of the tree canopy of forests other than forest 80. For example, it may be created using the method described in JP 2020-91640 A or the watershed segmentation method. According to the method described in JP 2020-91640 A, the computer 1 generates an orthophotograph using an aerial photograph 60, generates a DSM (Digital Surface Model) based on the orthophotograph, and generates a slope model based on the DSM. The canopy map 61 is then generated based on the red, green, and blue gradations of the orthophotograph, the DSM model, and the slope model.

It does not matter if the vegetation of the forest learned by the tree classification network 5 is different from the vegetation of forest 80. In other words, a CNN feature extraction network generated for a forest other than forest 80, regardless of the vegetation, can be used as feature extraction network 51 in computer 1.

The operator flies the drone 2 over the forest 80 and photographs part of or the entire forest 80 from a predetermined altitude, thereby obtaining an aerial photograph 60 such as that shown in Figure 5(A). The aerial photograph 60 is then input into the computer 1. Note that the aerial photograph 60 is an RGB color photograph.

The canopy map generator 41 (see Figure 3) of the computer 1 then detects the position and shape of objects such as canopies that appear in the aerial photograph 60, and generates a canopy map 61 as shown in Figure 5(B). The canopy map 61 shows the position and shape of each object as polygons 61A. The canopy map 61 can be created using known methods. For example, it can be created using the method described in JP 2020-91640 A.

The labeling unit 42 assigns labels corresponding to the object classes to images of some of the objects captured in the aerial photograph 60 as follows:

The labeling unit 42 displays the canopy map 61 on the touch panel display 18. Instead of the canopy map 61, the outlines of each polygon 61A may be displayed superimposed on the aerial photograph 60.

Here, the operator selects (samples) one to several polygons 61A representing the canopy for each class of trees growing in forest 80. For example, if the classes of trees growing in forest 80 are metasequoia, white pine, and cypress, the operator selects one to several polygons 61A representing metasequoia, white pine, and cypress. Furthermore, the operator selects one to several polygons 61A representing non-tree objects (for example, the ground). It is desirable to select the same number of polygons 61A for each. That is, for example, it is desirable to select three polygons 61A each for metasequoia, white pine, cypress, and the ground.

The labeling unit 42 then extracts an image of the object corresponding to each selected polygon 61A from the aerial photograph 60 as object image 62G, and assigns a label identifying the object to the object image 62G as label 62L. For example, if the selected polygon 61A is a metasequoia, the labeling unit 42 extracts an image of the object corresponding to the polygon 61A from the aerial photograph 60 as object image 62G, and assigns the metasequoia label to the object image 62G as label 62L. Then, it generates object data 62 indicating the object image 62G and label 62L.

Through the operator's operations and the processing of the labeling unit 42, sample object data 64 for each class is generated, as shown in Figure 6.

In the example of Figure 6, the values of label 62L are "Class_1", "Class_2", etc., but they may also be type names such as "Red Oak", "Red Pine", "Other". In this case, the operator simply inputs each type name when selecting polygon 61A.

(2) Learning Fig. 7 is a diagram showing an example of an inference method. Fig. 8 is a diagram showing an example of the distribution of feature points in a 2640-dimensional space. Fig. 9 is a diagram showing an example of a tree classification map 67.

When object data 62 is generated, the feature vector calculation unit 43 (see Figure 3) calculates the feature vector F of each object image 62G of the object data 62 by inputting the object image 62G into the feature extraction network 51, as shown in Figure 7. Note that the object image 62G is appropriately reduced in size to match the size of the input layer of the feature extraction network 51 before being input.

Then, the feature vector calculation unit 43 generates training data 63 by replacing the object image 62G in the object data 62 with the calculated feature vector F, and stores the training data 63 in the training data storage unit 44.

For example, if the operator selects five polygons 61A for each of 11 classes of objects (trees or other objects), processing by the canopy map generation unit 41 or feature vector calculation unit 43 generates 55 pieces of training data 63 and stores them in the training data storage unit 44.

The following description will be given taking as an example a case where P pieces of teacher data 63 are stored in the teacher data storage unit 44. The feature vectors F indicated in the respective teacher data 63 may be distinguished and described as "feature vector F ₁ ,""feature vector F ₂ ," ..., "feature vector F _P ."

These feature vectors F can be thought of as position vectors starting from the origin of 2640-dimensional space (Euclidean space). Hereinafter, the end points of feature vectors F in 2640-dimensional space will be referred to as "feature points." When the feature points of each object's feature vector F are plotted in 2640-dimensional space, the distribution of the features of the objects present in forest 80 appears, as shown in Figure 8 (A).

[Inference Phase]
The operator inputs an aerial photograph 65, which is a photograph of a part or the whole of the forest 80 taken by the drone 2 from a predetermined altitude, into the computer 1. The aerial photograph 65, like the aerial photograph 60, is an RGB color photograph.

When an aerial photograph 65 is input, the canopy detection unit 45 (see Figure 3) detects the position and shape of objects such as canopies that appear in the aerial photograph 65. The detection method is the same as the detection method used by the canopy map generation unit 41, and for example, the method described in JP 2020-91640 A is used.

When the canopy detection unit 45 detects the position and shape of each object, such as a canopy, captured in the aerial photograph 65, the feature calculation unit 46 and class inference unit 47 perform processing to infer the class of each object. Below, we will explain an example of inferring the class of a certain object α.

The feature amount calculation unit 46 extracts an image of the object α from the aerial photograph 65 as an object image 66 based on the position and shape of the object α. Then, the feature vector F of the object α is calculated by inputting the object image 66 to a feature extraction network 51. This feature extraction network 51 is the same as the feature extraction network 51 used to calculate the feature amounts F ₁ , F ₂ , ..., F _P in the learning phase. Hereinafter, the feature vector F calculated by the feature amount calculation unit 46 will be referred to as a "feature vector F _A ".

The class inference unit 47 uses the k-nearest neighbor method to select k feature points that are closest to the feature point of the feature vector F _A from among the feature points of the feature vector F (F ₁ , F ₂ , ..., F _P ) of each piece of teacher data 63 stored in the teacher data storage unit 44. Then, it extracts the labels 62L of the teacher data 63 that correspond to the feature vector F of each selected feature point, and infers that the class indicated by the most labels 62L is the class of the object α. The distance between two feature points is the Euclidean distance.

8B, when the end point of the feature vector F _A is located at "X" and k is "3", the three end points closest to the end point of the feature vector F _A are the three points surrounded by dotted lines, and of these three points, the most common are end points of class_1. Therefore, the class inference unit 47 infers that class_1 is the class of the object α.

The class inference unit 47 infers the class of each of the other objects whose position and shape have been determined by the canopy detection unit 45 using the k-nearest neighbor method, in the same way as for object α.

The tree classification map generation unit 48 generates a tree classification map 67 as shown in Figure 9 based on the class inferred by the class inference unit 47 for each object whose position and shape have been determined by the canopy detection unit 45. Specifically, the tree classification map 67 is generated by replacing the image of each object in the aerial photograph 65 with a polygon of a color corresponding to the class of each object.

The tree classification map output unit 49 then outputs the generated tree classification map 67 by displaying it on the touch panel display 18. Alternatively, the tree classification map 67 file may be transmitted to another computer via a communication line.

[Overall Processing Flow and Effects of This Embodiment]
Fig. 10 is a flowchart illustrating an example of the overall processing flow by the tree classification program 4. Fig. 11 is a diagram showing an example of a correct/incorrect result.

Next, the overall processing flow by the computer 1 will be explained with reference to the flowchart in Figure 10. The computer 1 executes processing based on the tree classification program 4, following the procedure shown in Figure 10.

When an aerial photograph 60 such as that shown in Figure 5(A) is input, the computer 1 generates a canopy map 61 such as that shown in Figure 5(B) based on the aerial photograph 60 (#101 in Figure 10) and displays it (#102).

When the operator selects one or several polygons 61A corresponding to each class, the computer 1 generates object data 62 as shown in FIG. 6 by assigning the corresponding class label 62L to the original image (object image 62G) of each selected polygon 61A (#103). Then, the computer 1 calculates the feature vector F for each object image 62G (#104), and generates and stores training data 63 indicating the feature vector F and label 62L for each object image 62G (#105). The learning phase is completed with the above processing.

When the aerial photograph 65 is input, the computer 1 detects the position and shape of objects shown in the aerial photograph 65 (#121) and calculates the feature vector F of the object image 66 of each object in the aerial photograph 65 (#122).

Furthermore, the computer 1 infers the class of each object in the aerial photograph 65 using the k-nearest neighbor method based on the feature vector F indicated in each training data 63 and the feature vector F calculated in step #122 (#123). Then, a tree classification map 67 like the one shown in Figure 9 is generated based on the inference results (#124) and output (#125).

According to this embodiment, the operator does not need to prepare a huge amount of training data as in the case of conventional deep learning; instead, it is sufficient to prepare one or several pieces of object data 62 for each class (type) of trees growing in a certain area (site). Based on this object data 62, the computer 1 generates training data 63, completing preparations for inference. Therefore, according to this embodiment, it is easier than ever to build AI that can infer classes such as trees.

Here, an example of experimental results will be presented to further explain the effects of this embodiment. A CNN was generated using deep learning to classify nine types of trees (Japanese koshiabura, Mizume, beech, Japanese walnut, magnolia, adzuki pear, Japanese cherry, sugar maple, and horse chestnut) and non-tree objects (gaps) into one of 10 classes. The object to be inferred was then classified using this CNN, and the actual class and the inferred class were tallied to obtain the true/false result_1 shown in Figure 11.

On the other hand, as per the method of this embodiment, a small amount of training data 63 (e.g., 10 for each class) is generated using an existing feature extraction network 51, and the object to be inferred is classified using the k-nearest neighbor method based on this training data 63. The actual classes and inferred classes are then tallied, resulting in a true/false result of _2.

The kappa values for correct/incorrect result_1 and correct/incorrect result_2 are 0.414 and 0.456, respectively. In other words, even with the k-nearest neighbor method using a set of training data 63 of approximately 10 items per class acquired for each region (site), it is possible to achieve classification accuracy on the same level as the CNN acquired through deep learning. Furthermore, the classification accuracy of some classes (sugar maple) can be significantly improved compared to deep learning.

[Modifications and Applications]
If the distances between the feature points of feature vector F (e.g., feature vector F _A in FIG. 7 ) of object image 66 of the object to be inferred and the feature points of each of feature vectors F ₁ , F ₂ , ..., F _P are all equal to or greater than a predetermined distance, it is highly likely that the object does not belong to any class. Therefore, in such a case, computer 1 may infer the class of the object using tree classification network 5 (see FIG. 4 ) without using the k-nearest neighbor method. That is, object image 66 of the object may be input to feature extraction network 51, and the class corresponding to the highest probability among probabilities p ₁ , p ₂ , ..., p _N output from probability calculation network 52 may be inferred to be the class of the object.

In this embodiment, as described with reference to FIG. 8 , the computer 1 infers the class of the target object using the k-nearest neighbor algorithm in 2640 dimensions. However, the inference may be performed after reducing the dimensions. For example, several (e.g., 10) most important elements of the 2640 elements (components) of the feature vector F may be determined using the Sequential Forward Selection (SFS) method, and the class of the target object may be inferred using the k-nearest neighbor algorithm based on the determined feature quantities. That is, for example, the feature vectors F ₁ , F ₂ , ..., F _P in FIG. 7 may be converted into low-dimensional vectors G ₁ , G ₂ , ..., G _P using the SFS method, and the feature vector F _A may be converted into a low-dimensional vector G _A using the SFS method. Then, the distances between the feature points of the low-dimensional vector G _A and the feature points of each of the feature vectors F ₁ , F ₂ , ..., F _P are calculated, and the class is inferred using the k-nearest neighbor algorithm.

The most important few elements may be determined based on the feature vector F of the image of the object to be inferred, i.e., object image 66, calculated in the inference phase (after step #122 in Figure 10). Alternatively, the most important few elements may be determined based on the feature vector F of one of the sample images, i.e., object image 62G, calculated in the learning phase (after #104).

Alternatively, important elements may be determined by a wrapper method other than the SFS method, such as the Sequential Backward Selection (SBS) method or a brute force method. Alternatively, important elements may be determined by a feature selection method other than the wrapper method.

Figure 12 shows an example of how to generalize the feature extraction network 51 and create a nationwide tree map.

By providing the tree classification program 4 to users throughout Japan and collecting data from each user, the provider of the tree classification program 4 can create a tree species map for the entire country of Japan and improve the accuracy of the tree classification network 5 (see Figure 4). This mechanism is explained below with reference to Figure 12.

The provider of the tree classification program 4 prepares in advance a server 31 that has a tree classification network 5. Furthermore, the provider standardizes in advance the identifiers (class codes or names) of the classes (types) of trees found in Japan, and prepares a tree list that lists these.

Each time the server 31 receives a request from a user, it sends the tree classification program 4, along with the feature extraction network 51 and tree list, to the user's computer (#131). The feature extraction network 51 is a neural network (general-purpose model) that is generally used in each user's region (site), and it is desirable that it be the latest version.

When the user receives the tree classification program 4 and feature extraction network 51 on a computer, the user installs the tree classification program 4 and feature extraction network 51 on the computer. This causes the computer to function as computer 1.

The user uses the tree classification program 4, feature extraction network 51, drone 2, etc. to tune the tree classification program 4 to the forests in their area and create a tree classification map 67 (#132). Labeling follows the tree list. The user then transmits the tree classification map 67 from their computer to the server 31 along with location information of the photographing location and object data 62 obtained during the creation process (#133).

When the server 31 receives the object data 62, tree classification map 67, and location information, it performs further learning (training) of the tree classification network 5 by deep learning using the object image 62G and label 62L shown in the object data 62 as explanatory variables and target variables, respectively (#134). Part of the object data 62 may also be used as validation data or test data. Furthermore, the tree classification map 67 is overlaid on a national map based on the location information (#135).

The above process progresses the training of the feature extraction network 51 and the creation of a tree species map for all of Japan. The latest version of the feature extraction network 51 is then provided to each user. By repeating this series of processes, object data 62 is collected from various locations, and the feature extraction network 51 is trained to become a large-scale, general-purpose model. Note that the tree classification network 5 may be subjected to deep learning to create a tree species map for the entire world, not just Japan, and to classify tree species worldwide.

In this embodiment, the k-nearest neighbor method is used when inferring the class of an object such as a tree. However, the k-nearest neighbor method may be modified as follows: From the feature points of the feature vectors F (F ₁ , F ₂ , ...) of each piece of training data 63, k feature points that are closest to the feature points of the feature vector F _A of the object to be inferred are selected. Each selected feature point is classified into a class indicated by the label 62L of the training data 63 corresponding to each feature point. Then, for each class, the sum of the distances between each selected feature point and the feature points of the object to be inferred is calculated, and the class with the shortest sum is inferred to be the class of the object to be inferred. Alternatively, instead of votes, a larger score may be assigned to a closer distance, and the class with the largest sum of scores may be inferred to be the class of the object to be inferred.

Alternatively, instead of the k-nearest neighbor method, a support vector machine (SVM) or random forest may be used to generate a learner, and a class may be inferred using the generated learner. Whether an SVM or a random forest is used, a learner may be generated by reducing the dimensions of the feature vectors F ₁ , F ₂ , ..., F _P using an SFS method or the like, and a class may be inferred by reducing the dimensions of the feature vector F _A in a similar manner.

In this embodiment, the functions shown in FIG. 3 are concentrated in computer 1, but they may be distributed across multiple devices. For example, the canopy map generation unit 41 and label assignment unit 42 may be provided in a first computer, the feature vector calculation unit 43 in a second computer, the teacher data storage unit 44 in a third computer, the canopy detection unit 45 in a fourth computer, the class inference unit 47 in a fifth computer, and the tree classification map generation unit 48 and tree classification map output unit 49 in a sixth computer. Computer programs for realizing each function are installed in the first through sixth computers, and by cooperating with each other via communication lines, they execute a series of learning and inference processes for generating a tree classification map 67 from an aerial photograph 60.

In this embodiment, the computer 1 is used to classify trees, but it may also be used to classify other objects. For example, it may be used to classify grassland flowers, shrubs, animal habitats (animal nests such as anthills and mole nests), or landforms. It may also be used to classify rocks (volcanic rocks, hypabyssal rocks, plutonic rocks, sedimentary rocks, etc.), or to classify buildings.

Regardless of the type of classification, the computer 1 and drone 2 can basically perform the same processing as in this embodiment.

However, when classifying flowers and plants, the drone 2 collects aerial photographs 60, 65 by taking photographs at a lower altitude than when classifying trees. Because flowers and plants have a steep slope like the shape of a building, it is effective to use a slope model as in this embodiment, but because they are smaller than trees, it is desirable to take photographs at a lower altitude to match their size. The same applies when classifying shrubs and lake plants (e.g., reeds).

On the other hand, when classifying landforms (lakes, marshes, plains, hills, rivers), the drone 2 may collect aerial photographs 60, 65 by taking photographs at a higher altitude than when classifying trees. Alternatively, the aerial photographs 60, 65 may be collected at a lower resolution.

In addition, the overall configuration of computer 1 or each part, the processing content, processing order, data structure, etc. may be modified as appropriate in accordance with the spirit of the present invention.

1. Computer (learning device)
3. Tree Classification System (Class Inference System)
31 Server (learning means)
4. Tree classification program (computer program)
42 Label assignment unit (first acquisition means)
46 Feature calculation unit (second acquisition means)
47 Class inference unit (inference means)
5. Tree Classification Network (CNN)
51 Feature extraction network (trained model)
63 Training data (data for inference)
62G Object image 63 Training data (sample feature)
66 Object image (image of the object to be inferred)

Claims (8)

a first acquisition means for acquiring a first feature vector representing features of an image of a crown of a tree belonging to each of a plurality of classes that grows in a specific region by inputting the image into a trained model;
a second acquisition means for acquiring a second feature vector representing features of an image of a crown of an inference target tree that is a target of inference and grows in the specific region by inputting the image into the trained model;
an inference means for inferring a class to which the inference target tree belongs from among the plurality of classes, based on the first feature vector of each of the plurality of classes and the second feature vector of the inference target tree;
and
The trained model is a network of a part of a CNN (Convolutional Neural Network) generated by deep learning using the classes of a plurality of trees and the images of tree crowns as objective variables and explanatory variables, respectively, that calculates the feature vectors of input images by convolution processing.
A class inference system characterized by: the inference means infers a class to which the inference target tree belongs among the plurality of classes by a k-nearest neighbor method;
The class inference system of claim 1 . the first acquisition means acquires, as the first feature vector of an image of a crown of a tree belonging to each of the plurality of classes, a first low-dimensional vector composed of each of a plurality of specific element values of a feature vector calculated by inputting the image into the trained model;
the second acquisition means acquires, as the second feature vector, a second low-dimensional vector consisting of each of the specific plurality of element values of a feature vector calculated by inputting an image of a crown of the inference target tree into the trained model;
3. The class inference system according to claim 1 or claim 2. The specific elements are obtained by performing a sequential forward selection (SFS) process on the feature vector calculated by the trained model.
4. The class inference system of claim 3. a learning means for further training the CNN using images of tree crowns belonging to each of the plurality of classes and each of the tree classes as explanatory variables and objective variables;
having
5. A class inference system according to claim 1. A tree species map generation system for generating a tree species map for an area having multiple regions where trees belonging to each of multiple classes grow, comprising:
a first acquisition means for acquiring, for each of the plurality of regions, a first feature vector representing features of an image of a crown of a tree growing in the region and belonging to each of the plurality of classes, by inputting the image into a trained model;
a second acquisition means for acquiring a second feature vector representing features of an image of the crown of each of a plurality of inference target trees growing in the region by inputting the image into the trained model;
an inference means for inferring a class to which each of the plurality of inference target trees belongs based on the first feature vector of each of the plurality of classes in an area where the inference target tree grows among the plurality of areas and the second feature vector of the inference target tree;
a map generating means for generating the tree species map based on the position information of each of the plurality of inference target trees and the inference result by the inference means;
A tree species map generation system comprising: Obtaining a first feature vector representing features of an image of a crown of a tree belonging to each of a plurality of classes growing in a specific region by inputting the image into a trained model;
A second feature vector representing features of an image of a crown of an inference target tree that is a target of inference and grows in the specific region is obtained by inputting the image into the trained model;
inferring a class to which the inference target tree belongs from among the plurality of classes based on the first feature vector of each of the plurality of classes and the second feature vector of the inference target tree;
As the trained model, a network of a part of a CNN (Convolutional Neural Network) that calculates feature vectors of input images by convolution processing is used, the part being generated by deep learning using the classes of a plurality of trees and the images of tree crowns as objective variables and explanatory variables, respectively.
A class inference method characterized by: A tree species map generation method for generating a tree species map for an area having multiple regions and in which trees belonging to multiple classes grow, comprising the steps of:
For each of the plurality of regions, a first feature vector representing features of an image of a crown of a tree belonging to each of the plurality of classes growing in the region is obtained by inputting the image into a trained model;
A second feature vector representing features of an image of a crown of each of a plurality of inference target trees growing in the region is obtained by inputting the image into the trained model;
inferring a class to which each of the plurality of inference target trees belongs based on the first feature vector of each of the plurality of classes in a region where the inference target tree grows among the plurality of regions and the second feature vector of the inference target tree;
generating the tree species map based on the position information and the inferred class of each of the plurality of inference target trees;
A tree species map generation method characterized by: