JP6884546B2 - Attribute judgment device, attribute judgment method, computer program, and recording medium - Google Patents

Description

The present invention relates to an attribute determination device, an attribute determination method, a computer program, and a recording medium.

There is known a technique of performing image processing on a target image acquired by a target image acquisition device to determine the attributes of an object included in the target image. The attribute of an object is the property or characteristic to which the object belongs. Determining the attribute of an object means determining the attribute to which the object belongs from a plurality of predefined attributes. Examples of the attributes of the object include plants, animals, man-made objects, water, sky, roads, and soil.

In the prior art, a small area patch image is extracted from the target image, the object is extracted from the patch image, and the feature amount of the extracted object is compared with the feature amount of the predefined object. By doing so, the attribute of the object is determined.

When determining the attributes of an object from a two-dimensional image, the accuracy of determining the attributes of the object may decrease unless the distance between the object image acquisition device and the object is taken into consideration. For example, when the image contains a complicated terrain, the image includes a plurality of objects having different distances from the target image acquisition device. When the patch image has a single size, if the distance between the target image acquisition device and each of the plurality of objects is significantly different, the proportion of the objects in the patch image will be different for each object. As a result, the determination accuracy of the attribute of the object may decrease.

Therefore, a technique has been devised in which the distance information of the object included in the target image is calculated and the size of the patch image is changed based on the calculated distance information. In Patent Document 1, the size of the window is changed based on the distance information of the pixel of interest, the feature amount of the pixel of interest in the window is extracted, and the extracted feature amount is compared with the feature amount obtained in advance. Discloses a technique for identifying an object.

Japanese Unexamined Patent Publication No. 2013-030183

For the calculation of the distance information of the object, a calculation method such as a direct meter side method using a three-dimensional measurement sensor such as a laser sensor or a stereo measurement method using a stereo camera is used. However, in a situation where a three-dimensional measurement sensor cannot be used or a stereo camera cannot be used, it becomes difficult to calculate the distance information of the object.

An object of the present invention is to be able to accurately determine the attributes of an object even in a situation where it is difficult to calculate the distance information of the object.

According to the first aspect of the present invention, a target image data acquisition unit that acquires image data of an object and a patch image that includes a target pixel of interest and generates a plurality of patch image data of different sizes from the target image data. Based on the data generation unit, the feature amount extraction unit that extracts the feature amount of the target pixel of interest of the patch image data, and the feature amount calculated from each of a plurality of library image data of different sizes including the learning attention pixel. An attribute for determining the attribute of the target pixel of interest using an attribute data storage unit that stores a plurality of classifiers generated in advance, a feature amount calculated from the patch image data, and the plurality of classifiers. An attribute determination device including a determination unit is provided.

In the first aspect of the present invention, the patch image data generation unit generates a plurality of the patch image data so that the size of the patch image data is the same as each of the plurality of sizes of the library image data. You may.

In the first aspect of the present invention, there is an imaging status data acquisition unit that acquires imaging status data indicating a status at the time of acquiring image data of the object, and the attribute determination unit has a plurality of imaging status data based on the imaging status data. A specific classifier may be selected from the classifiers, and the attributes of the target pixel of interest may be determined based on the selected classifiers.

According to the second aspect of the present invention, the target image data is acquired, a plurality of patch image data having different sizes including the target pixel of interest are generated from the target image data, and the patch image data Based on extracting the feature amount of the target pixel of interest, the feature amount calculated from the patch image data, and the feature amount calculated from each of a plurality of library image data having different sizes including the learning attention pixel. An attribute determination method including determining the attribute of the target pixel of interest using a plurality of classifiers generated in advance is provided.

According to the third aspect of the present invention, a computer can generate a plurality of patch image data having different sizes including the target pixel of interest from the target image data, and the characteristics of the target pixel of the patch image data. A plurality of features generated in advance based on extracting the amount, the feature amount calculated from the patch image data, and the feature amount calculated from each of a plurality of library image data having different sizes including learning attention pixels. A computer program for determining the attributes of the target pixel of interest using a classifier and executing the data is provided.

According to the fourth aspect of the present invention, the computer generates a plurality of patch image data having different sizes including the target pixel of interest, and the characteristics of the target pixel of the patch image data. A plurality of features generated in advance based on extracting the amount, the feature amount calculated from the patch image data, and the feature amount calculated from each of a plurality of library image data having different sizes including learning attention pixels. A recording medium in which a computer program for determining the attributes of the target pixel of interest using a classifier and executing the determination is recorded is provided.

According to the aspect of the present invention, even in a situation where it is difficult to calculate the distance information of the object, the attribute of the object can be determined with high accuracy.

FIG. 1 is a hardware configuration diagram showing an example of an attribute determination device according to the present embodiment. FIG. 2 is a functional block diagram showing an example of the attribute determination device according to the present embodiment. FIG. 3 is a diagram schematically showing the relationship between the target image and the patch image according to the present embodiment. FIG. 4 is a diagram schematically showing an example of the patch image data and the library image data stored in the attribute data storage unit according to the present embodiment. FIG. 5 is a diagram schematically showing an example of the classifier according to the present embodiment. FIG. 6 is a diagram schematically showing an example of a weak classifier according to the present embodiment. FIG. 7 is a diagram schematically showing an example of a camera and a sensor according to the present embodiment. FIG. 8 is a flowchart showing an example of the attribute determination method according to the present embodiment. FIG. 9 is a diagram schematically showing an example of a learning phase according to the present embodiment. FIG. 10 is a flowchart showing an example of the learning phase according to the present embodiment. FIG. 11 is a flowchart showing an example of the learning phase according to the present embodiment.

Hereinafter, embodiments according to the present invention will be described, but the present invention is not limited thereto. The components according to the embodiment can be combined as appropriate, or some components may not be used.

[Hardware configuration]
FIG. 1 is a hardware configuration diagram showing an example of the attribute determination device 100 according to the present embodiment. The attribute determination device 1 includes a computer system. As shown in FIG. 1, the attribute determination device 100 includes an arithmetic processing unit 1, a storage device 2, and a drive device 3.

The arithmetic processing device 1 includes a processor such as a CPU (Central Processing Unit). The storage device 2 includes a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory). The drive device 3 includes storage such as a flash memory or a hard disk. The storage device 2 stores a computer program 4 executed in the arithmetic processing of the arithmetic processing unit 1. The computer program 4 is read into the arithmetic processing unit 1 and executed.

The recording medium 5 may be a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, or an electrical, such as a ROM or a flash memory. It may be a semiconductor memory that records information on the optical disk.

The computer program 4 executed in the arithmetic processing of the arithmetic processing unit 1 may be recorded on the recording medium 5 and installed in the arithmetic processing unit 1 via the drive device 3. The computer program 4 executed in the arithmetic processing of the arithmetic processing unit 1 may be transmitted from the server to the arithmetic processing unit 1 and installed.

The attribute determination device 1 is connected to the recording medium 5, the input device 6, and the display device 7.

The input device 6 generates input data by being operated by an operator. The input data generated by the input device 6 is output to the arithmetic processing unit 1. The input device 6 includes, for example, at least one of a computer keyboard, a mouse, and a touch panel.

The output device 7 includes a flat panel display such as a liquid crystal display (LCD) or an organic EL display (OELD), and displays display data generated by the arithmetic processing device 1. The display data may be image data or character data. The output device 7 may include at least one of a printing device and a speaker device that outputs audio data.

The attribute determination device 1 is connected to the target image acquisition device 8 and the sensor 9. The target image acquisition device 8 acquires target image data including an object for attribute determination. A camera is exemplified as the target image acquisition device 8. The camera may be a visible camera or an infrared camera (thermography). A laser sensor or a temperature sensor may be used as the target image acquisition device 8. In the present embodiment, a camera is used as the target image acquisition device 8. In the following description, the target image acquisition device 8 is appropriately referred to as a camera 8.

The camera 8 captures target image data including an object for attribute determination. The camera 8 includes an optical system and an image sensor. The image sensor includes at least one of a CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor) and a CCD image sensor (Charge Coupled Device Image Sensor).

The sensor 9 detects imaging status data indicating the status of the camera 8 at the time of imaging. The situation at the time of imaging of the camera 8 includes the posture and position of the camera 8 at the time of imaging. The posture of the camera 8 includes the orientation of the optical system of the camera 8 (the angle of the optical axis of the optical system). The position of the camera 8 includes the installation position (installation height of the optical system) of the optical system of the camera 8 from the ground. The sensor 9 includes a posture sensor capable of detecting the posture of the camera 8 and a position sensor capable of detecting the position of the camera 8.

[Functional configuration]
FIG. 2 is a functional block diagram showing an example of the attribute determination device 100 according to the present embodiment. As shown in FIG. 2, the attribute determination device 100 includes a data analysis unit 10, a data storage unit 20, and an output control unit 30.

The data analysis unit 10 includes a data acquisition unit 11, a patch image data generation unit 12, a feature amount extraction unit 13, and an attribute determination unit 14. The functions of the data analysis unit 10 are included in the arithmetic processing unit 1 and the storage device 2.

The data storage unit 20 includes a target image data storage unit 21, an imaging status data storage unit 22, and an attribute data storage unit 23. The function of the data storage unit 20 is included in the storage device 2.

The data acquisition unit 11 acquires target image data indicating a target image including an object for attribute determination captured by the camera 8. In addition, the data acquisition unit 11 acquires imaging status data indicating the status of the camera 8 detected by the sensor 9 at the time of imaging. The data acquisition unit 11 functions as a target image data acquisition unit that acquires the target image data captured by the camera 8. In addition, the data acquisition unit 11 functions as an imaging status data acquisition unit that acquires imaging status data indicating the status of the camera 8 at the time of imaging.

The patch image data generation unit 12 generates patch image data indicating a patch image from the target image data. The patch image is a partial image of a small area of the target image. The patch image data generation unit 12 extracts a patch image from the target image. The patch image data generation unit 12 can cut out a patch image of an arbitrary size (patch size) from the target image. In the present embodiment, the patch image data generation unit 12 generates a plurality of patch image data of different sizes from the target image data.

In the present embodiment, the patch image data generation unit 12 generates a patch image so as to include the target pixel of interest. The patch image data generation unit 12 generates a plurality of patch image data having different sizes including the target pixel of interest from the target image data.

The feature amount extraction unit 13 extracts the feature amount of the target pixel of interest in the patch image data. The feature amount of the target attention pixel includes a set of feature amounts (feature amount vector) included in the target attention pixel. The feature amount extraction unit 13 extracts a feature amount including texture data such as brightness, brightness, hue, pattern, and texture of the surface of the object. The feature amount extraction unit 13 extracts the feature amount of the target pixel of interest by using a known feature amount calculation method such as HOG (Histograms of Oriented Gradients).

The attribute determination unit 14 uses the feature amount calculated from the patch image data including the target attention pixel and the classifier generated in advance based on the feature amount calculated from the library image data to attribute the target attention pixel. To judge. The feature amount of the target pixel of interest in the patch image data is extracted by the feature amount extraction unit 13. The attribute data storage unit 23 stores a pre-generated classifier. The library image data includes learning attention pixels. The feature amount of the learning attention pixel is known data calculated in advance. Further, the feature amount of the learning attention pixel and the attribute of the learning attention pixel are associated with each other. The attributes of the learning attention pixel are predefined known data. The attribute determination unit 14 divides the patch image data including the target attention pixel from which the feature amount is extracted by the feature amount extraction unit 13 and the library image data including the learning attention pixel whose feature amount is known and associated with the attribute. By collating, the attribute of the target pixel of interest can be determined.

The attribute of the target pixel of interest refers to a property or feature to which the target pixel of interest belongs. Determining the attribute of the target pixel of interest means determining the attribute to which the target pixel of interest belongs from a plurality of predefined attributes. Examples of the attributes of the object include plants, animals, man-made objects, water, sky, roads, and soil.

The target image data storage unit 21 stores or temporarily holds the target image data captured by the camera 8. The target image data storage unit 21 outputs the target image data to the data acquisition unit 11. The data acquisition unit 11 acquires the target image data from the camera 8 via the target image data storage unit 21.

The imaging status data storage unit 22 stores or temporarily holds the imaging status data acquired by the sensor 9. The imaging status data storage unit 22 outputs the imaging status data to the data acquisition unit 11. The data acquisition unit 11 acquires the imaging status data from the camera 8 via the imaging status data storage unit 22.

The attribute data storage unit 23 stores a plurality of library image data of different sizes, including learning attention pixels having known feature amounts. The attribute data storage unit 23 outputs the library image data to the data acquisition unit 11. The data acquisition unit 11 acquires library image data from the attribute data storage unit 23.

The learning attention pixel is associated with the attribute of the object corresponding to the learning attention pixel. The attribute data storage unit 23 collates the patch image data generated by the patch image data generation unit 12 with the plurality of library image data stored in the attribute data storage unit 23, and determines the target pixel of interest. Determine the attributes.

A plurality of library image data including learning attention pixels having known feature amounts are stored in the attribute data storage unit 23 in a database format. In the present embodiment, the attribute data storage unit 23 has a plurality of classifiers that hold a plurality of library image data of the same size. The classifier is a functional unit that provides a function of classifying the feature amount of the target pixel of interest extracted from the target image into a corresponding feature amount class as input data. In the present embodiment, the classifier includes at least a database of library image data used for determining the attributes of the target pixel of interest. A plurality of library image data of the same size are held in one classifier. The plurality of library image data held in one classifier have different attributes. The classifier is created in advance based on the feature amount calculated from the image data and the correct answer attribute data.

The attribute data storage unit 23 has a plurality of classifiers. The plurality of classifiers hold library image data of different sizes. That is, the first classifier holds a plurality of first size library image data. The second classifier holds a plurality of second size library image data. The third classifier holds a plurality of third size library image data. The attribute data storage unit 23 can hold any plurality of N classifiers that hold patch image data of different sizes.

The classifier also includes a plurality of weak classifiers. The plurality of weak classifiers hold a plurality of library image data of the same size. The plurality of library image data held in one weak classifier have different attributes. That is, one weak classifier holds a plurality of library image data of the same size including learning attention pixels having different feature amounts. The classifier does not have to include a weak classifier.

The output control unit 30 is connected to the data analysis unit 10. The function of the output control unit 30 is included in the arithmetic processing unit 1 and the storage device 2. The output control unit 30 processes the determination data indicating the determination result of the attribute determination unit 14 and generates the output data that can be output by the output device 7.

The output data generated by the output control unit 30 is transmitted to the output device 7. The output control unit 30 generates, for example, display data displayed on the display screen of the output device 7 or audio data output from the speaker device, and transmits the display data to the output device 7. The output device 7 operates based on the output data (display data or voice data) transmitted from the output control unit 30.

In the present embodiment, the output control unit 30 superimposes the determination data by the attribute determination unit 14 on the target image captured by the camera 8 and outputs it to the output device 7.

[Relationship between target image and patch image]
FIG. 3 is a diagram schematically showing the relationship between the target image and the patch image according to the present embodiment. The target image includes a plurality of pixels arranged in a matrix. FIG. 3 shows an example in which patch image data of the first size, patch image data of the second size, and patch image data larger than the second size are generated. The size of the patch image data includes the number of pixels. In the example shown in FIG. 3, the patch image data of the first size is composed of nine pixels. The patch image data of the second size is composed of 49 pixels. The patch image data of the third size is composed of 121 pixels.

The size of the patch image shown in FIG. 3 is an example. The patch image data generation unit 12 can generate an arbitrary plurality of N patch image data of different sizes.

As shown in FIG. 3, in the present embodiment, the patch image data generation unit 12 generates a plurality of patch image data of different sizes so that the target pixel of interest is arranged at the center of the patch image.

In the example shown in FIG. 3, the target pixel of interest is one pixel. The target pixel of interest may be composed of a plurality of adjacent pixels. That is, the target pixel of interest may be composed of a pixel region including a plurality of pixels.

[Patch image data and library image data]
FIG. 4 is a diagram schematically showing an example of the patch image data and the library image data stored in the attribute data storage unit 23 according to the present embodiment. The attribute data storage unit 23 stores a plurality of library image data of different sizes.

As shown in FIG. 4, the attribute data storage unit 23 has a first classifier that holds a plurality of library image data of the first size, a second classifier that holds a plurality of library image data of the second size, and a third classifier. It has a third classifier that holds a plurality of size library image data. Of the first size, the second size, and the third size, the first size is the smallest, the second size is the smallest next to the first size, and the third size is the largest.

The sizes of the plurality of library image data held in the first classifier are all the first size. The sizes of the plurality of library image data held in the second classifier are all the second size. The sizes of the plurality of library image data held in the third classifier are all the third size. That is, one classifier holds a plurality of library images of the same size.

The first classifier holds a plurality of library image data having different attributes. The second classifier holds a plurality of library image data having different attributes. The third classifier holds a plurality of library image data having different attributes.

In the example shown in FIG. 4, the first classifier holds the first size library image data 1a having the attribute "animal" and the first size library image data 1b having the attribute "plant". The second classifier holds the second size library image data 2a having the attribute "artificial object" and the second size library image data 2b having the attribute "plant".

The attribute data storage unit 23 shown in FIG. 4 is an example. The first classifier holds not only the library image data 1a and the library image data 1b, but also a plurality of X first-size library image data 1x having different attributes. The second classifier holds not only the library image data 2a and the library image data 2b, but also a plurality of X second-size library image data 2x having different attributes. Similarly, the third classifier holds a plurality of X third-sized library image data 3x having different attributes.

Further, the attribute data storage unit 23 can hold any plurality of N classifiers. The Nth classifier holds a plurality of X Nth size library image data NXs having different attributes.

As shown in FIG. 4, the target image data is captured by the camera 8. The patch image data generation unit 12 generates a plurality of patch image data of different sizes from the target image data. Each of the plurality of patch image data includes the target pixel of interest.

In the present embodiment, the patch image data generation unit 12 generates a plurality of patch image data so that the size of the patch image data is the same as each of the plurality of sizes of the library image data. In the example shown in FIG. 4, the attribute data storage unit 23 stores the first size library image data, the second size library image data, and the third size library image data. The patch image data generation unit 12 generates patch image data of the first size, patch image data of the second size, and patch image data of the third size.

The patch image data shown in FIG. 4 is an example. When the Nth size library image data is stored in the attribute data storage unit 23, the patch image data generation unit 12 performs the Nth size patch so that the target pixel of interest is arranged at the center from the target image data. Generate image data.

The attribute determination unit 14 collates the patch image data of the same size with the library image data. The attribute determination unit 14 collates the patch image data with the library image data in each of the plurality of N classifiers and calculates the collation data. The collation data includes a correlation value between the feature amount of the target pixel of interest in the patch image data and the feature amount of the learning attention pixel of the library image data. The attribute determination unit 14 determines the attributes of the target pixel of interest based on the plurality of collation data calculated by each of the plurality of N classifiers.

In the example shown in FIG. 4, the attribute determination unit 14 collates the patch image data of the first size with each of the plurality of M library image data of the first size held in the first classifier. Further, the attribute determination unit 14 collates the patch image data of the second size with each of the plurality of M library image data of the second size held in the second classifier. Further, the attribute determination unit 14 collates the patch image data of the third size with each of the plurality of M library image data of the third size held in the third classifier. Further, the attribute determination unit 14 collates the patch image data of the Nth size with each of the plurality of M library image data of the Nth size held in the Nth classifier.

The attribute determination unit 14 collates the patch data of the first size with each of the plurality of M library image data of the first size held in the first classifier, and the plurality of M of the first classifier. Calculate the matching data. The attribute determination unit 14 collates the patch data of the second size with each of the plurality of M library image data of the second size held in the second classifier, and collates the plurality of Ms of the second classifier. Calculate the matching data. The attribute determination unit 14 collates the patch data of the third size with each of the plurality of M library image data of the third size held in the third classifier, and the plurality of M of the third classifier. Calculate the collation data of. The attribute determination unit 14 collates the patch data of the Nth size with each of the plurality of M library image data of the Nth size held in the Nth discriminator, and collates the plurality of Ms of the Nth discriminator. Calculate the matching data.

That is, when there are N classifiers and M library image data are held in one classifier, the attribute determination unit 14 calculates the collation data according to [N × M]. The attribute determination unit 14 determines the attribute of the target pixel of interest in the patch image data based on the collation data of a plurality of [N × M].

In the example shown in FIG. 4, the attribute determination unit 14 determines that the second size patch image data generated by the patch image data generation unit 12 has the highest correlation with the library image data 2b of the second classifier. .. The attribute determination unit 14 determines that the attribute of the object centered on the target pixel of interest is the "plant" shown in the library image data 2b.

[Identifier]
FIG. 5 is a diagram schematically showing an example of the classifier according to the present embodiment. FIG. 5 is a diagram schematically showing an example of the Nth classifier.

The Nth classifier has a plurality of weak classifiers including learning attention pixels having different feature amounts and holding a plurality of library image data of the same size. In the example shown in FIG. 5, the Nth classifier has a first weak classifier, a second weak classifier, and an M weak classifier.

The first weak classifier holds a plurality of N-size library image data. The second weak classifier holds a plurality of N-size library image data. The M-th weak classifier holds a plurality of N-size library image data.

FIG. 6 is a diagram schematically showing an example of a weak classifier according to the present embodiment. As shown in FIG. 6, one classifier has a plurality of weak classifiers. Each of the discriminators includes a decision tree. In the present embodiment, the classifier has a plurality of weak classifiers, but the weak classifiers may not be included. Also, the classifier does not have to include a decision tree. The discriminator only needs to be able to express the discrimination result quantitatively, and may express the discrimination result as a score.

FIG. 6 is a diagram illustrating a known technique, a random forest. As shown in FIG. 6, the decision tree of the first weak classifier has an attribute determination label 41 attached to the non-terminating node and a plurality of attribute classes 42 attached to each of the plurality of terminating nodes. Similarly, the decision trees of the second weak classifier to the M weak classifier each have an attribute determination label 41 attached to the non-terminal node and a plurality of attribute classes 42 attached to each of the plurality of terminal nodes. ..

In FIG. 6, the attribute class 42 of the first weak classifier includes the attributes A, B, and C, and the attribute class 42 of the second weak classifier contains the attributes B, C, and D. An example is shown in which the attribute class 42 of the M weak classifier includes the attributes A, B, C, and D. In the present embodiment, each of the attribute classes 42 of the first weak classifier to the M weak classifier includes attributes from attribute A to arbitrary attribute X.

As described above, in one classifier, the plurality of weak classifiers hold a plurality of library image data of the same size. The plurality of library image data held in one weak classifier have different attributes.

In each of the plurality of weak classifiers, the attributes of the learning attention pixels of the plurality of library image data whose feature amounts (attributes) are known are set in the attribute class 42. Further, a decision tree including the attribute determination label 41 is constructed in each of the plurality of weak classifiers based on the feature amounts of the learning attention pixels of the plurality of library image data. The attribute determination unit 14 determines the attributes of the target pixel of interest by analyzing each of the plurality of decision trees constructed based on the library image data.

The attribute determination unit 14 determines the attribute of the target pixel of interest from the plurality of attribute classes 42 of the decision tree by the decision tree analysis in each of the plurality of weak classifiers. In the present embodiment, the collation data includes the attribute having the highest voter turnout among the attributes for which the voter turnout was calculated in each of the plurality of weak classifiers. That is, in the present embodiment, the attribute determination unit 14 has the attribute with the highest voter turnout among the attributes for which the voter turnout has been calculated in each of the plurality of weak classifiers, and the attribute of the target pixel of interest in the Nth classifier. To decide.

For example, when the attribute class 42 of the first weak classifier includes the attribute X from the attribute A and the attribute of the target pixel of interest is determined by the decision tree analysis, the number of votes obtained by the attribute A is A1 and the number of votes obtained by the attribute B. Is B1, and the number of votes obtained for the attribute X is X1. When the attribute class 42 of the second weak classifier includes the attribute X from the attribute A and the attribute of the target pixel of interest is determined by the decision tree analysis, the number of votes of the attribute A is A2 and the number of votes of the attribute B is B2. It is assumed that the number of votes obtained for the attribute X is X2. When the attribute class 42 of the M weak classifier includes the attribute X from the attribute A and the attribute of the target pixel of interest is determined by the decision tree analysis, the number of votes of the attribute A is AM and the number of votes of the attribute B is BM. , It is assumed that the number of votes obtained for the attribute X is XM. The attribute determination unit 14 totals the number of votes calculated in each of the first weak classifier to the M weak classifier. That is, the attribute determination unit 14 executes the calculation of [A1 + A2 + ... + AM], the calculation of [B1 + B2 + ... BM], ..., The calculation of [X1 + X2 + ... + XM]. The attribute determination unit 14 calculates the voter turnout of each attribute X from the attribute A based on the number of votes of the entire Nth classifier and the number of votes of each of the attributes A to X. For example, the attribute determination unit 14 calculates the ratio of [A1 + A2 + ... + AM] to the total number of votes obtained by the Nth classifier to calculate the voter turnout of the attribute A. Similarly, the attribute determination unit 14 calculates the ratio of [B1 + B2 + ... BM] to the total number of votes obtained by the Nth classifier, calculates the voter turnout of the attribute B, and [X1 + X2 +] to the total number of votes obtained by the Nth classifier. ... + XM] is calculated to calculate the voter turnout of attribute X. Similarly, the attribute determination unit 14 calculates the voter turnout from the attribute C to the attribute W. When the voter turnout of attribute A is the highest among the voter turnouts from attribute A to attribute X, the attribute determination unit 14 determines that the attribute with the highest voter turnout is the attribute of the target pixel of interest in the Nth classifier. ..

The attribute determination unit 14 performs the same processing for each of the first classifier to the Nth classifier. In the present embodiment, the attribute determination unit 14 determines that the attribute with the highest voter turnout among the attributes determined by each of the plurality of classifiers is the attribute of the target pixel of interest.

[Sensor]
FIG. 7 is a diagram schematically showing an example of the camera 8 and the sensor 9 according to the present embodiment. As shown in FIG. 7, the camera 8 is supported by the support device 50. The support device 50 can adjust the posture and position of the camera 8. The posture of the camera 8 includes the orientation and installation position of the optical system of the camera 8. The orientation of the optical system of the camera 8 includes the angle of the optical axis of the optical system of the camera 8 with respect to the horizontal plane. The installation position of the camera 8 includes the installation height of the optical system of the camera 8 from the ground.

The sensor 9 includes a posture sensor capable of detecting the posture of the camera 8 and a position sensor capable of detecting the position of the camera 8. The shooting status data including the shooting status of the camera 8 including the posture and position of the camera 8 detected by the sensor 9 is output to the data acquisition unit 11.

Based on the detection data of the sensor 9, the attribute determination unit 14 captures any of the target image data of the distant view, the target image data of the middle view, and the target image data of the near view. Determine if it is a situation. For example, when the angle of the optical axis of the optical system of the camera 8 with respect to the horizontal plane is close to 0 [°], the attribute determination unit 14 determines that the camera 8 is capturing a distant view. When the angle of the optical axis of the optical system of the camera 8 with respect to the ground is large (for example, when it is 60 [°] downward), the attribute determination unit 14 determines that the camera 8 is capturing a near view. When the angle of the optical axis of the optical system of the camera 8 with respect to the ground is medium (for example, when it is 30 [°] downward), the attribute determination unit 14 states that the camera 8 is capturing a middle view. judge.

As described above, the attribute data storage unit 23 has a plurality of classifiers that hold library image data of different N types of sizes. In the present embodiment, the attribute determination unit 14 selects a specific classifier from a plurality of classifiers based on the imaging status data acquired by the sensor 9, and determines the attributes of the target pixel of interest based on the selected classifiers. judge. For example, when it is determined that the camera 8 is capturing a distant view, the attribute determination unit 14 selects a classifier that holds a small size library image data from a plurality of classifiers, and selects a large size library image data. Reject the classifier that holds. When it is determined that the camera 8 is capturing a near view, the attribute determination unit 14 selects a classifier that holds a large size library image data from among a plurality of classifiers, and holds a small size library image data. Reject the classifier.

In the present embodiment, the attribute determination unit 14 can appropriately select the size and number of the classifiers to be rejected based on the imaging status data.

[Attribute judgment method]
Next, an example of the attribute determination method according to the present embodiment will be described. FIG. 8 is a flowchart showing an example of the attribute determination method according to the present embodiment. As shown in FIG. 8, in the attribute determination method according to the present embodiment, the learning phase SA for constructing a database of library image data is collated with the patch image data and the library image data to determine the attributes of the target pixel of interest. Includes identification phase SB.

(Learning phase)
FIG. 9 is a diagram schematically showing an example of the learning phase SA according to the present embodiment. As shown in FIG. 9, the teacher image is acquired in the learning phase SA. A teacher image is an image used in the learning phase from which the correct answer attribute can be derived. By processing the teacher image based on the machine learning algorithm, a plurality of classifiers including a decision tree are constructed.

FIG. 10 is a flowchart showing an example of the learning phase SA according to the present embodiment. As shown in FIG. 10, first, a teacher image is acquired (step SA1). Next, a process for generating the first classifier is performed (step SA2).

In the process of generating the first classifier, an arbitrary learning attention pixel is selected from the teacher image. In the present embodiment, first, the first learning attention pixel is selected (step SA11).

Next, first-sized library image data centered on the first learning attention pixel is generated from the teacher image. From the library image data of the first size including the first learning attention pixel, the feature amount of the first learning attention pixel is extracted by using a known feature amount calculation method such as HOG (Histograms of Oriented Gradients).

The feature amount of the first learning attention pixel is associated with the correct answer attribute for the first learning attention pixel. The correct answer attribute for the first learning attention pixel can be derived from the teacher image.

Further, an arbitrary second learning attention pixel is selected from the teacher image (step SA21). Next, the first size library image data centered on the second learning attention pixel is generated from the teacher image. From the library image data of the first size including the second learning attention pixel, the feature amount of the second learning attention pixel is extracted by using a known feature amount calculation method such as HOG (Histograms of Oriented Gradients).

The feature amount of the second learning attention pixel is associated with the correct answer attribute for the second learning attention pixel. The correct answer attribute for the second learning attention pixel can be derived from the teacher image.

The same processing as described above is performed for each of the first learning attention pixel to the Z learning attention pixel.

The first size library image data including the first learning attention pixel associated with the correct answer attribute is processed from the first size library image data including the first learning attention pixel associated with the correct answer attribute based on the machine learning algorithm. (Step SA3). As a result, a first classifier including a decision tree generated based on the library image data of the first size including a plurality of learning attention pixels is generated.

As described above, the first classifier including the first learning attention pixel to the Z learning attention pixel having different feature amounts and having a plurality of weak classifiers holding a plurality of library image data of the same first size is constructed. To.

The same process as the process for generating the first classifier described above is performed for each of the second size library image data, the third size library image data, ..., And the Nth size library image data. As described above, the Nth classifier is constructed from the first classifier that holds the library image data of the first size to the Nth size, respectively. The generated first to Nth classifiers are stored in the attribute data storage unit 23.

(Identification phase)
Next, the identification phase SB will be described. FIG. 11 is a diagram schematically showing an example of the identification phase SB according to the present embodiment. As shown in FIG. 11, in the identification phase SB, first, the target image is captured by the camera 8 (step SB1). The data acquisition unit 11 acquires the target image data captured by the camera 8.

In addition, the sensor 9 detects imaging status data indicating the status of the camera 8 at the time of imaging. The data acquisition unit 11 acquires the imaging status data detected by the sensor 9.

The attribute determination unit 14 selects a specific classifier from a plurality of classifiers stored in the attribute data storage unit 23 based on the imaging status data acquired by the data acquisition unit 11 (step SB2).

Based on the imaging status data acquired by the data acquisition unit 11, the attribute determination unit 14 determines which of the distant view target image data, the middle view target image data, and the near view target image data. Determine if the data is being imaged. For example, when it is determined that the target image data of the distant view has been acquired, the attribute determination unit 14 selects the classifier that holds the library image data of a small size from among the plurality of classifiers, and selects the library image data of a large size. Reject the classifier to hold. When it is determined that the target image data of the near view has been acquired, the attribute determination unit 14 selects the classifier that holds the large size library image data from among the plurality of classifiers, and holds the small size library image data. Reject the classifier.

In the present embodiment, some of the first classifiers to the Nth classifiers stored in the attribute data storage unit 23 are rejected, and the Kth classifier is selected from the first classifier. I will do it.

Next, an arbitrary learning attention pixel is selected from the target image. In the present embodiment, first, the first target pixel of interest is selected (step SB3).

Next, the patch image data generation unit 12 generates patch image data of the first size centered on the first target pixel of interest from the target image. Next, the feature amount extraction unit 13 uses a known feature amount calculation method such as HOG (Histograms of Oriented Gradients) from the patch image data of the first size including the first target pixel of interest to perform the first target. The feature amount of the pixel of interest is extracted (step SB11).

Next, the attribute determination unit 14 collates the patch image data of the first size with the library image data of the first size in the first classifier stored in the attribute data storage unit 23 (step SB12).

The attribute determination unit 14 determines the attribute of the target pixel of interest from the plurality of attribute classes of the decision tree by the decision tree analysis in each of the plurality of weak classifiers of the first classifier. The attribute determination unit 14 determines the attribute having the highest voter turnout among the attributes for which the voter turnout has been calculated in each of the plurality of weak classifiers of the first classifier (step SB13). In the present embodiment, the attribute determined by the first classifier is the attribute A, and the maximum voter turnout of the attribute A is 80 [%].

Next, the patch image data generation unit 12 generates patch image data of a second size centered on the first target pixel of interest from the target image. Next, the feature amount extraction unit 13 uses a known feature amount calculation method such as HOG (Histograms of Oriented Gradients) from the patch image data of the second size including the first target pixel of interest to perform the first target. The feature amount of the pixel of interest is extracted (step SB21).

Next, the attribute determination unit 14 collates the patch image data of the second size with the library image data of the second size in the second classifier stored in the attribute data storage unit 23 (step SB22).

The attribute determination unit 14 determines the attribute of the target pixel of interest from the plurality of attribute classes of the decision tree by the decision tree analysis in each of the plurality of weak classifiers of the second classifier. The attribute determination unit 14 determines the attribute having the highest voter turnout among the attributes for which the voter turnout has been calculated in each of the plurality of weak classifiers of the second classifier (step SB23). In the present embodiment, the attribute determined by the second classifier is the attribute B, and the maximum voter turnout of the attribute B is 60 [%].

Hereinafter, the same processing as described above is performed for each of the patch image data of the first size to the patch image data of the K size centering on the first target image of interest. The attribute determined by the K-th classifier is attribute B, and the maximum voter turnout of the attribute B is 40 [%].

In the present embodiment, the highest voter turnout among the attributes determined in each of the first classifier to the Kth classifier is 80 [%], and the attribute that has obtained the highest voter turnout 80 [%] is an attribute. It shall be A.

The attribute determination unit 14 determines that the attribute with the highest voter turnout among the attributes determined in each of the first classifier to the Kth classifier is the attribute of the first target pixel of interest (step SB5).

The output control unit 30 outputs an attribute that is the identification result of the first target pixel of interest (step SB6). That is, the output control unit 30 outputs that the attribute of the first target pixel of interest is the attribute A.

The same processing as described above is performed for each of the first target pixel of interest to the Z-learning attention pixel.

The same process as the process for generating the first classifier described above is performed from the first size to K size target image data centered on the second target pixel of interest and the first size centered on the third target pixel of interest. This is performed for each of the K-size target image data, ..., And the Z-target attention pixel-centered first-size to K-size target image data. As described above, each attribute of the first target pixel of interest to the Zth target pixel of interest is determined and output.

The attribute determination unit 14 determines the attribute of the target pixel based on the determination result of each attribute of the first target pixel of interest to the Zth target pixel of interest. The output control unit 30 outputs the attribute which is the identification result of the target pixel determined by the attribute determination unit 14 (step SB7).

[Action and effect]
As described above, according to the present embodiment, a database of a plurality of library image data having different sizes including learning attention pixels having known feature amounts is constructed in advance. Therefore, the attribute determination device 100 can accurately determine the attribute of the target pixel of interest by collating each of the patch image data with the plurality of library image data even if there is no distance information of the object.

Further, according to the present embodiment, the patch image data generation unit 12 generates a plurality of patch image data so that the size of the patch image data is the same as each of the plurality of sizes of the library image data, and determines the attributes. Part 14 collates the patch image data of the same size with the library image data. As a result, the patch image data and the library image data are collated with high accuracy, and the attributes of the target pixel of interest are determined with high accuracy.

Further, according to the present embodiment, the attribute data storage unit 23 has a plurality of classifiers that hold a plurality of library image data of the same size. The plurality of classifiers hold library image data of different sizes. As a result, the attribute determination unit 14 can collate the patch image data and the library image data in each of the plurality of classifiers, and accurately determine the attributes of the target pixel of interest based on the plurality of collation data.

Further, according to the present embodiment, the classifier has a plurality of weak classifiers including learning attention pixels having different feature amounts and holding a plurality of library image data of the same size. Weak classifiers include decision trees. As a result, the attribute determination unit can determine that the attribute with the highest voter turnout among the attributes determined by each of the plurality of classifiers is the attribute of the target pixel of interest.

Further, according to the present embodiment, an imaging status data acquisition unit 22 for acquiring imaging status data indicating the status of the camera 8 at the time of imaging is provided. The attribute determination unit 14 selects a specific classifier from a plurality of classifiers based on the imaging status data. As a result, the number of classifiers used is reduced, and the matching between the patch image data and the library image data is performed with a low load.

1 ... Arithmetic processing device, 2 ... Storage device, 3 ... Drive device, 4 ... Computer program, 5 ... Recording medium, 6 ... Input device, 7 ... Output device, 8 ... Camera, 9 ... Sensor, 10 ... Data analysis unit, 11 ... Data acquisition unit, 12 ... Patch image data generation unit, 13 ... Feature amount extraction unit, 14 ... Attribute determination unit, 20 ... Data storage unit, 21 ... Target image data storage unit, 22 ... Imaging status data storage unit, 23 ... Attribute data storage unit, 30 ... Output control unit, 41 ... Attribute determination label, 42 ... Attribute class, 50 ... Support device, 100 ... Attribute determination device.

Claims (6)

A target image data acquisition unit that acquires target image data indicating a target image including an object,
A patch image data generation unit that generates patch image data from the target image data indicating a plurality of patch images of different sizes including the target pixel of interest, and a patch image data generation unit.
A feature amount extraction unit that extracts the feature amount of the target pixel of interest of the patch image data, and a feature amount extraction unit.
An attribute data storage unit that stores a plurality of classifiers generated in advance based on feature quantities calculated from each of a plurality of library image data having different sizes including learning attention pixels.
A feature amount calculated from the patch image data and an attribute determination unit for determining the attributes of the target pixel of interest using the plurality of classifiers are provided .
Each of the plurality of patch images is a partial image of the target image cut out from the target image.
A plurality of patch image data of different sizes are generated so that the target pixel of interest is arranged at the center of each of the plurality of patch images.
Attribute judgment device. The patch image data generation unit generates a plurality of the patch image data so that the size of the patch image data is the same as each of the plurality of sizes of the library image data.
The attribute determination device according to claim 1. It is provided with an imaging status data acquisition unit that acquires imaging status data indicating the status at the time of acquiring image data of the object.
The attribute determination unit selects a specific classifier from the plurality of classifiers based on the imaging status data, and determines the attribute of the target pixel of interest based on the selected classifier.
The attribute determination device according to claim 2. Acquiring target image data indicating a target image including an object, and
Generating patch image data indicating a plurality of patch images of different sizes including the target pixel of interest from the target image data, and
Extracting the feature amount of the target pixel of interest in the patch image data,
Using the feature amount calculated from the patch image data and a plurality of classifiers generated in advance based on the feature amount calculated from each of the plurality of library image data of different sizes including the learning attention pixel, respectively. and determining an attribute of the object pixel of interest, only including,
Each of the plurality of patch images is a partial image of the target image cut out from the target image.
A plurality of patch image data of different sizes are generated so that the target pixel of interest is arranged at the center of each of the plurality of patch images.
Attribute judgment method. On the computer
Acquiring target image data indicating a target image including an object, and
Generating patch image data indicating a plurality of patch images of different sizes including the target pixel of interest from the target image data, and
Extracting the feature amount of the target pixel of interest in the patch image data,
Using the feature amount calculated from the patch image data and a plurality of classifiers generated in advance based on the feature amount calculated from each of the plurality of library image data of different sizes including the learning attention pixel, respectively. and determining an attribute of the object pixel of interest, is the execution,
Each of the plurality of patch images is a partial image of the target image cut out from the target image.
A plurality of patch image data of different sizes are generated so that the target pixel of interest is arranged at the center of each of the plurality of patch images.
Computer program. On the computer
Acquiring target image data indicating a target image including an object, and
Generating patch image data indicating a plurality of patch images of different sizes including the target pixel of interest from the target image data, and
Extracting the feature amount of the target pixel of interest in the patch image data,
Using the feature amount calculated from the patch image data and a plurality of classifiers generated in advance based on the feature amount calculated from each of a plurality of library image data having different sizes including learning attention pixels. A computer program that determines the attributes of the target pixel of interest and executes it is recorded .
Each of the plurality of patch images is a partial image of the target image cut out from the target image.
A plurality of patch image data of different sizes are generated so that the target pixel of interest is arranged at the center of each of the plurality of patch images.
recoding media.

Images