JP7716068B2 - Image object detection device, image object detection method, and program - Google Patents

Description

The present invention relates to an image object detection device, an image object detection method, and a program.

In recent years, numerous technologies have been proposed that use deep learning to detect objects in images. It is expected that such object detection technologies will be used in the field of autonomous driving in the future. It has been reported that the object detection technologies proposed to date have excellent object detection capabilities for images taken in good weather conditions, such as sunny skies. However, in order to apply these technologies to the field of autonomous driving, they must have object detection capabilities in a variety of weather conditions that are equivalent to those in good weather.

Claudio Michaelis et al. ,”Benchmarking Robustness in Object Detection : Autonomous Driving when Winter is Coming”,[online], 31th March 2020, arXiv preprint, arXiv:1907.07484, [Retrieved March 17, 2020], Internet <URL: https://arxiv.org/pdf/1907.07484.pdf>

However, in harsh weather conditions, low-light locations, or nighttime environments, the quality of captured images can deteriorate due to increased sensor noise in the camera's imaging element. Therefore, when object detection is performed using previously proposed object detection technologies on images captured in poor weather conditions, such as rain or fog, the accuracy of object detection is lower than when images captured in good weather conditions are used (see, for example, Non-Patent Document 1).

The present invention aims to provide technology that can detect objects from images taken in poor weather conditions with the same accuracy as in good weather conditions.

One aspect of the present invention is an image object detection device that includes a weather feature model generation unit that generates a weather feature model that indicates the characteristics of the weather displayed in an image; a feature map generation unit that generates a feature map from image data of a target for object detection; a combination unit that generates a feature map in which the influence of weather is suppressed by causally combining the weather feature model generated by the weather feature model generation unit with the feature map generated by the feature map generation unit; and an object detection unit that detects objects from the feature map generated by the combination unit.

One aspect of the present invention is an image object detection method including: a weather feature model generation step of generating a weather feature model that indicates the characteristics of the weather displayed in an image; a feature map generation step of generating a feature map from image data of a target for object detection; a combination step of causally combining the weather feature model generated in the weather feature model generation step with the feature map generated in the feature map generation step to generate a feature map in which the influence of weather is suppressed; and an object detection step of detecting objects from the feature map generated in the combination step.

One aspect of the present invention is a program that causes a computer to function as weather feature model generation means that generates a weather feature model that indicates the characteristics of the weather displayed in an image, feature map generation means that generates a feature map from image data of a target for object detection, combination means that generates a feature map in which the influence of weather is suppressed by causally combining the weather feature model generated by the weather feature model generation means with the feature map generated by the feature map generation means, and object detection means that detects objects from the feature map generated by the combination means.

This invention makes it possible to detect objects from images taken in poor weather conditions with the same accuracy as in good weather conditions.

1 is a block diagram showing a configuration of an image object detection device according to a first embodiment. FIG. 2 is a block diagram showing a detailed configuration of the image object detection device according to the first embodiment. FIG. 4 is a diagram illustrating a processing flow by a weather feature model generation unit of the first embodiment. FIG. 4 is a diagram illustrating an outline of processing by a weather feature model generation unit according to the first embodiment. FIG. 4 is a diagram illustrating a processing flow of object detection according to the first embodiment. FIG. 10 is a diagram illustrating an outline of processing by a combining unit according to the first embodiment. FIG. 10 is a block diagram showing the configuration of an image object detection device according to a second embodiment. FIG. 10 is a diagram illustrating a processing flow by a weather feature model generation unit of the second embodiment.

(First embodiment)
[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing the configuration of an image object detection device 1 according to a first embodiment. The image object detection device 1 includes a weather feature model generation unit 11, a weather feature model storage unit 12, an image data storage unit 13, a feature map generation unit 14, a combination unit 15, and an object detection unit 16.

The weather feature model generation unit 11 generates a weather feature model that indicates the characteristics of the weather displayed in the image. The weather feature model storage unit 12 stores the weather feature model generated by the weather feature model generation unit 11. The image data storage unit 13 stores image data that is the target of object detection. The feature map generation unit 14 generates a feature map from the image data stored in the image data storage unit 13. The combination unit 15 combines the feature map generated by the feature map generation unit 14 with the weather feature model stored in the weather feature model storage unit 12 to generate a feature map that suppresses the effects of weather. Here, the process of combining the feature map and the weather feature model performed by the combination unit 15 is a combination process using causal intervention, which is an intervention in a so-called structural causal model. By performing this process, if the image data corresponding to the feature map is, for example, image data taken in rainy conditions, the effects of rain in the image data are suppressed. The object detection unit 16 performs object detection processing to detect the type, position, and range of objects contained in the image data corresponding to the feature map generated by the combination unit 15.

Figure 2 is a block diagram showing an example of the detailed configuration of the image object detection device 1, illustrating an example in which the mask R-CNN (Region-based Convolutional Neural Network) method is applied as the object detection method. The image data storage unit 13, for example, stores in advance image data to be used for object detection. The image data is, for example, color image data, and is two-dimensional image data for each of the R (Red), G (Green), and B (Blue) channels, forming a three-dimensional array in which three channels of two-dimensional image data of the same size are overlaid. Here, size refers to the number of vertical and horizontal pixels in the two-dimensional image data, or in other words, the size determined by the number of vertical and horizontal data elements in the two-dimensional array data.

The feature map generation unit 14 includes a CNN (Convolutional Neural Network) layer 21, which performs a convolution operation on the image data stored in the image data storage unit 13 to generate a feature map 50. Here, the feature map 50 is defined by the following equation (1):

In equation (1), "F" represents the feature map 50, and "R" is a mathematical symbol representing real space. The superscript "d" of R represents the number of dimensions, and "c" represents the number of channels. In other words, equation (1) indicates that the feature map 50 is expressed as an element of a dxc-dimensional real space. Here, the CNN layer 21 performs a convolution operation on data in which three pieces of two-dimensional image data stored in the image data storage unit 13 are superimposed in the channel direction. Therefore, the number of dimensions "d" of the feature map 50 obtained by the convolution operation is d = 2. The number of channels "c" is a value that corresponds to the number of channels of the filter applied in the CNN layer 21, etc.

More specifically, the CNN layer 21 may employ a deep neural network such as a fully convolutional network (FCN) that includes only a CNN, or a deep neural network that includes a CNN such as a visual geometry group (VGG) or a residual network (ResNet) and a neural network other than a CNN. The feature map 50 obtained by the convolution operation by the CNN layer 21 is a three-dimensional array of data in which two-dimensional arrays of the same size for the number of channels "c" are superimposed in the channel direction. The superimposed two-dimensional array data has the same number of data columns and columns. The size of the two-dimensional array data is a reduced size of the image data read by the feature map generation unit 14 from the image data storage unit 13, and corresponds to the size of the image data read by the feature map generation unit 14 from the image data storage unit 13 and the size of the filter in the CNN layer 21.

The object detection unit 16 includes an RPN (Region Proposal Network) layer 22 and a Box Head unit 23. The RPN layer 22 is a neural network that detects the location and range of an object in the feature map 50. The Box Head unit 23 includes an RoI (Region of Interest)-Align layer 31, fully connected layers 32, 33-1, and 33-2, and CNN layers 34 and 35. The RoI-Align layer 31 is a neural network that extracts areas where objects exist in the feature map 51 based on the feature map 51 generated by the combining unit 15 (i.e., the feature map 51 in which the effects of weather have been suppressed) and the output of the RPN layer 22, performs pooling processing on the data for the extracted areas, and outputs data indicating the results of the pooling processing.

The fully connected layers 32, 33-1, and 33-2 are fully connected neural networks. When data output by the RoI-Align layer 31 is provided to the fully connected layer 32, the fully connected layers 32 and 33-1 calculate class data indicating the type of object from the provided data. The output stage of the fully connected layer 33-1 is a Softmax function, and the fully connected layers 32 and 33-1 calculate data indicating the object's class, probabilistically indicating which of multiple predetermined classes the object belongs to. When data output by the RoI-Align layer 31 is provided to the fully connected layer 32, the fully connected layers 32 and 33-2 calculate data indicating the position and range of the object, or the position and range of the so-called bounding box, from the provided data.

CNN layers 34 and 35 are neural networks that perform convolutional operations. When data output by RoI-Align layer 31 is provided to CNN layer 34, CNN layers 34 and 35 output data indicating the type of mask to apply to the object portion from the provided data. Note that trained weights and biases are applied to the neurons of the neural networks that make up CNN layer 21, RPN layer 22, fully connected layers 32, 33-1, 33-2, and CNN layers 34 and 35 before object detection processing is performed.

The weather feature model generation unit 11 includes a weather image data storage unit 41, a feature map generation unit 42, a classification unit 43, a detection unit 44, and a synthesis unit 45. The weather image data storage unit 41 pre-stores multiple image data captured under various weather conditions. The image data stored in the weather image data storage unit 41 is RGB color image data, similar to the image data stored in the image data storage unit 13.

Like the feature map generation unit 14, the feature map generation unit 42 has a CNN layer such as FCN, VGG, or ResNet, and generates a feature map from each of the multiple image data stored in the weather image data storage unit 41. The classification unit 43 clusters the multiple feature maps generated by the feature map generation unit 42. The detection unit 44 detects the central feature vector of each cluster clustered by the classification unit 43. The synthesis unit 45 synthesizes the central feature vectors detected by the detection unit 44 to generate a weather feature model 70, which is a model that collectively represents various weather features. Here, the weather feature model 70 is defined by the following equation (2).

In equation (2), "W" represents the weather feature model 70, and equation (2) indicates that the weather feature model 70 is expressed as an element of a dxh-dimensional real space. In equation (2), the number of dimensions "d" is d = 2, the same as the feature map 50. The synthesis unit 45 generates the weather feature model 70 by synthesizing the central feature vectors so that two-dimensional array data of the same size as the two-dimensional array data for one channel of the feature map 50 becomes three-dimensional array data in which the number of channels "h" of data are superimposed. The number of channels "h" is the number of clusters obtained when the classification unit 43 performs clustering, i.e., the number of weather types.

(Processing by the weather feature model generation unit of the first embodiment)
The processing by the weather feature model generation unit 11 of the first embodiment will be described with reference to FIGS. 3 and 4. Before the processing shown in FIG. 3 is started, a plurality of image data captured under a plurality of predetermined weather conditions is written in advance to the weather image data storage unit 41. As an example, as shown in FIG. 4, the weather image data storage unit 41 is assumed to have a plurality of image data pre-written therein, including rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, and lightly cloudy image data 52-5, each captured under each of the following weather conditions: "rain,""fog,""snow,""cloudy," and "slightly cloudy." While FIG. 4 shows one rain image data 52-1 as image data for rainy weather, it is assumed that a plurality of image data for each weather type is stored in the weather image data storage unit 41.

Rainy image data 52-1, foggy image data 52-2, snowy image data 52-3, cloudy image data 52-4, slightly cloudy image data 52-5, etc. are image data captured at arbitrary locations, and the more image data captured at different locations, the more general the weather feature model 70 that is ultimately obtained. However, some image data may contain image data captured at the same location.

The feature map generation unit 42 reads out the rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, lightly cloudy image data 52-5, etc. stored in the weather image data storage unit 41 one by one, and performs a convolution operation on each of the read image data to generate a feature map. That is, as shown in FIG. 4, the feature map generation unit 42 generates feature map 53-1 for the rain image data 52-1. The feature map generation unit 42 also generates corresponding feature maps 53-2, 53-3, 53-4, 53-5, etc. for each of the fog image data 52-2, snow image data 52-3, cloudy image data 52-4, lightly cloudy image data 52-5, etc. other than the rain image data 52-1. The feature map generation unit 42 outputs the generated feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ... to the classification unit 43 (step Sa1).

The classification unit 43 imports the feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ... output by the feature map generation unit 42 and clusters the imported feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ... using, for example, a Gaussian Mixture Model (GMM). For example, the classification unit 43 plots the feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ... in a vector space, as shown in the scatter diagram 60 of Figure 4. In the scatter diagram 60, the marks "◯", "☆", "◇", "□", and "△" represent the plot results of the feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ....

As mentioned above, multiple image data items are stored in the weather image data storage unit 41 for each weather type, so for example, there will be multiple rain feature maps in addition to feature map 53-1. In the scatter plot 60, the positions indicated by each rain feature map, including feature map 53-1, are indicated with a "◯" mark. Similarly, the positions indicated by each fog feature map, including feature map 53-2, are indicated with a "☆" mark. The positions indicated by each snow feature map, including feature map 53-3, are indicated with a "◇" mark. The positions indicated by each cloudy feature map, including feature map 53-4, are indicated with a "□" mark. The positions indicated by each lightly cloudy feature map, including feature map 53-5, are indicated with a "△" mark.

When plotted in vector space, the plotted points are not classified by weather type, such as "◯," "☆," "◇," "□," or "△." However, by performing clustering, the classification unit 43 classifies the plotted points into weather types. Note that, for ease of understanding, the scatter plot 60 shown in FIG. 4 shows an example in which clustering is performed in a two-dimensional vector space. However, the feature maps 53-1, 53-2, 53-3, 53-4, 53-5, ... that are the subject of clustering are three-dimensional array data. Therefore, clustering by the classification unit 43 may also be performed in a multidimensional vector space with more than two dimensions. The classification unit 43 outputs data indicating the clustering results to the detection unit 44 (step Sa2).

The detection unit 44 imports data indicating the clustering results output by the classification unit 43 and detects central feature vectors 54-1, 54-2, 54-3, 54-4, and 54-5 of each cluster based on the imported data. The detection unit 44 outputs data indicating each of the detected central feature vectors 54-1, 54-2, 54-3, 54-4, and 54-5 to the synthesis unit 45 (step Sa3). The synthesis unit 45 imports data indicating each of the central feature vectors 54-1, 54-2, 54-3, 54-4, and 54-5 output by the detection unit 44 and synthesizes the imported data to generate a weather feature model 70, which is a three-dimensional array of data and collectively represents the characteristics of five types of weather: "rain," "fog," "snow," "cloudy," and "partly cloudy." Here, the process of synthesizing the data representing each of the central feature vectors 54-1, 54-2, 54-3, 54-4, and 54-5 is, for example, the process of calculating the inner product of each of the central feature vectors 54-1, 54-2, 54-3, 54-4, and 54-5 (step Sa4). The synthesizing unit 45 writes and stores the data for the generated weather feature model 70 in the weather feature model storage unit 12 (step Sa5).

(Object detection process in the first embodiment)
The object detection process by the image object detection device 1 of the first embodiment will be described with reference to Figures 5 and 6. Before the process shown in Figure 5 starts, the weather feature model generation unit 11 generates a weather feature model 70, and data of the weather feature model 70 generated by the weather feature model generation unit 11 is written to the weather feature model storage unit 12. Learned weights and biases are applied to each neuron of the neural networks that make up the CNN layer 21, the RPN layer 22, the fully connected layers 32, 33-1, 33-2, and the CNN layers 34, 35.

The feature map generation unit 14 reads image data from the image data storage unit 13 and performs a convolution operation on the read image data using the CNN layer 21 to generate a feature map 50. The feature map generation unit 14 outputs the generated feature map 50 to the combination unit 15 and the object detection unit 16 (step Sb1). The combination unit 15 imports the feature map 50 output by the feature map generation unit 14. The combination unit 15 reads data of the weather feature model 70 from the weather feature model storage unit 12. The combination unit 15 performs a process of combining the feature map 50 and the weather feature model 70, i.e., a process of combining the causal interventions shown in the following equation (3).

In equation (3), the circumflex F on the left side represents the feature map 51 generated by the combiner 15. The first term on the right side is an equation that applies a softmax function to the product of the transpose of the feature map 50 (F ^T ) and the weather feature model 70 (W), i.e., the product of two three-dimensional arrays. Here, transposing the feature map 50 means swapping the elements of the feature map 50 as follows. Specifically, assume that the feature map 50 is a three-dimensional array of data (X×Y×c), and each element of the two-dimensional array of data (X and Y) for each of the c channels is represented by (x, y), where x = 1 to X and y = 1 to Y. Transposing the feature map 50 means swapping the elements (x, y) of the two-dimensional array data for each channel in the feature map 50 so that the elements become (y, x) while maintaining the channel direction.

According to the first term on the right-hand side of equation (3), when h = c, in other words, when the number of channels h of the weather feature model 70 (i.e., weather feature model W) matches the number of channels c of the feature map 50 (i.e., feature map F), a three-dimensional array of data of the same size and number of channels as the feature map 50 is obtained, where the values of elements in each channel whose quality has deteriorated due to the effects of weather are large. For example, as shown in FIG. 6, suppose that the feature map 50 is a three-dimensional array of data in which 4x4 two-dimensional array data are superimposed in a number equal to the number of channels "c," and the weather feature model 70 is a three-dimensional array of data in which 4x4 two-dimensional array data are superimposed in a number equal to the number of channels "h."

When h = c, the combiner 15 calculates the product of the three-dimensional array data obtained by transposing the feature map 50 and the three-dimensional array data of the weather feature model 70, thereby obtaining three-dimensional array data of the same size and number of channels as the feature map 50, i.e., 4 x 4 x c. The combiner 15 applies a softmax function to the 16 element values for each channel of the calculated three-dimensional array data, and sets each of the output values of the softmax function to the values of the 16 elements. As a result, the sum of the values of the 16 elements is "1," and c pieces of two-dimensional array data are obtained in which elements in the feature map 50 whose quality has deteriorated due to the effects of weather have large values. The combiner 15 overlays the c pieces of two-dimensional array data in the channel direction, thereby generating three-dimensional array data 71 of the same size and number of channels as the feature map 50.

The "x" on the right-hand side of equation (3) is an operation indicating the Hadamard product for each channel, and the combination unit 15 performs an operation to multiply, for each channel, the values of elements at corresponding positions in the three-dimensional array data 71 obtained as a result of equation 1 and the three-dimensional array data of the feature map 50. In other words, the combination unit 15 multiplies the value of the element at row x and column y of the zth channel of the three-dimensional array data 71 obtained as a result of equation 1 by the value of the element at row x and column y of the zth channel of the feature map 50, and sets the result as the value of the pixel at row x and column y of the zth channel of the feature map 51. Here, x, y, and z are positive integers. In the example shown in FIG. 6, x = 1 to 4, y = 1 to 4, and z = 1 to c. As a result, the values of elements in the feature map 50 whose quality has deteriorated due to the effects of weather are emphasized, resulting in a feature map 51 in which the effects of weather are suppressed.

On the other hand, if the number of channels h of the weather feature model 70 is not h = c, the above procedure cannot calculate the product of the feature map 50 and the weather feature model 70. When h = c is not true, the procedure by which the combiner 15 generates the feature map 51 is as follows. The portion of the feature map 50 containing the weather features for each channel is designated WF (hereinafter referred to as the weather feature portion WF). Here, the weather feature portion WF is a two-dimensional array of data defined by the following equation (4), in other words, one channel's worth of three-dimensional array data. If the feature map 50 is a 4 x 4 x c three-dimensional array of data as shown in Figure 6, the weather feature portion WF will be a 4 x 4 x 1 three-dimensional array of data.

The combining unit 15 calculates the weather feature portion WF for c channels using the following equation (5):

In equation (5), j = 1 to c, and WF _j is the weather feature part WF of the j-th channel of the feature map 50 having c channels. i ranges from 1 to h, and m _{j,i indicates} the j-th m _i , which can be calculated by the following equation (6). In equation (6), "F" on the right side indicates the j-th channel of the feature map 50 (i.e., feature map F).

In equations (5) and (6), "W _,i " is the feature value of the i-th channel of the weather feature model 70 (i.e., the weather feature model W). Since _mi is defined by the following equation (7), mj _,i , i.e., matrix m, is defined by the following equation (8).

As shown in equation (8), matrix m indicates the correlation between the h feature quantities of the weather feature model 70 and the c feature quantities of the feature map 50. That is, equation (6) is an equation for calculating a correlation matrix m i as the similarity between the jth feature of the feature map 50 (i.e., feature map _F ) of the image data of the object detection target and the weather feature model 70 (i.e., weather feature model W) obtained by combining the central feature vectors of all weather types. By calculating this similarity for each of the c channels of the feature map 50, a correlation matrix m _j,i is obtained. Using this correlation matrix m _j,i as the weight of the weather feature indicating the weather characteristics, a weighted sum of the weather features is calculated using equation (5), and WF _j , which indicates the jth feature of the feature map 50 that is influenced by the weather, is calculated. Because the number of channels of the feature map 50 is c, the combining unit 15 calculates c WF _j .

The combining unit 15 uses the feature map 50 (i.e., feature map F) and a three-dimensional array of data in which the calculated c weather feature parts WF _j (j = 1 to c) are superimposed in the channel direction to perform residual calculations shown on the right side of the following equation (9) for each of the c channels, and removes the influence of weather from the feature map 50 to generate the feature map 51 shown on the left side.

The combining unit 15 outputs the generated feature map 51 to the object detection unit 16 (step Sb2). Note that the above procedure for when h = c is not true can also be applied when h = c.

The object detection unit 16 takes in the feature map 50 output by the feature map generation unit 14 and the feature map 51 output by the combination unit 15, and performs the object detection process described below. The RPN layer 22 takes in the feature map 50 and detects the location of an object. The RoI-Align layer 31 takes in the feature map 51 and the output of the RPN layer 22, and extracts the portion of the feature map 51 where the object is located based on the output of the RPN layer 22. The RoI-Align layer 31 performs pooling processing on the data of the extracted portion, and outputs data indicating the results of the pooling processing to the fully connected layer 32 and the CNN layer 34.

When the fully connected layer 32 takes in the data output by the RoI-Align layer 31, the fully connected layer 33-1 outputs data indicating the class of object contained in the area indicated by the taken-in data, and the fully connected layer 33-2 outputs data indicating the position and range of the object contained in the area indicated by the taken-in data. When the CNN layer 34 takes in the data output by the RoI-Align layer 31, the CNN layer 35 outputs data indicating the type of mask to apply to the portion of the object contained in the area indicated by the taken-in data (step Sb3).

The feature map generation unit 14 applies data indicating the object class, data indicating the position and range in which the object exists, and data indicating the type of mask to be applied to the object to the image data read from the image data storage unit 13. As a result, for example, each object displayed in the image data is surrounded by a bounding box, data indicating the object class is displayed in association with the bounding box, and further, parts of each object are masked in different colors.

In the image object detection device 1 of the first embodiment described above, the weather feature model generation unit 11 generates a weather feature model 70 that indicates the features of the weather displayed in the image. The feature map generation unit 14 generates a feature map 50 from image data of the object detection target. The combination unit 15 causally combines the weather feature model 70 generated by the weather feature model generation unit 11 with the feature map 50 generated by the feature map generation unit 14 to generate a feature map 51 that suppresses the influence of weather. The object detection unit 16 performs object detection from the feature map 51 generated by the combination unit 15. The weather feature model 70 generated by the weather feature model generation unit 11 is a model that represents weather-specific information contained in features extracted from images captured under various weather conditions, not for each weather type, but for all weather types collectively. In other words, it can be said to be a general-purpose model that comprehensively represents a variety of weather types. By causally combining the weather feature model 70 having such characteristics with the feature map 50 of the image data of the object detection target by the combining unit 15, a feature map 51 in which the influence of weather is suppressed is obtained. Therefore, by performing object detection processing on the feature map 51 in which the influence of weather is suppressed, the object detection unit 16 can detect objects with the same accuracy as in good weather, even if the image data of the object detection target was captured in poor weather conditions. Because the weather feature model 70 is a general-purpose model for various weather types, even if the image data of the object detection target is corrupted by the influence of any type of weather, the influence of any type of weather can be suppressed regardless of the type or extent of the corruption. Therefore, applying the weather feature model 70 makes it possible to improve the robustness of object detection.

In the first embodiment described above, the classification unit 43 performs clustering using a Gaussian mixture model, but clustering may also be performed using other clustering methods, such as the K-means method.

In the first embodiment described above, five weather types, "rain," "fog," "snow," "cloudy," and "slightly cloudy," are shown as examples of weather types for image data stored in the weather image data storage unit 41. However, it is sufficient if image data for at least two different weather types is stored in the weather image data storage unit 41. Image data captured in weather conditions other than "rain," "fog," "snow," "cloudy," and "slightly cloudy" may also be stored in the weather image data storage unit 41. In this case, the detection unit 44 detects each of the clusters generated by the clustering performed by the classification unit 43, i.e., the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5, ... corresponding to each weather type.

Second Embodiment
7 is a block diagram showing the configuration of an image object detection device 1a according to the second embodiment. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and only the different components will be described below. The image object detection device 1a includes a weather feature model generation unit 11a, a weather feature model storage unit 12, an image data storage unit 13, a feature map generation unit 14, a combination unit 15, and an object detection unit 16.

In the first embodiment, the weather feature model generation unit 11 generated the weather feature model 70 from image data of various types of weather, i.e., rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, slightly cloudy image data 52-5, etc. When photographing in rainy or foggy weather conditions, it is observed that the effects of the weather are uniformly distributed across each pixel in the photographed image. Based on this, the weather feature model generation unit 11a in the second embodiment generates a random noise image in which noise appears uniformly across each pixel, rather than generating the weather feature model 70 from image data of actually photographed weather, and generates a weather feature model from the generated random noise image. For ease of explanation, the weather feature model in the second embodiment is designated by the symbol 70a and will hereinafter be referred to as the weather feature model 70a.

(Processing by the weather feature model generation unit of the second embodiment)
8 is a flowchart showing the flow of processing by the weather feature model generation unit 11a of the second embodiment. The weather feature model generation unit 11a generates "c" pieces of two-dimensional image data of random noise, each of which has the same size as the two-dimensional array data of each channel of the feature map 50 (step Sc1). The weather feature model generation unit 11a generates three-dimensional array data by superimposing the "c" pieces of generated two-dimensional image data of random noise in the channel direction as data for the weather feature model 70a (step Sc2).

Note that random noise must be uniformly distributed within a single two-dimensional image. Therefore, the weather feature model generation unit 11a generates two-dimensional image data so that the pixel values of each pixel in the two-dimensional image data are normally distributed random numbers. Specific examples of random noise include Gaussian noise, which is irregular noise with a normally distributed distribution, and white Gaussian noise is particularly desirable. The weather feature model 70a is defined by the following equation (10):

As can be seen from equation (10), the weather feature model 70 of the first embodiment and the weather feature model 70a of the second embodiment are both elements of a d-dimensional real number space, which is the same number of dimensions, but are three-dimensional array data with different numbers of channels. The weather feature model 70a of the second embodiment is three-dimensional array data expressed as elements of a dxc-dimensional real number space with the same size and number of channels as the feature map 50. The weather feature model generation unit 11a writes and stores the data of the generated weather feature model 70a in the weather feature model storage unit 12 (step Sc3).

The object detection process in the second embodiment is the same as the object detection process in the first embodiment shown in Figure 5. However, the weather feature model to which the combining unit 15 performs causal combination on the feature map 50 generated by the feature map generating unit 14 is the weather feature model 70a of the second embodiment. As a result, in the second embodiment, as in the first embodiment, it is possible to perform object detection while suppressing the influence of weather in the feature map 50 obtained from image data of the object detection target.

In the second embodiment described above, the weather feature model generation unit 11a may generate h random noises, a number different from c, and generate the weather feature model 70a from the generated h random noises. In this case, the combination unit 15 generates the feature map 51 from the feature map 50 and the weather feature model 70a using the procedure for when h = c, as described in the first embodiment.

In the first and second embodiments described above, an example is shown in which mask R-CNN is applied as a specific example of the object detection unit 16. However, other object detection methods using deep neural networks, such as Faster R-CNN or YOLO (You Only Look Once), may also be applied to the object detection unit 16. When applying other object detection methods, similar to the example in which mask R-CNN is applied, a combining unit 15 is inserted at a location where a feature map corresponding to the feature map 50 is obtained as an output, and the weather feature models 70, 70a are combined with the feature map 50.

In the first and second embodiments described above, the image data stored in the weather image data storage unit 41 and the image data storage unit 13 is RGB color image data. In contrast, the image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be CMYK color image data or grayscale image data. The image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be image data captured by the same camera, or may be image data captured by different cameras. However, the images must be captured so that the domains of both image data are the same, so that there is no difference in domain, such as one being RGB color image data and the other being grayscale image data.

In the first and second embodiments described above, the number of vertical and horizontal pixels of the image data stored in the image data storage unit 13 may be the same or different. In the first embodiment described above, the size of the image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be the same or different. The number of vertical and horizontal pixels of the image data stored in the weather image data storage unit 41 may be the same or different. However, as described above, the size of the two-dimensional array data for each channel of the feature map 50 must match the size of the two-dimensional array data for each channel of the weather feature models 70 and 70a. Therefore, the CNN layer 21 provided in the feature map generation unit 14 and the weather feature model generation units 11 and 11a must be configured so that the size of the two-dimensional array data for each channel of the feature map 50 and the weather feature models 70 and 70a match.

The image object detection device 1, 1a in the above-described embodiments may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or over communication lines such as telephone lines, or media that store programs for a fixed period of time, such as volatile memory within the computer system that serves as the server or client. The program may also be a program that implements some of the above-described functions, or may be a program that can implement the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and includes designs that do not deviate from the gist of the present invention.

1... Image object detection device, 11... Weather feature model generation unit, 12... Weather feature model storage unit, 13... Image data storage unit, 14... Feature map generation unit, 15... Combination unit, 16... Object detection unit

Claims (8)

a weather feature model generation unit that generates a weather feature model that indicates the features of the weather displayed in the image;
a feature map generation unit that generates a feature map from image data of an object detection target;
a combining unit that generates a feature map in which the influence of weather is suppressed by causally combining the weather feature model generated by the weather feature model generating unit with the feature map generated by the feature map generating unit;
an object detection unit that performs object detection from the feature map generated by the combination unit;
An image object detection device comprising: The weather feature model generation unit
generating a feature map from each of a plurality of image data in which weather conditions corresponding to each of a plurality of weather types are captured, clustering the generated feature maps, and generating the weather feature model by combining central feature vectors of each cluster obtained by the clustering;
The image object detection device according to claim 1 , comprising: The weather feature model generation unit
generating the weather feature model from an image of random noise;
The image object detection device according to claim 1 , comprising: The random noise is white Gaussian noise.
The image object detection device according to claim 3 . The coupling portion is
A Hadamard product of the feature map and data obtained by applying a softmax function to the product of the transpose of the feature map and the weather feature model is calculated, thereby causally combining the weather feature model with the feature map.
The image object detection device according to any one of claims 1 to 4. The coupling portion is
calculating a correlation matrix between each channel of the feature map and each channel of the weather feature model, extracting a portion of the feature map including weather features for each channel based on the calculated correlation matrix and the weather feature model, and performing a residual calculation to remove the weather features from the feature map using the extracted portion including the weather features and the feature map, thereby causally combining the weather feature model with the feature map;
The image object detection device according to any one of claims 1 to 4. a weather feature model generation step of generating a weather feature model indicating the features of the weather displayed in the image;
a feature map generation step of generating a feature map from image data of an object detection target;
a combining step of generating a feature map in which the influence of weather is suppressed by causally combining the weather feature model generated in the weather feature model generating step with the feature map generated in the feature map generating step;
an object detection step of detecting an object from the feature map generated by the combining step;
An image object detection method comprising: Computer,
a weather feature model generating means for generating a weather feature model indicating the features of the weather displayed in the image;
a feature map generating means for generating a feature map from image data of an object detection target;
a combining means for generating a feature map in which the influence of weather is suppressed by causally combining the weather feature model generated by the weather feature model generating means with the feature map generated by the feature map generating means;
an object detection means for detecting an object from the feature map generated by the combining means;
A program to function as a