JP7663341B2 - DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, DATA PROCESSING PROGRAM, AND LEARNING MODEL GENERATION METHOD - Google Patents

Description

The present invention relates to a data processing device, a data processing method, a data processing program, and a learning model generation method, and in particular to a data processing device, a data processing method, a data processing program, and a learning model generation method that perform data processing on target data that represents an object.

For purposes such as automatically recognizing a scene captured in an image, technologies have been researched and developed that divide an image into regions for each of multiple objects or multiple body parts captured in the image, and recognize the objects or body parts captured in each region. Hereinafter, the captured objects or body parts are referred to as subjects. Region division that involves subject recognition is called semantic segmentation.

In particular, in recent years, there has been active research into technologies that perform the above-mentioned segmentation and recognition based on learning. For example, Non-Patent Document 1 describes a method of preparing a large number of learning images in which a class representing the subject is assigned to each pixel in an area that has been segmented in advance for each subject, and having a computer perform machine learning on these learning images. The information that is assigned in advance is called an annotation or the like. If an arbitrary image is input into a trained model generated by this learning, a class for each pixel of the input image is output. In other words, the input image is segmented into areas (label areas) labeled with a class for each subject.

“Fully Convolutional Networks for Semantic Segmentation”, Jonathan Long, Evan Shelhamer, and Trevor Darrell (Proceedings of the IEEE conference on computer vision and pattern recognition, 2015)

Conventionally, in these machine learning methods, machine learning is often performed using a dataset consisting of learning images made of perspective projection images with little lens distortion and their annotations (see FIG. 11). On the other hand, perspective projection images show objects in a significantly different way from fisheye images taken by fisheye cameras used for surveillance (see FIG. 2). Therefore, when a fisheye image is input to a trained model created with a dataset of perspective projection images, the accuracy may be significantly reduced, particularly when there is a misalignment between the top and bottom of the image and the top and bottom of the real world, people cannot be detected correctly on the left and right or at the bottom of the fisheye image, or the floor may be mistaken for a wall (see FIG. 12). FIG. 12 shows an example of segmentation into incorrectly labeled regions as shown in FIG. 12(B) as a result of segmentation processing on the fisheye image shown in FIG. 12(A).

One way to prevent this loss of accuracy is, for example, to convert the fisheye image into a perspective projection image before processing it. In this case, multiple perspective projection images are obtained from a single fisheye image, so the processing time increases because all of these images need to be processed and then integrated, and it is also difficult to integrate the results when inconsistencies occur in the results.

Another approach would be to rotate the dataset 360 degrees in every direction during training, allowing the trained model to process objects in every orientation and therefore fisheye images. However, this would result in false positives, such as mistaking the top and bottom of a person for the bottom, as the system tries to find objects in every orientation across the entire fisheye image.

In addition, when using a convolutional neural network to estimate segmentation, it may be possible to estimate weights with high accuracy for fisheye images by rotating the filter used for convolution according to the tilt of the object in the image. However, the filters used in convolution operations generally have small widths and heights, and it is difficult to define continuous rotations for this weight array.

The present invention aims to provide a data processing device, a data processing method, a data processing program, and a learning model generation method that can process target data representing an object while taking into account the reference direction of the object.

In order to achieve the above object, the data processing device according to the present invention processes data using a filter that uses coefficients that have been previously learned using learning data representing an object and a direction on the learning data that corresponds to the reference direction of the object, and is configured to include a reception unit that receives object data representing an object and a direction on the object data that corresponds to the reference direction of the object, and a data processing unit that performs convolution processing on the object data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object, to perform data processing.

In the data processing device according to the present invention, the receiving unit receives target data representing an object and a direction on the target data that corresponds to a reference direction of the object. Then, the data processing unit performs data processing by performing a convolution process on the target data using a modified filter in which the coefficients of the filter are modified based on the direction that corresponds to the reference direction of the object.

In this way, by performing convolution processing on the target data using a modified filter whose coefficients have been changed based on the direction in the target data that corresponds to the reference direction of the object, data processing can be performed on the target data representing the object, taking into account the reference direction of the object.

The target data is data in which the reference direction of the object differs for each element, and the reception unit receives an angle map that stores a direction corresponding to the reference direction of the object for each element, and the data processing unit performs a convolution process on the target data using a modified filter in which the coefficient of the filter is modified based on the angle map, to perform data processing.

The data processing unit can also change the filter coefficients for each element based on the direction, and perform convolution processing on the target data using the changed filter for each element to process the data.

The data processing unit changes the coefficients of the filters corresponding to each of a plurality of specific directions, which are generated using a filter using the pre-learned coefficients, by weighting the filters according to the relative angle between the direction and the specific direction, and can perform data processing by performing a convolution process on the target data using the changed filters.

The data processing unit performs data processing by performing a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, which is generated using a filter using the pre-learned coefficients, and can integrate the results of the data processing using the filter corresponding to each of the plurality of specific directions by weighting them according to the relative angle between the direction corresponding to the reference direction of the target object and the specific direction.

The data processing unit can perform data processing by performing convolution processing on the target data using the modified filter as a filter for at least a portion of the layers of the convolutional neural network.

The target data may be a fisheye image taken by a fisheye camera.

The data processing method according to the present invention performs data processing using a filter using coefficients that have been previously trained using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, in which a receiving unit receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object, and a data processing unit performs data processing by performing a convolution process on the object data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object.

The data processing program according to the present invention is a data processing program for performing data processing using a filter using coefficients that have been previously trained using learning data representing an object and a direction on the learning data that corresponds to the reference direction of the object, and causes a computer to function as a reception unit that receives object data representing an object and a direction on the object data that corresponds to the reference direction of the object, and a data processing unit that performs data processing by performing a convolution process on the object data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object.

In the learning model generation method according to the present invention, a learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data that corresponds to a reference direction of the object, and a learning unit performs convolution processing on the learning data using a modified filter in which the coefficients of the filter are changed based on the direction, and learns so that the result of data processing matches the correct data processing result.

The data processing device, data processing method, data processing program, and learning model generation method of the present invention have the advantage that data processing can be performed on target data representing an object while taking into account the reference direction of the object.

1 is a schematic diagram showing a configuration of an image processing system according to an embodiment of the present invention; FIG. 2 is a diagram illustrating an example of a fisheye image. 1 is a schematic diagram showing a configuration of a learning device according to an embodiment of the present invention; 2 is a schematic diagram showing a configuration of a learning unit of the learning device according to the embodiment of the present invention; FIG. 1 is a schematic diagram showing a configuration of a data processing unit of a learning device and an image processing device according to an embodiment of the present invention. 1 is a schematic diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. 5 is a flowchart showing the operation of a learning process by the learning device according to the embodiment of the present invention. 4 is a flowchart showing an image processing operation by the image processing device according to the embodiment of the present invention. FIG. 1A is a diagram showing an example of a fisheye image, and FIG. 1B is a diagram showing an example of a segmentation processing result. FIG. 13 is a schematic diagram showing the configuration of a data processing unit of a learning device and an image processing device according to a modified example. FIG. 2 is a diagram illustrating an example of a perspective projection image. FIG. 1A is a diagram showing an example of a fisheye image, and FIG. 1B is a diagram showing an example of a segmentation processing result in the prior art.

Below, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the present invention will be described as being applied to an image processing system that classifies fisheye images taken by a fisheye photography device by subject class and divides them into label areas. A fisheye image is an example of target data, and a subject is an example of an object.

<System Configuration>
Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1 which shows a schematic configuration of an image processing system 1000 to which the present invention is applied.

(Image Processing System 1000)
The image processing system 1000 includes a fisheye photography device 1100, a network 1200, a learning device 100, and an image processing device 150. The image processing device 150 is an example of a data processing device.

(Fisheye photography device 1100)

The fisheye photography device 1100 captures a fisheye image as shown in FIG. 2. Specifically, the fisheye photography device 1100 is a surveillance camera installed for the purpose of monitoring a specified surveillance space, and is installed, for example, on a ceiling, with its optical center facing approximately vertically downward to capture images downward. In a fisheye image, the reference direction of the subject (e.g., the upward direction) differs for each pixel, and the reference direction of the subject for each pixel is determined in advance. The fisheye image captured by the fisheye photography device 1100 is transmitted to the image processing device 150. Note that a pixel is an example of an element.

(Network 1200)
The network 1200 is a line used for transmitting and receiving data between the fisheye photography device 1100, the image processing device 150, and the learning device 100. A local area network (LAN) or a public line such as the Internet can be used as the network 1200 of the present invention. It is desirable to take security measures for messages on the network 1200, such as encrypting the messages using known VPN technology or the like.

(Learning device 100)
The parameters of the model used in the image processing device 150 are learned by the learning device 100 .

The learning device 100 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and learns the parameters of a model for accepting a learning dataset and performing segmentation processing to classify the subjects of an image by class and divide them into label regions. The segmentation processing is an example of data processing.

As shown in FIG. 3, the learning device 100 functionally comprises a reception unit 10 and a calculation unit 20.

The receiving unit 10 receives a plurality of training data sets. For example, the training data set includes a set of training images of perspective projection images and correct answer labels for segmentation processing. Here, in the perspective projection images, the reference direction of the subject is the upward direction. The correct answer labels for segmentation processing are an array with the same width and height as the training images, with each element indicating the segmentation label of the corresponding image position.

As shown in FIG. 3, the calculation unit 20 includes a learning image generation unit 22, a learning unit 24, and a model storage unit 26.

The learning image generation unit 22, learning unit 24, and model storage unit 26 that make up the calculation unit 20 are described in detail below.

The training image generation unit 22 rotates the training images and the correct labels by a certain angle θ _aug , where θ _aug is selected randomly from 0°≦θ _aug <360°. The training image generation unit 22 also creates an angle map, which is an array with the same width and height as the training images, and outputs the map with the values of all elements set to θ _aug .

As shown in FIG. 4, the learning unit 24 includes a data processing unit 30, an error calculation unit 40, and an update unit 50.

The data processing unit 30, error calculation unit 40, and update unit 50 that make up the learning unit 24 are described in detail below.

The data processing unit 30 performs a segmentation process on the training image using the training image and the angle map. Here, the data processing unit 30 is assumed to be configured with a convolutional neural network, and inputs the training image to a convolutional neural network using a modified filter whose coefficients have been changed based on the angle map, performs a convolution process on the training image to perform a segmentation process, and outputs a segmentation result in which the training image is classified by subject class and divided into label regions.

The following provides a detailed explanation of each of the filter synthesis unit 32 and the convolution calculation unit 34 that constitute the data processing unit 30 shown in FIG. 5.

The data processing unit 30 holds, as base filters, a filter w _↑ for a region where the upper direction of the subject is the upper direction of the image, a filter w _↓ for a region where the lower direction of the subject is the upper direction of the image, a filter w _→ for a region where the right direction of the subject is the upper direction of the image, and a filter w _← for a region where the left direction of the subject is the upper direction of the image. These base filters are synthesized in the filter synthesis unit 32 based on the angle map. Note that the upper direction, the lower direction, the right direction, and the left direction are examples of the specific direction. Also, FIG. 5 shows an example of synthesizing N base filters based on the angle map.

The filter synthesis unit 32 changes the filter coefficients for each pixel based on the angle map values.

Specifically, if the angle map value at pixel i in the learning image is θ _i , the synthesis filter w _i is calculated as follows.

...(1)

As shown in the above formula (1), the data processing unit 30 obtains a composite filter by weighting the basis filters corresponding to each of the multiple specific directions according to the relative angle between the direction indicated by the angle map value and the specific direction, and by changing the filter coefficients, it is possible to create a filter that changes within the image according to the angle map value.

In addition, if the rotation of the filter used in the convolution is 90 degrees, 180 degrees, or 270 degrees, it can be defined by transposing the array and inverting it up and down and left and right. Therefore, for example, the data processing unit 30 can determine only w _↑ as a basis filter, and for w _← , w _↓ , and w _→ , by inverting and transposing w _↑ . In this case, it is possible to prevent an increase in the learning parameters, and to impose natural constraints on the relationship between the basis filters.

For each pixel in each layer, the convolution calculation unit 34 performs convolution processing on the learning image based on the synthesis filter calculated for that pixel by the filter synthesis unit 32, and outputs the results to the next layer.

The convolution calculation unit 34 finally outputs the accuracy of each class at each pixel as the segmentation processing result.

The error calculation unit 40 calculates the cross-entropy error at each pixel based on the accuracy of each class at each pixel and the rotated correct label.

The update unit 50 updates parameters including the filter coefficients of the convolutional neural network based on the calculated error.

The learning unit 24 minimizes the calculated error by repeating the processes of the data processing unit 30, the error calculation unit 40, and the update unit 50, and learns the parameters of the convolutional neural network that constitutes the data processing unit 30.

In this embodiment, the error is calculated for multiple learning data sets, and the average of the errors is used to determine the amount of parameter update. The angle by which the learning image is rotated in the learning image generation unit 22 may be different for each learning data set.

The model memory unit 26 stores the parameters of the model learned by the learning unit 24.

(Image Processing Device 150)
The image processing device 150 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and receives fisheye images sent by the fisheye photography device 1100, and performs segmentation processing to classify the fisheye images by subject class and divide them into label areas.

As shown in FIG. 6, the image processing device 150 functionally comprises a reception unit 60, a data processing unit 70, and an output unit 80.

The reception unit 60 receives the received fisheye image. In addition, the reference direction (upward) of the subject in the fisheye image is assumed to be known for each pixel, and the reception unit 60 receives an angle map that stores the reference direction of the subject in the fisheye image for each pixel.

To accept a fisheye image such as that shown in Figure 2 above, the value of pixel i in the angle map is set according to the following formula, where c is the center of the image.

Here, arctan2 is a function that satisfies the following:

The data processing unit 70 is equipped with a convolutional neural network trained by the learning device 100, and a fisheye image is input to the convolutional neural network using a modified filter whose filter coefficients have been changed based on the angle map. A convolution process is performed on the fisheye image to perform segmentation processing, and the segmentation results in which the fisheye image is classified by subject class and divided into label regions are output by the output unit 80.

Specifically, as shown in FIG. 5 above, the data processing unit 70 includes a filter synthesis unit 72 and a convolution calculation unit 74.

Similar to the filter synthesis unit 32, the filter synthesis unit 72 changes the filter coefficients for each pixel of the fisheye image based on the angle map values.

Specifically, the angle map value for pixel i in the fisheye image is θi, and the synthesis filter wi is calculated according to the above formula (1).

Similar to the convolution calculation unit 34, the convolution calculation unit 74 performs convolution processing on the fisheye image for each pixel in each layer based on the synthesis filter obtained by the filter synthesis unit 72, and outputs the results to the next layer.

The convolution calculation unit 74 finally determines the class with the highest probability for each pixel as the class of that pixel, and outputs the segmentation processing result via the output unit 80.

<Operation of image processing system>
The operation of the image processing system 1000 to which the present invention is applied will be described below with reference to the flowcharts shown in Figures 7 and 8. Note that the description will be given taking as an example a case in which a plurality of training data sets prepared in advance are input to the training device 100 by the receiving unit 10.

7 is executed in advance. In the learning process, the learning image generating unit 22 rotates the learning image and the correct label by a randomly selected angle θ _aug for each learning data set, and generates an angle map in which the value of each pixel is the angle θ _aug (step S100).

For each training data set, the data processing unit 30 performs segmentation processing on the training images using the training images and the angle map. Here, the data processing unit 30 inputs the training images to a convolutional neural network using a modified filter whose filter coefficients have been changed based on the angle map, performs convolution processing on the training images to perform segmentation processing, and outputs the segmentation results (step S102).

Then, for each learning dataset, the error calculation unit 40 calculates the cross-entropy error at each pixel based on the accuracy of each class at each pixel and the rotated correct label (step S104).

The update unit 50 updates various parameters, including the filter coefficients of the convolutional neural network, based on the error calculated for each learning dataset (step S106).

Then, it is determined whether or not a predetermined iteration end condition is satisfied (step S108). If the predetermined iteration end condition is not satisfied, the process returns to step S102, whereas if the predetermined iteration end condition is satisfied, the process proceeds to step S110.

The condition for ending the iterations can be when the calculated error value has converged or when the number of iterations has reached an upper limit.

Then, the final trained parameters of the convolutional neural network are stored in the model storage unit 26 (step S110), and the training process is terminated.

The parameters of the convolutional neural network learned by the learning device 100 are then set in the data processing unit 70 of the image processing device 150.

The image processing device 150 that receives the fisheye image from the fisheye photography device 1100 performs the image processing shown in FIG. 8. Note that the operation of the image processing device 150 in FIG. 8 described below is executed each time a fisheye image captured by the fisheye photography device 1100 is received.

First, the reception unit 60 receives the received fisheye image. The reception unit 60 also receives an angle map that stores the reference direction of the subject in the fisheye image for each pixel (step S110).

The filter synthesis unit 72 changes the filter coefficients for each pixel of the fisheye image based on the angle map values (step S112).

For each layer, the convolution calculation unit 74 performs convolution processing on the fisheye image based on the filter calculated by the filter synthesis unit 72 for each pixel, and outputs the results to the next layer. Finally, for each pixel, the convolution calculation unit 74 determines the class with the highest probability as the class of that pixel, and outputs the segmentation processing results from the output unit 80 (step S114).

As described above, in the image processing device 150 according to the embodiment of the present invention, a modified filter whose coefficients are changed based on the direction corresponding to the reference direction of the subject in the fisheye image is used to perform convolution processing on the fisheye image to perform segmentation processing, thereby making it possible to perform segmentation processing on the fisheye image while taking into account the reference direction of the subject. For example, as shown in FIG. 9, it is possible to perform correct segmentation processing on the fisheye image. This makes it unnecessary to perform processing to expand the fisheye image into multiple images. FIG. 9 shows an example in which the fisheye image shown in FIG. 9(A) is correctly divided into labeled regions as shown in FIG. 9(B) as a result of segmentation processing on the fisheye image shown in FIG. 9(A).

In addition, a machine learning model can be used that changes the processing depending on the orientation of the subject in the image by changing the filter coefficients used in the convolution process based on the direction that corresponds to the reference direction of the subject in the fisheye image.

<Modification>
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, in the present embodiment, the data processing units 30 and 70 are configured as shown in FIG. 5, but the present invention is not limited to this. For example, instead of the data processing units 30 and 70, data processing units 230 and 270 may be configured as shown in FIG. 10. FIG. 10 shows an example in which the data processing unit 230 of the learning device 100 includes N convolution calculation units 234 and an output integration unit 236 in each layer of the neural network, and the data processing unit 270 of the image processing device 150 includes N convolution calculation units 274 and an output integration unit 276 in each layer of the neural network.

The N convolution calculation units 234 in each layer of the data processing unit 230 of the learning device 100 perform convolution processing on an image using basis filters corresponding to each of a plurality of specific directions. For example, the first convolution calculation unit 234 performs convolution processing on the learning image based on the first basis filter for each pixel, and outputs the result to the corresponding output integration unit 236. Similarly, the Nth convolution calculation unit 234 performs convolution processing on the learning image based on the Nth basis filter for each pixel, and outputs the result to the corresponding output integration unit 236. Furthermore, the N convolution calculation units 274 in each layer of the image processing device 150 perform convolution processing on a fisheye image, similar to the convolution calculation unit 234 of the learning device 100.

The output integration unit 236 of the learning device 100 integrates the results of the convolution processing obtained from the corresponding N convolution calculation units 234 for each pixel by weighting them according to the relative angle between the reference direction of the subject of the pixel and the specific direction. For example, the output integration unit 236 integrates the results of the convolution processing as in the above formula (1) and outputs the integrated result to the next layer. Finally, based on the integration result in the last layer, the accuracy of each class at each pixel is output as the segmentation processing result. Also, the output integration unit 276 of the image processing device 150 integrates the results of the convolution processing performed by the N convolution calculation units 274, similar to the output integration unit 236 of the learning device 100. By such processing, it is possible to output the same results as the configuration of FIG. 5 above. In this configuration, since the filter does not change depending on the position when performing the convolution operation, it is possible to perform calculations using a GPU or the like at high speed.

The number of basis filters used in the data processing units 30 and 70 may be three or less, or may be five or more. In this case, for example, the basis filters may be arranged so that their orientations are equally spaced, and at any angle, a basis filter may be defined as the weighted sum of the two basis filters closest to that angle. By setting the number of basis filters to a multiple of four, a set of basis filters spaced at 90-degree intervals can be created, and within that set, the rotation relationship of the basis filters can be defined by transposing or inverting the basis filters.

In addition, the convolution operation using the angle map may be used in all layers of the convolutional neural network, or in only a part of it, such as only the first layer, only the first half, or only the second half.

The technology described in the above embodiment relates to convolution operations and can be used for data processing other than segmentation processing. For example, it can be applied to pose estimation and detection.

In addition, some or all of the images used for learning may be fisheye images. In this case, the angle map used in the data processing of the fisheye images in the image processing device 150 is input.

The technology described in the above embodiment can also be applied to three-dimensional convolution. In this case, for example, six basis filters can be taken for the positive and negative directions of the x, y, and z axes, and the angle map can be given as the coordinates of a point on a unit sphere.

Three-dimensional convolution is applied to the analysis of time-series images such as videos. Consider a space-time in which images are arranged in the x and y directions, and the time axis is set in the z direction, so that the time-series images are arranged three-dimensionally. In this case, when an object moves, the object will exist in various orientations and distortions in the space-time. The orientation of the object in this space-time can be found by calculating the interval between the capture of the time-series images and the distance traveled between two consecutive images. By inputting this orientation as an angle map and performing three-dimensional convolution, it becomes possible to analyze time-series images with higher accuracy. In addition, during learning, a single still image is repeatedly input and learned, and by changing the angle map during data processing, it becomes possible to respond to time-series images.

In addition, in this embodiment, an example has been described in which an image is used as the target data for data processing, but the present invention is not limited to this. For example, point cloud data measured using LiDAR (Light Detection and Ranging) may be processed as the target data. For example, as in the above embodiment, by using an angle map that stores the reference direction of the subject, it is possible to identify the subject from point cloud data from a LiDAR equipped with a fisheye lens.

In addition, the learning unit 24 may be provided in the image processing device 150, rather than in a learning device 100 separate from the image processing device 150.

As described above, those skilled in the art can make various modifications to suit the implementation form within the scope of the present invention.

10, 60 Reception unit 20 Calculation unit 22 Learning image generation unit 24 Learning unit 26 Model storage unit 30, 70, 230, 270 Data processing unit 32, 72 Filter synthesis unit 34, 74, 234, 274 Convolution calculation unit 80 Output unit 100 Learning device 150 Image processing device 236, 276 Output integration unit 1000 Image processing system 1100 Fisheye photography device 1200 Network

Claims (12)

1. A data processing device that processes data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object,
a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object;
a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter obtained by modifying a coefficient of the filter based on a direction corresponding to a reference direction of the target object;
Including,
The data processing unit changes the coefficients of the filters corresponding to each of a plurality of specific directions, which are generated using a filter using the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and performs a convolution process on the target data using the changed filters to process data. 1. A data processing device that processes data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object,
a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object;
a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter obtained by modifying a coefficient of the filter based on a direction corresponding to a reference direction of the target object;
Including,
the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing,
a data processing device that weights and integrates the results of the data processing using the filters corresponding to each of the plurality of specific directions in accordance with a relative angle between the direction corresponding to a reference direction of the object and the specific directions. The target data is data in which a reference direction of an object differs for each element,
The reception unit receives an angle map that stores a direction corresponding to a reference direction of the object for each of the elements;
3 . The data processing device according to claim 1 , wherein the data processing unit performs data processing by performing a convolution process on the target data using a filter in which a coefficient of the filter is changed based on the angle map. The data processing unit changes a coefficient of the filter for each of the elements based on the direction;
4. The data processing apparatus according to claim 3, further comprising: a data processing unit for performing a convolution process on the target data for each of the elements using the changed filter. The data processing device according to any one of claims 1 to 4, wherein the data processing unit performs data processing by performing convolution processing on the target data using the changed filter as a filter for at least a portion of a layer of a convolutional neural network. A data processing device according to any one of claims 1 to 5, wherein the target data is a fisheye image taken by a fisheye camera. 1. A data processing method for processing data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising:
a receiving unit receiving object data representing an object and a direction on the object data corresponding to a reference direction of the object;
a data processing unit performing data processing by performing a convolution process on the target data using a modified filter in which a coefficient of the filter is modified based on a direction corresponding to a reference direction of the target object;
The data processing method includes changing the coefficients of the filters corresponding to each of a plurality of specific directions, the filters being generated using the filters that use the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and performing a convolution process on the target data using the changed filters to process data. 1. A data processing method for processing data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising:
a receiving unit receiving object data representing an object and a direction on the object data corresponding to a reference direction of the object;
a data processing unit performing data processing by performing a convolution process on the target data using a modified filter obtained by modifying a coefficient of the filter based on a direction corresponding to a reference direction of the target object;
the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing,
A data processing method in which the results of the data processing using filters corresponding to each of the plurality of specific directions are weighted and integrated according to a relative angle between the direction corresponding to a reference direction of the object and the specific directions. 1. A data processing program for performing data processing using a filter using coefficients that have been learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising:
Computer,
a reception unit that receives target data representing an object and a direction on the target data that corresponds to a reference direction of the object; and a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter in which a coefficient of the filter is changed based on the direction that corresponds to the reference direction of the object, the data processing program comprising:
The data processing unit changes the coefficients of the filters corresponding to each of a plurality of specific directions, which are generated using a filter using the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and uses the changed filters to perform a convolution process on the target data and process the data. 1. A data processing program for performing data processing using a filter using coefficients that have been learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising:
Computer,
a reception unit that receives target data representing an object and a direction on the target data that corresponds to a reference direction of the object; and a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter in which a coefficient of the filter is changed based on the direction that corresponds to the reference direction of the object, the data processing program comprising:
the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing,
A data processing program that integrates the results of the data processing using the filters corresponding to each of the plurality of specific directions by weighting them according to a relative angle between the direction corresponding to a reference direction of the object and the specific directions. A learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data corresponding to a reference direction of the object;
a learning unit performs convolution processing on the learning data using a modified filter, the coefficients of which are changed based on a direction corresponding to a reference direction of the object, to learn so that a result of data processing matches the correct data processing result;
When the learning unit performs the data processing, the learning model generation method changes the coefficients of the filters generated using the filters that use the coefficients, by weighting the filters corresponding to each of a plurality of specific directions according to the relative angle between the direction corresponding to the reference direction of the object and the specific directions, and performs a convolution process on the learning data using the changed filters to process the data. A learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data corresponding to a reference direction of the object;
a learning unit performs convolution processing on the learning data using a modified filter, the coefficients of which are changed based on a direction corresponding to a reference direction of the object, to learn so that a result of data processing matches the correct data processing result;
When the learning unit performs the data processing, the learning unit performs a convolution process on the learning data using a filter corresponding to each of a plurality of specific directions, the filter being generated using a filter using the coefficient, and performs data processing on each of the learning data;
A learning model generation method in which the results of the data processing using filters corresponding to each of the multiple specific directions are weighted and integrated according to the relative angle between the direction corresponding to the reference direction of the object and the specific direction.