JP7523312B2 - OBJECT RECOGNITION DEVICE, OBJECT RECOGNITION SYSTEM, LEARNING METHOD FOR OBJECT RECOGNITION DEVICE, OBJECT RECOGNITION METHOD FOR OBJECT RECOGNITION DEVICE, LEARNING PROGRAM FOR OBJECT RECOGNITION DEVICE, AND OBJECT RECOGNITION PROGRAM FOR OBJECT RECOGNITION DEVICE - Google Patents

Description

The present disclosure relates to an object recognition device, an object recognition system, a learning method for an object recognition device, an object recognition method for an object recognition device, a learning program for an object recognition device, and an object recognition program for an object recognition device.

As an object recognition device, for example, an object recognition device that recognizes objects captured while driving is known (see Patent Document 1). The object recognition device of Patent Document 1 includes a scene determination unit, a learning model selection unit, and an object recognition unit. The scene determination unit determines the scene of the vehicle while driving based on information acquired by various sensors that detect the state of the vehicle and information obtained while the vehicle is being driven. The learning model selection unit selects a learning model to be used for object recognition from two or more learning models according to the determined scene. The object recognition unit uses the selected learning model to recognize objects in images captured while the vehicle is being driven.

International Publication No. 2020/090251

The object recognition device shown in Patent Document 1 recognizes objects in images captured while driving a vehicle, and the information required for object recognition is obtained from an imaging device. In other words, the object recognition device shown in Patent Document 1 performs object detection such as object recognition using a single sensor. On the other hand, so-called multimodal object detection, in which object detection is performed using multiple sensors, is being considered.

When performing object detection using multiple sensors, if some of the sensors are replaced with different types or new sensors are added, the object detection model needs to be retrained. When training an object detection model, a training dataset that includes input information (sensor information) and teacher information is required. Since the training dataset needs to be prepared for all scenes in which object detection will be performed, the amount of data becomes large, making it difficult to reduce training costs.

The present disclosure aims to provide an object recognition device, an object recognition system, a learning method for an object recognition device, an object recognition method for an object recognition device, a learning program for an object recognition device, and an object recognition program for an object recognition device that can reduce the amount of data required for learning and reduce learning costs even when changing the configuration of multiple sensors.

An object recognition device according to one aspect includes an object detection unit that performs object detection using an object detection model based on sensor information acquired from a plurality of sensors including a main sensor and a sub-sensor. The object detection model includes a main model for performing object detection based on first sensor information acquired by the main sensor, and a sub-model for performing object detection based on second sensor information acquired by the sub-sensor. The object detection unit includes a first object detection unit that performs object detection using the main model, a second object detection unit that performs object detection using the sub-model, and a feature fusion unit that fuses a first feature related to the first sensor information obtained from the first object detection unit and a second feature related to the second sensor information obtained from the second object detection unit to obtain a fused feature. The main model and the sub-model are the same network having a hierarchy of features.

An object recognition system according to one aspect includes a plurality of sensors including a main sensor and a sub-sensor, and the object recognition device described above.

A learning method for an object recognition device according to one aspect is a learning method for an object recognition device executed by the object recognition device described above, in which a learning data set including teacher information, which is output information serving as a teacher output from the object detection unit, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information is used for learning the object recognition device, the first sensor information and the teacher information are used to learn the main model in the first object detection unit, object detection is performed from the first sensor information using the main model after learning, and the output result is obtained as target information, the first sensor information and the target information are used to learn the main model in the first object detection unit, and the second sensor information and the teacher information are used to learn the sub-model in the second object detection unit.

An object recognition method for an object recognition device according to one aspect is an object recognition method for an object recognition device executed by the object recognition device described above, which acquires the sensor information from a plurality of the sensors, and in the first object detection unit, performs object detection using the main model from the first sensor information acquired from the main sensor and outputs an output result, and in the second object detection unit, performs object detection using the sub-model from the second sensor information acquired from the sub-sensor and outputs an output result, integrates the output result by the first object detection unit and the output result by the second object detection unit, and outputs the integrated result as output information.

The learning program for the object recognition device according to one aspect is a learning program for the object recognition device executed by the object recognition device described above, in which a learning data set including teacher information, which is output information serving as a teacher output from the object detection unit, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information is used for learning the object recognition device, the object recognition device is caused to learn the main model in the first object detection unit using the first sensor information and the teacher information, and to perform object detection from the first sensor information using the main model after learning and obtain the output result as target information, and the main model is learned in the first object detection unit using the first sensor information and the target information, and the sub-model is learned in the second object detection unit using the second sensor information and the teacher information.

An object recognition program for an object recognition device according to one aspect is an object recognition program for an object recognition device executed by the object recognition device described above, which causes the object recognition device to acquire the sensor information from the multiple sensors, causes the first object detection unit to perform object detection using the main model from the first sensor information acquired from the main sensor and output an output result, causes the second object detection unit to perform object detection using the sub-model from the second sensor information acquired from the sub-sensor and output an output result, integrates the output result from the first object detection unit and the output result from the second object detection unit, and outputs the integrated result as output information.

According to the present disclosure, even when changing the configuration of multiple sensors, it is possible to reduce the amount of data required for learning and reduce learning costs.

FIG. 1 is a diagram illustrating an object recognition system according to an embodiment. FIG. 2 is a diagram relating to the object detection unit. FIG. 3 is a flowchart illustrating an example of an object recognition method of the object recognition device according to the embodiment. FIG. 4 is a diagram of a training data set. FIG. 5 is an explanatory diagram relating to a learning method of the object recognition device according to the embodiment. FIG. 6 is a flowchart illustrating an example of a learning method for an object recognition device according to the embodiment. FIG. 7 is a diagram showing a comparison of the training data sets used in conventional training and training according to the embodiment.

The embodiments of the present disclosure will be described in detail with reference to the drawings. In the following description, similar components may be given the same reference numerals. Furthermore, duplicated descriptions may be omitted. Furthermore, descriptions and illustrations of matters that are not closely related to the description of the embodiments of the present disclosure may be omitted. Note that the present disclosure is not limited to the following embodiments. Furthermore, the following embodiments include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range.

(Embodiment)
1 is a diagram of an object recognition system according to an embodiment. The object recognition system 1 is a system that detects objects based on sensor information acquired by a plurality of sensors 5. The object recognition system 1 is mounted on, for example, a vehicle and detects objects included in various scenes acquired in the vehicle. The scenes include daytime, nighttime, rainy, and the like.

As shown in FIG. 1, the object recognition system 1 includes a plurality of sensors 5 and an object recognition device 10. The object recognition system 1 is a system related to multimodal object detection using a plurality of sensors 5. The plurality of sensors 5 include a main sensor 7 and a sub-sensor 8. The main sensor 7 and the sub-sensor 8 are different types of sensors. The main sensor 7 is a sensor that is mainly used in object detection. For example, a camera that captures a scene and generates an image is applied to the main sensor 7. The main sensor 7 is connected to the object recognition device 10, and first sensor information acquired by the main sensor 7 is output to the object recognition device 10. The sub-sensor 8 is a sensor that is used auxiliary to the main sensor 7 in object detection. For example, a LIDAR (Light Detection and Ranging), a RADAR (Radio Detection and Ranging), or the like is applied to the sub-sensor 8. The sub-sensor 8 is connected to the object recognition device 10 like the main sensor 7, and second sensor information acquired by the sub-sensor 8 is output to the object recognition device 10.

The object recognition device 10 performs object detection by fusing the first sensor information and the second sensor information and using a learning model (object detection model) such as FPN (Feature Pyramid Networks). The object recognition device 10 includes a control unit 11, a memory unit 12, and an object detection unit 13.

The memory unit 12 stores programs and data. The memory unit 12 may also be used as a working area for temporarily storing the processing results of the control unit 11. The memory unit 12 may include any memory device, such as a semiconductor memory device or a magnetic memory device. The memory unit 12 may also include multiple types of memory devices. The memory unit 12 may also include a combination of a portable storage medium, such as a memory card, and a storage medium reader.

The storage unit 12 stores a learning program P1 and an object recognition program P2 as programs. The learning program P1 is a program for learning the learning model of the object recognition device 10. The object recognition program P2 is a program for performing object detection using the learning model of the object recognition device 10. The storage unit 12 also stores a learning data set D used for learning as data. The learning data set D is data used when learning the learning model. Therefore, while the learning data set D is stored in the storage unit 12 when learning the learning model, it does not have to be stored in the storage unit 12 when detecting an object using the learning model. In other words, when the object recognition device 10 is used for object detection, the learning data set D does not have to be stored in the storage unit 12. The learning data set D will be described later. When the object recognition device 10 is used for object detection, the storage unit 12 stores the first sensor information and the second sensor information acquired from the main sensor 7 and the sub-sensor 8.

The control unit 11 comprehensively controls the operation of the object recognition device 10 to realize various functions. The control unit 11 includes an integrated circuit such as a CPU (Central Processing Unit). Specifically, the control unit 11 realizes various functions by executing instructions included in programs stored in the memory unit 12. The control unit 11 executes, for example, a learning program P1 to perform learning of the learning model of the object recognition device 10. The control unit 11 also executes, for example, an object recognition program P2 to perform object detection using the learning model of the object recognition device 10.

FIG. 2 is a diagram relating to the object detection unit. The object detection unit 13 includes an integrated circuit such as a GPU (Graphics Processing Unit). The object detection unit 13 performs object detection using a learning model such as FPN. As shown in FIG. 2, the object detection unit 13 has a first main object detection unit 21, a second sub object detection unit 22, an integration unit 23, and a feature fusion unit 24. Note that the object detection unit 13 may have a configuration including multiple second object detection units 22.

Here, the learning model used in the object detection unit 13 includes a main model 35 used in the first object detection unit 21 and a sub-model 45 used in the second object detection unit 22. The main model 35 is a model for performing object detection based on the first sensor information acquired by the main sensor 7. The sub-model 45 is a model for performing object detection based on the second sensor information acquired by the sub-sensor 8. The main model 35 and the sub-model 45 are the same learning model, and are networks having a hierarchy of features, that is, networks using feature maps.

The first object detection unit 21 performs object detection using a main model 35. The first object detection unit 21 uses the main model 35 including a first encoder 31 and a first decoder 32.

The first encoder 31 performs encoding processing on the first sensor information. When the first sensor information is image information, the encoding processing generates a feature map that extracts the features of the image information, while performing downsampling (also called pooling) to reduce the resolution of the feature map. Specifically, in the encoding processing, image information is processed in a convolution layer and a pooling layer. In the convolution layer, a kernel (filter) for extracting the features of the image information is moved in the image information at a predetermined stride. Then, in the convolution layer, a convolution calculation is performed to extract the features of the image information based on the weight of the convolution layer, and a feature map from which the features are extracted is generated by this convolution calculation. The number of generated feature maps is based on the number of kernel channels. In the pooling layer, the feature map from which the features are extracted is reduced to generate a feature map with a low resolution. In the encoding processing, a feature map having downsampled features is generated by repeatedly executing the processing in the convolution layer and the processing in the pooling layer multiple times.

The first decoder 32 performs a decoding process on the feature map after the encoding process. The decoding process is a process of performing upsampling (also called unpooling) to increase the resolution of the feature map. Specifically, the decoding process is performed on the feature map in the deconvolution layer and the unpooling layer. In the unpooling layer, a low-resolution feature map including features is enlarged to generate a high-resolution feature map. In the deconvolution layer, a deconvolution calculation for restoring the features included in the feature map is performed based on the weights of the deconvolution layer, and a feature map with the features restored by this calculation is generated. In the unpooling layer, the feature map is enlarged to generate a high-resolution feature map. In the decoding process, the feature map including the features is upsampled by repeatedly performing the process in the unpooling layer and the process in the deconvolution layer multiple times.

The first object detection unit 21 assigns class likelihood to the upsampled feature map and outputs this as an output result to the integration unit 23. In other words, the first object detection unit 21 outputs the class likelihood assigned to the upsampled feature map on a pixel-by-pixel basis as an output result.

As described above, the first object detection unit 21 performs encoding and decoding processes on the first sensor information, and performs class inference (class classification) on a pixel-by-pixel basis. The first object detection unit 21 then outputs the likelihood of the class assigned to the upsampled feature map on a pixel-by-pixel basis as an output result.

The second object detection unit 22 performs object detection using a sub-model 45. The second object detection unit 22 uses a sub-model 45 including a second encoder 41 and a second decoder 42. Note that the second object detection unit 22 is similar to the first object detection unit 21. In other words, the second encoder 41 and the second decoder 42 have the same configuration as the first encoder 31 and the first decoder 32. For this reason, a description of the second object detection unit 22 will be omitted.

Here, the first object detection unit 21 and the second object detection unit 22 are connected via the feature amount fusion unit 24. The feature amount fusion unit 24 transfers the first feature amount extracted in the main model 35 of the first object detection unit 21 to the sub-model 45 of the second object detection unit 22. Specifically, a predetermined layer of the feature map in the main model 35 of the first object detection unit 21 is connected to a predetermined layer of the feature map in the sub-model 45 of the second object detection unit 22. The predetermined layer of the sub-model 45 corresponds to the next layer that is the output side with respect to the predetermined layer of the main model 35. In other words, if the predetermined layer of the main model 35 is the nth layer, the predetermined layer of the sub-model 45 is the (n+1)th layer. The first object detection unit 21 extracts the first feature amount included in the feature map generated from the first sensor information and outputs the first feature amount to the feature amount fusion unit 24. In addition, the second object detection unit 22 extracts a second feature amount contained in the feature map generated from the second sensor information, and outputs the second feature amount to the feature amount fusion unit 24.

The feature fusion unit 24 fuses the first feature extracted by the first object detection unit 21 and the second feature extracted by the second object detection unit 22 to generate a fused feature.

The second object detection unit 22 performs encoding and decoding processes on the feature map including the acquired fusion features, and performs class inference (class classification) on a pixel-by-pixel basis. The second object detection unit 22 then outputs the likelihood of the class assigned to the upsampled feature map on a pixel-by-pixel basis as an output result.

The integration unit 23 integrates the output result of the first object detection unit 21 and the output result of the second object detection unit 22, and outputs the integration result as output information. The integration unit 23 integrates the class likelihood assigned on a pixel-by-pixel basis to the feature map upsampled by the first object detection unit 21 and the class likelihood assigned on a pixel-by-pixel basis to the feature map upsampled by the second object detection unit 22. The integration unit 23 integrates the output results by using, for example, the average value of the likelihood or the maximum value of the likelihood.

Next, referring to FIG. 3, the object recognition operation of the object recognition device 10 equipped with the above-mentioned object detection unit 13 will be described. FIG. 3 is a flowchart of an example of an object recognition method of the object recognition device according to the embodiment. The object recognition device 10 executes an object recognition program P2 stored in the memory unit 12, thereby performing object detection by the object detection unit 13 based on sensor information from the main sensor 7 and the sub-sensor 8.

First, the control unit 11 acquires first sensor information from the main sensor 7 (step S11). In the object detection unit 13, the first object detection unit 21 generates a feature map using the main model 35 based on the first sensor information and extracts a first feature amount (step S12). After that, the first object detection unit 21 performs object detection based on the extracted first feature amount and outputs the detection result as an output result (step S13).

The control unit 11 also acquires second sensor information from the sub-sensor 8 (step S14). In the object detection unit 13, the second object detection unit 22 generates a feature map using the sub-model 45 based on the second sensor information and extracts a second feature (step S15). After that, the second object detection unit 22 fuses the second feature extracted by the second object detection unit 22 with the first feature extracted by the first object detection unit 21 to generate a fusion feature (step S16). After that, the second object detection unit 22 performs object detection based on the fusion feature and outputs the detection result as an output result (step S17).

The object detection unit 13 integrates the output result output from the first object detection unit 21 in step S13 and the output result output from the second object detection unit 22 in step S17 by the integration unit 23 to generate an integrated result (step S20). Specifically, in step S20, the integration unit 23 acquires, as the output result, the likelihood of the class assigned to the feature maps output from the first object detection unit 21 and the second object detection unit 22 on a pixel-by-pixel basis. Then, based on the acquired likelihood, the integration unit 23 generates the integrated result by using the average value of the likelihood or the maximum value of the likelihood.

Then, the object detection unit 13 performs post-processing on the generated integration result to generate output information (step S21). An example of post-processing is NMS (Non-Maximum Suppression). NMS is a process for suppressing overlapping of image regions of the same class when the image regions overlap due to object detection. After generating the output information, the object detection unit 13 outputs the output information (step S22). After executing step S22, the object detection unit 13 ends the object recognition operation.

Next, the learning operation of the object recognition device 10 equipped with the above-mentioned object detection unit 13 will be described with reference to Figs. 4 to 6. Fig. 4 is a diagram of a learning dataset. Fig. 5 is an explanatory diagram of a learning method for the object recognition device according to the embodiment. Fig. 6 is a flowchart of an example of the learning method for the object recognition device according to the embodiment. The object recognition device 10 executes a learning program P1 stored in the memory unit 12 to perform learning of object detection by the object detection unit 13 using the learning dataset D.

As shown in FIG. 4, the learning dataset D includes first sensor information, second sensor information, and teacher information, which is output information that serves as a teacher. The first sensor information, second sensor information, and teacher information are associated with each other, and multiple sets of these are prepared as one set.

When the main sensor 7 is a camera, the first sensor information is captured image information. When the sub-sensor 8 is a LIDAR, the second sensor information is information on a point cloud of measurement points measured by a laser. The teacher information is ground truth (GT) in which a true value class is labeled for the image information that becomes the output information, and is, for example, annotated image information.

As shown in FIG. 5, the learning of the object recognition device 10 is broadly divided into three steps. Of the three steps, the first step is a step of learning the main model 35 in the first object detection unit 21 (step S31). The second step is a step of outputting target information using the learned main model 35 (step S32). The third step is a step of learning a sub-model in the second object detection unit 22 (step S33).

The learning operation of the object recognition device 10 will be described with reference to FIG. 5 and FIG. 6. First, the object detection unit 13 performs learning of the main model 35 using the first sensor information and teacher information of the learning data set D (step S31). In step S31, learning is performed using a loss function so that the output information output from the first object detection unit 21 based on the first sensor information becomes teacher information. In addition, in step S31, learning is performed on scenes that are statistically frequent in object detection. In other words, the first sensor information and teacher information used in learning the main model 35 are information on scenes that are statistically frequent, for example, information on daytime and nighttime scenes. In other words, the scenes that are statistically frequent can be said to be information on all scenes.

The object detection unit 13 then uses the trained main model 35 to perform object detection based on the first sensor information of the training dataset D, and acquires output information as target information (step S32). That is, in step S32, target information is acquired as a training result of the main model 35. In step S32, object detection is performed for scenes that are statistically infrequent, for example, object detection is performed for nighttime scenes, and target information for nighttime scenes is acquired. In other words, scenes that are statistically infrequent can be said to be information about some scenes compared to all scenes.

Next, the object detection unit 13 performs learning of the sub-model 45 using the acquired target information, the first sensor information corresponding to the target information, the second sensor information corresponding to the first sensor information, and the teacher information (step S33). In step S33, the main model 35 is learned using a loss function so that the output information output from the first object detection unit 21 based on the first sensor information becomes the target information. That is, in the learning of the main model 35, learning is performed so that the difference between the first sensor information and the target information becomes small. In addition, in step S33, the sub-model 45 is learned. In the learning of the sub-model 45, the feature amount fusion unit 24 acquires the first feature amount extracted from the feature map generated in the first object detection unit 21. In step S33, the feature amount fusion unit 24 fuses the first feature amount acquired from the first object detection unit 21 with the second feature amount extracted from the feature map generated in the second object detection unit 22. In the learning of the sub-model 45 in step S33, the second object detection unit 22 performs learning based on the fusion feature using the second sensor information and the teacher information. In the learning of the sub-model 45 in step S33, learning is performed using a loss function so that the output information output from the second object detection unit 22 based on the second sensor information becomes teacher information. Then, in step S33, learning of the main model 35 and the sub-model 45 is performed so that the sum of the loss in the learning of the main model 35 and the loss in the learning of the sub-model 45 is minimized. Here, since the target information is used in step S33, learning is performed on a scene that is statistically infrequent in object detection. In other words, the first sensor information, the second sensor information, and the teacher information used in the learning of the sub-model 45 are information on a scene that is statistically infrequent, for example, information on a nighttime scene. After executing step S33, the object recognition device 10 ends the learning operation of the object detection unit 13.

Next, with reference to FIG. 7, a comparison of the amount of information between the learning data set used in conventional learning and the learning data set used in the learning of the embodiment will be described. FIG. 7 is a diagram relating to a comparison of the learning data set used in conventional learning and the learning of the embodiment. FIG. 7 compares the amount of information and the need for re-learning in four patterns. Of the four patterns, the first is a standard configuration pattern using one main sensor 7 and one sub-sensor 8 (sub-sensor A). The second is a change pattern when the sub-sensor 8 (sub-sensor A) of the standard configuration is changed to another sub-sensor 8 (sub-sensor B). The third is an addition pattern when another sub-sensor 8 (sub-sensor B) is added to the sub-sensor 8 (sub-sensor A) of the standard configuration. The fourth is a deletion pattern when the sub-sensor 8 (sub-sensor A) of the standard configuration is deleted.

In the standard configuration pattern, in the conventional learning, it is necessary to prepare information necessary for learning for the number of sensors. That is, in the conventional learning, it is necessary to prepare the first sensor information for learning the main model 35 and the first teacher information corresponding to the first sensor information, and to prepare the second sensor information for learning the sub-model 45 and the second teacher information corresponding to the second sensor information. That is, if the amount of information necessary for learning one sensor is n, the amount of information necessary for learning the main model 35 and the sub-model 45 is "2n". In contrast, in the learning of the embodiment, since the target information is used, it is only necessary to prepare the first sensor information, the teacher information, and the second sensor information for learning the main model 35 and the sub-model 45, which shows that the amount of information required can be reduced. Note that in the standard configuration pattern, re-learning is necessary whether it is the conventional learning or the learning of the embodiment.

In the change pattern, in conventional learning, when the sub-sensor 8 is changed, the first sensor information, the first teacher information, the second sensor information, and the second teacher information all need to be changed. Therefore, the amount of information required for learning the main model 35 and the sub-model 45 is "2n". In contrast, in the learning of the embodiment, since target information is used, there is no need to change the first sensor information or teacher information for learning the main model 35 and the sub-model 45, and it is sufficient to change the second sensor information anew. This shows that the amount of information required can be reduced. Note that in the change pattern, re-learning is required whether it is conventional learning or learning of the embodiment.

In the addition pattern, in conventional learning, when the sub-sensor 8 is added, the first sensor information, the first teacher information, the second sensor information, and the second teacher information must all be changed, and new third sensor information and third teacher information must be prepared. Therefore, the amount of information required for learning the main model 35 and the two sub-models 45 is "3n". In contrast, in the learning of the embodiment, since target information is used, it is not necessary to change the first sensor information, the second sensor information, and the teacher information for learning the main model 35 and the two sub-models 45, and it is sufficient to add new third sensor information. Therefore, it is possible to reduce the amount of information required. Note that in the addition pattern, re-learning is required whether it is conventional learning or learning of the embodiment.

In the deletion pattern, in conventional learning, when the sub-sensor 8 is deleted, it is necessary to re-learn the main model 35 using the first sensor information and the first teacher information. Note that the first sensor information and the first teacher information can use information prepared in the standard configuration pattern. In contrast, in the learning of the embodiment, since the target information is used, even if the sub-sensor 8 is deleted, the learning of the main model 35 is maintained, and so there is no need to re-learn the main model 35. Therefore, in the deletion pattern, re-learning is not required in the learning of the embodiment.

As described above, the object recognition device 10 according to the embodiment includes an object detection unit 13 that performs object detection using an object detection model based on sensor information acquired from a plurality of sensors 5 including a main sensor 7 and a sub-sensor 8. The object detection model includes a main model 35 for performing object detection based on first sensor information acquired by the main sensor 7 and a sub-model 45 for performing object detection based on second sensor information acquired by the sub-sensor 8. The object detection unit 13 includes a first object detection unit 21 that performs object detection using the main model 35, a second object detection unit 22 that performs object detection using the sub-model 45, and a feature fusion unit 24 that fuses a first feature amount related to the first sensor information obtained from the first object detection unit 21 and a second feature amount related to the second sensor information obtained from the second object detection unit 22 to obtain a fused feature amount. The main model 35 and the sub-model 45 form the same network having a hierarchy of features.

Therefore, even when object detection is performed using multiple sensors 5, object detection based on the first feature by the first object detection unit 21 can be performed, and object detection based on the fused feature by the second object detection unit 22 can be performed. In this case, even when the configuration of the sub-sensor 8 is changed, the configuration is such that learning by the first object detection unit 21 can be maintained, and learning only needs to be performed by the sub-sensor 8, so the amount of information required for learning can be reduced, and learning costs can be reduced.

The object detection unit 13 further includes an integration unit 23 that integrates the output result from the first object detection unit 21 and the output result from the second object detection unit 22 and outputs the integrated result as output information.

Therefore, by integrating the output results of the first object detection unit 21 and the second object detection unit 22, robustness can be improved and object detection can be performed with high accuracy. Furthermore, even if the second sensor information cannot be obtained from the sub-sensor 8, object detection can be performed using the main sensor 7, so that a lack of second sensor information can be tolerated and highly robust object detection can be performed.

The main model 35 is an object detection model that is trained in the first object detection unit 21 using the first sensor information, and the sub-model 45 is an object detection model that is trained in the second object detection unit 22 using the second sensor information and the learning result of the main model.

Therefore, the main model 35 and the sub-model 45 can be appropriately trained as object detection models that complement sensor information. In addition, the main model 35 can be appropriately trained as object detection models that can perform object detection using only the main model 35.

The sub-model 45 also uses the output results output from the main model 35 after learning as target information, as the learning result of the main model 35.

Therefore, by using it as target information, the amount of information required for learning can be reduced, and learning costs can be reduced.

The feature fusion unit 24 also connects the feature hierarchy of the main model 35 to the feature hierarchy of the sub-model 45 that corresponds to the next hierarchy that is the output side of the feature hierarchy of the main model 35, and fuses the first feature obtained from the feature hierarchy of the main model 35 with the second feature obtained from the next feature hierarchy that is the output side of the sub-model 45.

Therefore, the first feature obtained from a given feature hierarchy can be reflected in the second feature obtained from the next feature hierarchy, which is the output side. Therefore, in object detection by the second object detection unit 22, object detection based on the fusion feature can be appropriately performed.

The object recognition system 1 according to the embodiment includes a plurality of sensors 5 including a main sensor 7 and a sub-sensor 8, and the object recognition device 10 described above.

As a result, since the configuration is such that learning by the first object detection unit 21 can be maintained, learning can be performed by the sub-sensor 8, thereby reducing the amount of information required for learning and making the system lower in learning costs.

In addition, the main sensor 7 and the sub-sensor 8 are different types.

Therefore, by using multiple sensors 5 of different types, object detection can be performed in a variety of scenes.

The learning method of the object recognition device 10 according to the embodiment is a learning method of the object recognition device 10 executed by the object recognition device 10 described above, in which a learning data set D including teacher information, which is output information serving as a teacher output from the object detection unit 13, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information is used for learning the object recognition device 10, the first object detection unit 21 learns the main model 35 using the first sensor information and the teacher information, the main model 35 after learning is used to perform object detection from the first sensor information and obtain the output result as target information, the first object detection unit 21 learns the main model 35 using the first sensor information and the target information, and the second object detection unit 22 learns the sub-model 45 using the second sensor information and the teacher information.

The learning program of the object recognition device 10 according to the embodiment is a learning program of the object recognition device 10 executed by the object recognition device 10 described above, and the learning of the object recognition device 10 uses a learning data set D including teacher information, which is output information to be a teacher output from the object detection unit 13, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information. The object recognition device 10 uses the first sensor information and the teacher information to learn the main model 35 in the first object detection unit 21, performs object detection from the first sensor information using the main model 35 after learning, and obtains the output result as target information. The first sensor information and the target information are used to learn the main model 35 in the first object detection unit 21, and the second sensor information and the teacher information are used to learn the sub-model 45 in the second object detection unit 22.

This prevents the first object detection unit 21 from forgetting its learning, and allows the second object detection unit 22 to perform learning using the learning results of the first object detection unit 21. Therefore, even if the configuration of the sub-sensor 8 is changed, the learning by the first object detection unit 21 can be maintained, and learning by the sub-sensor 8 only needs to be performed, reducing the amount of information required for learning and the learning cost.

In addition, in the learning of the sub-model 45, the main model 35 is used to extract the first feature obtained from the target information, the sub-model 45 is used to extract the second feature obtained from the second sensor information, the feature fusion unit 24 fuses the extracted first feature and the extracted second feature to calculate the fusion feature, and the learning of the sub-model 45 is performed based on the calculated fusion feature.

Therefore, the learning of the sub-model 45 can be performed based on the fusion feature. Therefore, in object detection by the second object detection unit 22, object detection based on the fusion feature can be appropriately performed.

In addition, in the learning of the sub-model 45, learning is performed so that in the learning of the main model 35, the difference between the first sensor information and the target information becomes small, and in the learning of the sub-model 45, the difference between the second sensor information and the teacher information becomes small.

This makes it possible to appropriately suppress forgetting of learning in the main model 35, thereby improving the accuracy of learning in the sub-model 45.

The teaching information is ground truth (GT).

Therefore, it is possible to perform learning of the main model 35 and the sub-model 45 using highly accurate teacher information.

In addition, the first sensor information and the teacher information used in the learning of the main model 35 in the learning dataset D are information about scenes that are statistically frequent in object detection, and the second sensor information and the teacher information used in the learning of the sub-model 45 are information about scenes that are statistically less frequent in object detection compared to the main model 35.

Therefore, object detection suitable for scenes that are statistically frequent can be performed mainly using the main model 35, and object detection suitable for scenes that are statistically infrequent can be performed using the main model 35 and the sub-model 45.

The object recognition method of the object recognition device 10 according to the embodiment is an object recognition method of the object recognition device 10 executed by the object recognition device 10 described above, in which the sensor information is acquired from the multiple sensors 5, and in the first object detection unit 21, object detection is performed using the main model 35 from the first sensor information acquired from the main sensor 7 and an output result is output, and in the second object detection unit 22, object detection is performed using the sub-model 45 from the second sensor information acquired from the sub-sensor 8 and an output result is output, the output result by the first object detection unit 21 and the output result by the second object detection unit 22 are integrated, and the integrated result is output as output information.

The object recognition program P2 of the object recognition device 10 according to the embodiment is an object recognition program P2 of the object recognition device 10 executed by the object recognition device 10, and causes the object recognition device 10 to acquire the sensor information from the plurality of sensors 5, and causes the first object detection unit 21 to execute object detection using the main model 35 from the first sensor information acquired from the main sensor 7 and output an output result, and causes the second object detection unit 22 to execute object detection using the sub-model 45 from the second sensor information acquired from the sub-sensor 8 and output an output result, and integrates the output result by the first object detection unit 21 and the output result by the second object detection unit 22, and outputs the integrated result as output information.

Therefore, even when object detection is performed using multiple sensors 5, object detection can be performed based on the first feature by the first object detection unit 21, and object detection can be performed based on the fused feature by the second object detection unit 22. The output results of the first object detection unit 21 and the output results of the second object detection unit 22 can be integrated by the integration unit 23 and output as output information. Therefore, robustness can be improved by fusion of the features and integration of the output results, and object detection can be performed with high accuracy.

REFERENCE SIGNS LIST 1 Object recognition system 5 Sensor 7 Main sensor 8 Sub-sensor 10 Object recognition device 11 Control unit 12 Memory unit 13 Object detection unit 21 First object detection unit 22 Second object detection unit 23 Integration unit 24 Feature fusion unit 31 First encoder 32 First decoder 35 Main model 41 Second encoder 42 Second decoder 45 Sub-model P1 Learning program P2 Object recognition program D Learning dataset

Claims (14)

An object detection unit that performs object detection using an object detection model based on sensor information acquired from a plurality of sensors including a main sensor and a sub sensor,
The object detection model is
a main model for performing object detection based on first sensor information acquired by the main sensor;
A sub-model for performing object detection based on second sensor information acquired by the sub-sensor;
The object detection unit is
a first object detection unit that performs object detection using the main model;
a second object detection unit that performs object detection using the sub-model;
a feature fusion unit that fuses a first feature related to the first sensor information obtained from the first object detection unit and a second feature related to the second sensor information obtained from the second object detection unit to generate a fused feature;
having
The main model and the sub-model are the same network having a hierarchy of features,
The feature amount fusion unit is
An object recognition device that connects the feature hierarchy of the main model to the feature hierarchy of the sub-model corresponding to the next hierarchy that is the output side of the feature hierarchy of the main model, and fuses the first feature obtained from the feature hierarchy of the main model with the second feature obtained from the next feature hierarchy that is the output side of the sub-model . The object detection unit is
The object recognition device according to claim 1 , further comprising an integration unit that integrates an output result from the first object detection unit and an output result from the second object detection unit, and outputs the integrated result as output information. the main model is an object detection model that is learned by the first object detection unit using the first sensor information,
The object recognition device according to claim 1 or 2, wherein the sub-model is an object detection model that is trained in the second object detection unit using the second sensor information and the learning results of the main model. The object recognition device according to claim 3, wherein the sub-model uses the output result output from the main model after learning as target information as the learning result of the main model. A plurality of sensors including a main sensor and a sub sensor;
An object recognition system comprising: the object recognition device according to any one of claims 1 to 4 . The object recognition system according to claim 5 , wherein the main sensor and the sub-sensor are of different types. A learning method for an object recognition device executed by the object recognition device according to any one of claims 1 to 4 , comprising:
A learning data set including teacher information, which is output information to be a teacher output from the object detection unit, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information is used for learning of the object recognition device;
learning the main model in the first object detection unit using the first sensor information and the teacher information;
Using the main model after learning, perform object detection from the first sensor information and obtain an output result as target information;
A learning method for an object recognition device, comprising: learning the main model in the first object detection unit using the first sensor information and the target information; and learning the sub-model in the second object detection unit using the second sensor information and the teacher information. In the learning of the sub-model,
extracting the first feature amount obtained from the target information using the main model;
Extracting a second feature amount obtained from the second sensor information using the sub-model;
The feature amount fusion unit fuses the extracted first feature amount and the extracted second feature amount to calculate the fused feature amount;
The training method for an object recognition apparatus according to claim 7 , further comprising: training the sub-model based on the calculated fusion feature amount. In the learning of the sub-model,
In the learning of the main model, a difference between the first sensor information and the target information becomes small,
9. The learning method for an object recognition device according to claim 7 , wherein in learning the sub-model, learning is performed so that a difference between the second sensor information and the teaching information becomes small. The training method for an object recognition device according to claim 7 , wherein the teaching information is ground truth (GT). The training data set is
the first sensor information and the teacher information used in learning the main model are information about a scene that is statistically frequent in object detection,
A learning method for an object recognition device described in any one of claims 7 to 10 , wherein the second sensor information and the teacher information used in learning the sub-model are information about scenes that are statistically less frequent in object detection compared to the main model. An object recognition method for an object recognition device executed by the object recognition device according to any one of claims 1 to 4 , comprising:
acquiring the sensor information from a plurality of the sensors;
In the first object detection unit, object detection is performed using the main model from the first sensor information acquired from the main sensor, and an output result is output;
In the second object detection unit, object detection is performed using the sub-model from the second sensor information acquired from the sub-sensor, and an output result is output;
An object recognition method for an object recognition device, comprising: integrating an output result from the first object detection unit and an output result from the second object detection unit; and outputting the integrated result as output information. A learning program for an object recognition device executed by the object recognition device according to any one of claims 1 to 4 ,
A learning data set including teacher information, which is output information to be a teacher output from the object detection unit, the first sensor information corresponding to the teacher information, and the second sensor information corresponding to the teacher information is used for learning of the object recognition device;
The object recognition device includes:
training the main model in the first object detection unit using the first sensor information and the teacher information;
Using the main model after learning, object detection is performed from the first sensor information, and an output result is obtained as target information;
A learning program for an object recognition device that uses the first sensor information and the target information to train the main model in the first object detection unit, and that uses the second sensor information and the teacher information to train the sub-model in the second object detection unit. An object recognition program for an object recognition device executed by the object recognition device according to any one of claims 1 to 4 ,
The object recognition device includes:
acquiring the sensor information from a plurality of the sensors;
In the first object detection unit, object detection is performed using the main model from the first sensor information acquired from the main sensor, and an output result is output;
In the second object detection unit, object detection is performed using the sub-model from the second sensor information acquired from the sub-sensor, and an output result is output;
An object recognition program for an object recognition device that integrates an output result from the first object detection unit and an output result from the second object detection unit, and outputs the integrated result as output information.

Images