JP7679746B2 - Learning device and learning method - Google Patents

Description

The present invention relates to a learning device and a learning method.

There is a known invention that uses the evaluation results of the presentation position of food shown in an image (of plated food as a subject) to obtain a learning model by inputting an image of plated food as a subject and using the evaluation results of the presentation position of food shown in the image as training data (Patent Document 1).

JP 2020-181436 A

However, if there are differences in the number of image capture angles, training the model without any changes may result in a reduced robustness of the model.

The present invention was made in consideration of the above problems, and its purpose is to provide a learning device and a learning method capable of generating a highly robust model.

A learning device according to one aspect of the present invention identifies the type of vehicle depicted in a first image, generates a second image different from the first image using three-dimensional model data corresponding to the identified type of vehicle, learns a model related to the user's evaluation comments using the first image, the second image, and the user's evaluation comments as input data, and stores the learned model in a storage device.

The present invention makes it possible to generate highly robust models.

FIG. 1 is a configuration diagram of a learning device 1 according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating an example of the operation of the learning device 1. FIG. 3 is a diagram illustrating an example of an imaging angle. FIG. 4 is a flowchart illustrating an example of the operation of the learning device 1.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

An example of the configuration of the learning device 1 will be described with reference to FIG. 1. As shown in FIG. 1, the learning device 1 includes a communication I/F 10, a controller 20, and a storage device 30. As an example, the learning device 1 is installed in a general-purpose computer.

The communication I/F 10 is implemented as hardware such as a network adapter, various communication software, or a combination of these, and is configured to realize wired or wireless communication over a network. The communication I/F 10 also functions as an input unit and an output unit for transmitting and receiving data. In this embodiment, the communication I/F 10 will be described as performing Internet communication.

The storage device 30 is composed of a hard disk drive (HDD), a solid state drive (SSD), etc. A plurality of databases are stored in the storage device 30. As shown in FIG. 1, the plurality of databases include an image database 31, an angle data set 32, a three-dimensional vehicle body model data set 33, and a learning model database 34. Details of each database will be explained together with each function of the controller 20.

The controller 20 is an electronic control unit (ECU: Electronic Control Unit) having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, etc. A computer program for functioning as the learning device 1 is installed in the controller 20. By executing the computer program, the controller 20 functions as multiple information processing circuits equipped in the learning device 1. Note that, although an example of realizing multiple information processing circuits equipped in the learning device 1 by software is shown here, it is of course possible to configure the information processing circuit by preparing dedicated hardware for executing each information processing shown below. In addition, multiple information processing circuits may be configured by individual hardware. The controller 20 includes multiple information processing circuits, including a data acquisition unit 21, an angle estimation unit 22, a classification unit 23, a comparison unit 24, a generation unit 25, and a learning unit 26.

Next, the functions of the data acquisition unit 21, angle estimation unit 22, and classification unit 23 will be explained with reference to Figures 2 and 3.

The data acquisition unit 21 acquires vehicle images (two-dimensional images) from the Internet via the communication I/F 10 (step S101 in FIG. 2). When acquiring vehicle images, the data acquisition unit 21 also acquires user comments on the images (sometimes called evaluation comments). To acquire user comments on the images, the data acquisition unit 21 acquires vehicle images and user comments on the images mainly from word-of-mouth sites, SNS (Social Networking Service), or sites that publish survey results. As shown in FIG. 1, the data acquisition unit 21 associates the acquired images with the comments and stores them in the image database 31.

The data acquisition unit 21 also acquires the angle at which the vehicle was imaged (the angle seen from the camera with respect to the vehicle) and the distance from the camera to the vehicle when the vehicle was imaged. In the following, the "angle at which the vehicle was imaged" may be simply referred to as the "image capture angle," and the "distance from the camera to the vehicle when the vehicle was imaged" may be simply referred to as the "image capture distance." In this embodiment, the "image capture angle" is defined as an angle when the front of the vehicle is imaged from directly in front as shown in FIG. 3, and the angle increases clockwise. After 180 degrees, the positive and negative signs are reversed. Note that this definition of the image capture angle is one example, and other definitions may be used. The data acquisition unit 21 links the acquired image, comment, image capture angle, and image capture distance and stores them in the image database 31. The method of linking data is well known, so a description will be omitted.

The angle estimation unit 22 reads out the image stored in the image database 31 and estimates the imaging angle and imaging distance (step S103 in FIG. 2). In the above description, it has been described that the imaging angle and imaging distance are acquired by the data acquisition unit 21. However, when an image is acquired, there are cases where the imaging angle and imaging distance cannot be acquired together. For example, this corresponds to a case where only the image and the user's comments are made public. In such a case, the angle estimation unit 22 estimates the imaging angle and imaging distance. More specifically, the angle estimation unit 22 estimates the imaging angle and imaging distance for images acquired by the data acquisition unit 21 that are not associated with the imaging angle and imaging distance. Therefore, if the imaging angle and imaging distance are associated with all the images acquired by the data acquisition unit 21, the angle estimation unit 22 becomes unnecessary. The method of estimating the imaging angle and imaging distance from the image is not particularly limited, but for example, a method of estimating using a machine learning model can be mentioned. The angle estimation unit 22 associates the estimated imaging angle and imaging distance with the image and stores them in the image database 31.

The classification unit 23 classifies the images stored in the image database 31 into predetermined angles according to the imaging angle (step S105 in FIG. 2). The predetermined angle is, for example, 45 degrees as shown in FIG. 3. Reference numeral 40 in FIG. 3 indicates a range of -22.5 degrees to 22.5 degrees. Similarly, reference numeral 41 indicates a range of 22.5 degrees to 67.5 degrees, reference numeral 42 indicates a range of 67.5 degrees to 112.5 degrees, reference numeral 43 indicates a range of 112.5 degrees to 157.5 degrees, reference numeral 44 indicates a range of 157.5 degrees to -157.5 degrees, reference numeral 45 indicates a range of -157.5 degrees to -112.5 degrees, reference numeral 46 indicates a range of -112.5 degrees to -67.5 degrees, and reference numeral 47 indicates a range of -67.5 degrees to -22.5 degrees. For example, when the imaging angle is 0 degrees, it is classified into class 40. In this way, the classification unit 23 classifies the images stored in the image database 31 into eight classes according to the imaging angle. The classification unit 23 stores the classification results in the angle dataset 32. The classification unit 23 also stores the number of images classified into each class (classes 40 to 47) in the angle dataset 32 (step S107 in FIG. 2). The classified images are also linked with comments, imaging angles, and imaging distances.

Next, the functions of the comparison unit 24 and the generation unit 25 will be described with reference to FIG. 4. The number of pairs of images and comments (the number of images classified into each class) may differ depending on the imaging angle. For classes with a small number of pairs, data needs to be added. In step S201, the comparison unit 24 acquires classified data by referring to the angle data set 32. The process proceeds to step S203, where the comparison unit 24 compares the number of pairs of each class. If there is a difference in the number of pairs of each class, the classes are classified into the class with the largest number of pairs and the other classes. There may be multiple classes with the largest number of pairs. The comparison unit 24 outputs the comparison result to the generation unit 25. Note that there may be no difference in the number of pairs of each class. Here, it is assumed that the class with the largest number of pairs is class 40. In other words, it is assumed that the number of pairs of classes 41 to 47 is smaller than the number of pairs of class 40 (YES in step S205).

The generation unit 25 generates data based on the results from the comparison unit 24. The generation unit 25 generates data until the number of pairs of classes 41 to 47 reaches the number of pairs of class 40. An example of a data generation method will be described with reference to steps S209 to S217. In step S207, the generation unit 25 refers to the angle dataset 32 and acquires a set of pairs (a series of datasets) from the class with the largest number of pairs (class 40 in this case). The "series of datasets" refers to a dataset that links vehicle images, user comments on the images, imaging angles, and imaging distances. The process proceeds to step S209, where the generation unit 25 extracts the vehicle region of the acquired image. The "car region" refers to the area in the image in which a vehicle appears.

The process proceeds to step S211, where the generation unit 25 uses semantic segmentation to divide the vehicle area into its individual parts and extract the shape features of each part. Semantic segmentation is a well-known deep learning algorithm that associates a label or category with every pixel in an image. By using semantic segmentation, for example, shape features such as the grille and headlights are extracted for a front view image (an image classified as class 40).

The process proceeds to step S213, where the generation unit 25 searches for a three-dimensional model by referring to the three-dimensional vehicle body model data set 33. The three-dimensional vehicle body model data set 33 stores three-dimensional model data of the vehicle body for each vehicle type. Furthermore, the three-dimensional vehicle body model data set 33 stores shape features for each part of each vehicle type. The generation unit 25 compares the shape features extracted in step S211 by referring to the three-dimensional vehicle body model data set 33, and outputs the three-dimensional model data with the highest similarity. Note that although the three-dimensional vehicle body model data set 33 stores three-dimensional model data of the vehicle body for each vehicle type, shape features for each part of each vehicle type may not be stored. In this case, the generation unit 25 may output the three-dimensional model data with the highest similarity by the following process. Since the generation unit 25 acquires a series of data sets by referring to the angle data set 32, it knows the imaging angle and imaging distance. The generation unit 25 takes a photo of the three-dimensional model data (multiple different three-dimensional model data) for each vehicle type using the imaging angle and imaging distance. This process means taking a virtual photo from a virtual viewpoint. This allows a virtual image captured at a specified distance (imaging distance) and a specified angle (imaging angle) to be acquired for each of the three-dimensional model data for each vehicle type. The generation unit 25 compares each of the multiple virtual images with vehicle images related to a series of data sets, and outputs the three-dimensional model data related to the virtual image with the highest similarity as the three-dimensional model data with the highest similarity. Note that the similarity includes at least one of pixel value similarity, edge similarity, and shape feature similarity.

The process proceeds to step S215, where the generation unit 25 uses the 3D model data output in step S213 to take pictures from a prescribed distance for the angle classes with the fewest number of pairs (classes 41 to 47 in this case). This makes it possible to obtain virtual images for classes 41 to 47. Note that the angle may be any angle for each class. For example, when obtaining a virtual image for class 41, this means that the imaging angle is not limited as long as it is between 22.5 degrees and 67.5 degrees.

The process proceeds to step S217, where the generator 25 processes the virtual image. Specifically, the generator 25 acquires features such as color and brightness of the images related to a series of data sets acquired from the class with the largest number of pairs (class 40 in this case), and assigns the acquired color, brightness, etc. to the virtual image. The process proceeds to step S219, where the generator 25 associates the processed virtual image with a comment, shooting angle, and shooting distance randomly selected from the same class with the smallest number of pairs, and stores them in the angle data set 32. This makes the number of pairs in classes 40 to 47 equal. If step S205 is NO, the generator 25 does not perform processing.

Next, the function of the learning unit 26 will be described. The learning unit 26 refers to the angle data set 32 to acquire a series of data sets for each class (classes 40 to 47) to generate a learning model. The learning unit 26 generates a learning model (evaluation comment generation model) in which three pieces of data, the vehicle image, the imaging angle, and the imaging distance, are input data from the series of data sets, and the evaluation comments on the vehicle image are output data. The model generation method is not particularly limited, and a well-known method is used. The learning unit 26 compares the evaluation comments generated by the evaluation comment generation model with the evaluation comments of actual users and calculates an error. Examples of methods for calculating the error include a method of comparing keywords, and a method of inputting the results into a machine learning language model represented by Transformer, BERT, etc., calculating the difference in the features of the output sentences, and outputting the results as an error. The learning unit 26 performs backward calculation using the calculated error, and updates the parameters of the learning model by the backpropagation method. The learning unit 26 stores the updated parameters in the learning model database 34. The learning unit 26 repeats learning hundreds of times for all data sets stored in the angle data set 32, searching for model parameters with small error.

When an image of the vehicle, the angle at which the image was captured, and the distance from the camera to the vehicle at the time the image was captured are input to the learning model (trained model) generated by the learning unit 26, an evaluation comment for this image is output. In order to output an image of the entire vehicle, the controller 20 generates images from different angles from a three-dimensional model of the vehicle, and generates evaluation comments for the images from each angle using the trained model. The controller 20 extracts adjectives related to keywords for specific parts from the generated evaluation comments, and judges the quality of the parts. However, if an adjective is extracted even though there is no keyword for the part, the controller 20 records this as the quality of the entire vehicle. The controller 20 maps the evaluation of the entire vehicle and the evaluation of each part to the three-dimensional model data.

(Action and Effect)
As described above, the learning device 1 according to this embodiment provides the following advantageous effects.

The learning device 1 includes a storage device 30 that associates and stores a first image of a vehicle with a user's evaluation comment on the first image, and a controller 20. The first image is an image obtained from the Internet via the communication I/F 10. The controller 20 identifies the vehicle model of the vehicle shown in the first image. The controller 20 generates a second image different from the first image using three-dimensional model data corresponding to the identified vehicle model. The controller 20 learns a model related to the user's evaluation comment using the first image, the second image, and the user's evaluation comment as input data, and stores the learned model in the storage device 30. According to the learning device 1, it is possible to increase the number of images by generating images at a small number of angles using a three-dimensional model of the identified vehicle model in images captured at a small number of angles. Then, since the model is learned after the number of images is increased, it is possible to generate a learning model with high robustness.

The storage device 30 stores three-dimensional model data of the vehicle body for each vehicle model, and shape features for each part of each vehicle model. The controller 20 extracts shape features of each part of the vehicle shown in the first image. The controller 20 compares the extracted shape features with the shape features stored in the storage device 30, and outputs the three-dimensional model data with the highest similarity. The similarity includes at least one of pixel value similarity, edge similarity, and shape feature similarity. The learning device 1 searches for three-dimensional model data of the vehicle body based on the shape features of the parts, making it possible to identify the vehicle model from the features of fine parts such as the grille and headlights.

The storage device 30 stores three-dimensional model data of the vehicle body for each vehicle model. The controller 20 extracts the shape characteristics of each part of the vehicle shown in the first image. If the shape characteristics of each part of each vehicle model are not stored in the storage device 30, the controller 20 acquires the angle at which the first image was captured and the distance from the camera to the vehicle at the time the first image was captured. The controller 20 acquires a virtual image for each vehicle model by virtually taking a photograph using the imaging angle and imaging distance for the three-dimensional model data for each vehicle model. The controller 20 compares each of the multiple virtual images with the first image and outputs the three-dimensional model data with the highest similarity. The similarity includes at least one of pixel value similarity, edge similarity, and shape feature similarity. This makes it possible to output the three-dimensional model data with the highest similarity even if the shape characteristics of each part of each vehicle model are not stored in the storage device 30.

A number of vehicle images, including a first image, are stored in the storage device 30. The controller 20 classifies the images according to the angles at which the images were captured. As an example, the controller 20 classifies the images into classes 40 to 47 according to the angle (see FIG. 3). The controller 20 obtains the color and brightness from the image classified into the angle with the highest number of images, and processes the second image corresponding to an angle other than the angle with the highest number of images using the color and brightness. This makes it possible to increase the variety of added images. An example of processing is the addition of color and brightness.

The controller 20 uses the output three-dimensional model data to generate an image with an angle different from that of the first image, and generates an evaluation comment from the model learned for each angle. This makes it possible to analyze the quality of the evaluation comment.

The controller 20 maps the evaluation of the entire vehicle and the evaluation of each part onto the three-dimensional model data. This makes it possible to intuitively grasp the quality of the entire vehicle and each part for each vehicle model.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processing devices, such as processors that include electrical circuitry. Processing circuits also include devices, such as application specific integrated circuits (ASICs) or circuit components, arranged to perform the described functions.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

1 Learning device, 20 Controller, 30 Storage device

Claims (7)

a storage device that stores a first image of a vehicle and a user's evaluation comment on the first image in association with each other;
A controller,
The controller:
Identifying the type of the vehicle shown in the first image;
generating a second image different from the first image using three-dimensional model data corresponding to the identified vehicle type;
Learning a model regarding the user's evaluation comments using the first image, the second image, and the user's evaluation comments as input data;
A learning device characterized in that a learned model is stored in the storage device. The storage device stores three-dimensional model data of a vehicle body for each vehicle type, and shape characteristics of each part of each vehicle type,
The controller:
Extracting shape features of each part of the vehicle shown in the first image;
comparing the extracted shape features with shape features stored in the storage device, and outputting the three-dimensional model data having the highest similarity;
2. The learning device according to claim 1, wherein the similarity includes at least one of a pixel value similarity, an edge similarity, and a feature similarity of the shape. The storage device stores three-dimensional model data of a vehicle body for each vehicle type,
The controller:
Extracting shape features of each part of the vehicle shown in the first image;
If the storage device does not store the shape characteristics of each part of each vehicle model, the angle at which the first image was captured and the distance from the camera to the vehicle at which the first image was captured are acquired;
A virtual image of each vehicle model is obtained by virtually taking a photograph using the angle and the distance for the three-dimensional model data of each vehicle model;
comparing each of the plurality of virtual images with the first image, and outputting three-dimensional model data having the highest similarity;
2. The learning device according to claim 1, wherein the similarity includes at least one of a pixel value similarity, an edge similarity, and a feature similarity of the shape. A plurality of images of the vehicle including the first image are stored in the storage device;
The controller:
classifying the plurality of images, including the image of the vehicle , according to angles at which the images were captured;
Obtain the color and brightness from the image classified into the angle with the most number of images,
The learning device according to any one of claims 1 to 3, characterized in that the second images corresponding to angles other than the angle with the largest number of images are processed using the color and brightness. The controller:
The learning device according to claim 2 or 3, characterized in that an image having an angle different from that of the first image is generated using the output three-dimensional model data, and an evaluation comment is generated from the learned model for each angle. The controller:
6. The learning device according to claim 5, wherein the evaluation of the entire vehicle and the evaluation of each part are mapped onto the three-dimensional model data. A learning method for a learning device including a storage device that stores a first image of a vehicle and a user's evaluation comment on the first image in association with each other, and a controller, the learning method comprising:
The controller:
Identifying the type of the vehicle shown in the first image;
generating a second image different from the first image using three-dimensional model data corresponding to the identified vehicle type;
Learning a model regarding the user's evaluation comments using the first image, the second image, and the user's evaluation comments as input data;
A learning method comprising storing a learned model in the storage device.

Images