JP7850193B2 - Training data generation system, training data generation program, and training data generation method - Google Patents

Description

This invention relates to a training data generation system, a training data generation program, and a training data generation method.

In recent years, with the legalization of "Level 4" autonomous driving, demonstration experiments of autonomous vehicles are being conducted throughout Japan. In autonomous driving, a vast amount of data is collected from the vehicle and used for the vehicle's AI to perceive events, make future predictions, plan driving routes, and improve (learn) the AI. For example, Non-Patent Document 1 below discloses the current status of such demonstration experiments of autonomous vehicles.

"Autonomous Driving Demonstrations Become Active Domestically and Internationally! What are the Challenges in Data Management?", [online], October 15, 2020, Autonomous Driving Lab, [Accessed July 7, 2023], Internet <URL: https://jidounten-lab.com/u_data-management-1>

AI (trained models) are generally built using a large amount of training data available on the internet. Therefore, trained models specialized for specific domains, such as remote monitoring of autonomous vehicles, have not yet been developed to achieve sufficient accuracy (they suffer from problems such as hallucination). These problems can be solved by preparing domain-specific training data. However, training data specifically for remote monitoring of autonomous vehicles is not readily available and still needs to be created manually. The amount of training data required is enormous, and its creation requires a great deal of time and effort.

Furthermore, building a pre-trained model requires not only video footage but also text describing the content of the video. While language models capable of generating text from video have existed, building a pre-trained model specifically for remote monitoring of autonomous vehicles requires text accurately describing the vehicle's driving conditions, obtained from the vehicle's onboard camera. However, currently, technology for generating text accurately describing a vehicle's driving conditions has not been established.

To solve the above problems, a learning data generation system according to one aspect of the present invention includes an acquisition unit that, when captured video footage obtained from a camera mounted on a vehicle is input, acquires an explanatory text from a language model that outputs an explanatory text in a predetermined format that describes the situation of the vehicle, and an evaluation unit that evaluates the estimation accuracy of the language model for each of the multiple elements that constitute the explanatory text.
The system includes a first generation unit that generates training text for the language model based on the evaluation results of the estimation accuracy, and a second generation unit that generates training video showing the content of the training text, which is to be trained on the language model together with the training text.

Furthermore, a learning data generation program according to another aspect of the present invention causes a computer to execute the following processes: an acquisition process to obtain explanatory text from a language model that outputs explanatory text in a predetermined format describing the situation of a vehicle when captured video footage obtained from a camera mounted on the vehicle is input; an evaluation process to evaluate the estimation accuracy of the language model for each of the multiple elements constituting the explanatory text; a first generation process to generate learning text for the language model to learn based on the evaluation results of the estimation accuracy; and a second generation process to generate learning video showing the content of the learning text, to be learned by the language model together with the learning text. A computer-readable recording medium on which the learning data generation program is recorded also falls within the scope of the present invention.

Furthermore, another aspect of the present invention relates to a method for generating learning data, which includes: an acquisition step in which a computer obtains an explanatory text from a language model that outputs an explanatory text in a predetermined format describing the situation of a vehicle when an image captured by a camera mounted on the vehicle is input; an evaluation step in which the computer evaluates the estimation accuracy of the language model for each of the multiple elements constituting the explanatory text; a first generation step in which the computer generates a learning text for the language model to learn based on the evaluation result of the estimation accuracy; and a second generation step in which the computer generates a learning video showing the content of the learning text, to be learned by the language model together with the learning text.

This block diagram shows an example of the functional configuration of a learning data generation system and a model learning system according to an embodiment of the present invention. This figure illustrates the format of explanatory text generated by the language model included in the model learning system according to the same embodiment. This figure shows an example of a video feed that is input to the language model. This table shows an example of the evaluation results of the estimation accuracy of the language model by the learning data generation system according to the same embodiment. This figure illustrates how training texts are generated by the training data generation system according to the same embodiment. This figure illustrates how to generate training videos using the training data generation system according to the same embodiment. This table shows the contents of the training data (dataset) accumulated by the model learning system according to the embodiment of this model. This flowchart shows an example of the flow of a training data generation method and a model training method according to an embodiment of the present invention.

<Training Data Generation System 1 (Model Learning System 100)>
Hereinafter, a learning data generation device according to one embodiment of the present invention will be described in detail.

[Configuration of Model Learning System 100 (Model Learning System 100)]
As shown in Figure 1, the learning data generation system 1 according to this embodiment constitutes a model learning system 100. The model learning system 100 includes the learning data generation system 1, a language model 2, a storage unit 3, and a tuning unit 4.

[Language Model 2]
Language model 2 is a trained model constructed using machine learning, with training data consisting of a video and a sentence describing the content of the video. Therefore, when language model 2 receives video footage captured by a camera mounted on a vehicle, it outputs a descriptive sentence in a predetermined format that describes the situation of the vehicle. The video footage may be of the outside or the inside of the vehicle.

The prescribed format refers to the inclusion of essential elements in the explanatory text. These "elements" include at least the vehicle's driving conditions. Specifically, as shown in Figure 2, the "elements" include at least the vehicle's position and its intended driving behavior. Furthermore, the "elements" may include at least one of the following: objects visible in the captured video, their positions, their movements, and the vehicle's current movements. Note that "object positions" may further include both relative and absolute positions relative to the vehicle.

The language model 2 according to this embodiment may output multiple types of explanatory texts, each describing different driving conditions, when a single captured video is input. For example, when a captured video showing both a car and a bicycle (or pedestrian) simultaneously is input, as shown in Figure 3, the language model 2 generates explanatory texts about the car and explanatory texts about the bicycle. The language model 2 may be installed in the vehicle, or in the learning data generation system 1. Furthermore, the language model 2 may be installed in another device independent of the vehicle and the learning data generation system 1.

[Learning Data Generation System 1]
The learning data generation system 1 according to this embodiment includes an acquisition unit 11, an evaluation unit 12, a first generation unit 13, and a second generation unit 14. The learning data generation system 1 according to this embodiment further includes a selection unit 15, a second language model 16, a second storage unit 17, and a third storage unit 18.

(Second Language Model 16)
Second language model 16 is a pre-trained model different from language model 2. The second language model is a Large Language Model (LLM). Specifically, second language model 16 consists of at least one of "BestScore,""SentenceBert," and "GPT4." Note that second language model 16 may be a pre-trained model other than a Large Language Model.

(Second storage unit 17)
The second storage unit 17 stores the correct answer sentences used by the evaluation unit 12, which will be described later. The second storage unit 17 in this embodiment is composed of, for example, a semiconductor memory, a hard disk, or the like.

(Third storage unit 18)
The third storage unit 18 stores template images used by the second generation unit 14, which will be described later. In this embodiment, the third storage unit 18 is composed of, for example, a semiconductor memory, a hard disk, etc. The third storage unit 18 may be configured integrally with the second storage unit 17.

(Acquisition unit 11)
The acquisition unit 11 executes the acquisition process. In the acquisition process, the acquisition unit 11 acquires explanatory text from the language model. As described above, the language model 2 according to this embodiment outputs multiple types of explanatory text when a single captured video is input. Therefore, the acquisition unit 11 according to this embodiment acquires multiple types of explanatory text from the language model 2.

(Selection section 15)
The selection unit 15 performs a selection process. In the selection process, the selection unit 15 selects the explanatory text that has the greatest impact on the vehicle's future driving from among several types of explanatory texts. The selection of explanatory texts may be performed using a trained model, or the content of each explanatory text may be quantified and the selection may be based on these numerical values. Furthermore, if the language model 2 generates only one explanatory text for each captured video, the training data generation system 1 does not need to include the selection unit 15.

(Evaluation section 12)
The evaluation unit 12 performs an evaluation process. In the evaluation process, the evaluation unit 12 evaluates the estimation accuracy of the language model for each of the multiple elements that make up the explanatory text. As described above, the learning data generation system 1 according to this embodiment includes a selection unit 15. Therefore, the evaluation unit 12 according to this embodiment evaluates the estimation accuracy of the language model using the explanatory text selected by the selection unit 15. The evaluation unit 12 according to this embodiment calculates the accuracy rate for each element by comparing the explanatory text with the correct answer text using the second language model 16. As a result, as shown in Figure 4, the language model 2 is evaluated such that, for a certain element (for example, the position of a vehicle), it outputs an explanatory text stating that the vehicle is at an intersection with an accuracy rate of 30% for a video showing an intersection, and outputs an explanatory text stating that the vehicle is on a straight road with an accuracy rate of 100% for a video showing a straight road. The evaluation unit 12 may also be configured to check whether the format of the explanatory text is in English first before evaluating the explanatory text.

(First generation unit 13)
The first generation unit 13 executes the first generation process. In the first generation process, the first generation unit 13 generates training texts for the language model to learn from, based on the evaluation results of the estimation accuracy. In this embodiment, the first generation unit 13 generates training texts in which elements with low accuracy rates are weighted, as shown in Figure 5. That is, the learning data generation system 1 generates training data (a pair of training video and training text: teacher data) that improves the accuracy of elements with low estimation accuracy in the language model. In addition, the first generation unit 13 in this embodiment generates the training texts in the same format as the explanatory texts. The first generation unit 13 may also be configured to generate training data that improves the accuracy of all elements, or training data that further improves the accuracy of elements with high estimation accuracy.

(Second generation unit 14)
The second generation unit 14 executes a second generation process. In the second generation process, the second generation unit 14 generates a training video based on the training text generated by the first generation unit 13. The training video is a video that shows the content of the training text, which is to be trained on the language model together with the training text. In this embodiment, the second generation unit 14 generates the training video using pre-prepared template videos (stored in the third storage unit 18). Each template video has content that corresponds to each element of the explanatory text. For example, the template video includes a video of a road (straight road, intersection, T-junction, etc.) at the location of a vehicle indicated in the explanatory text. For example, if it is desired to generate a training video of a pedestrian crossing a straight road while a vehicle is moving straight down the road, the second generation unit 14 obtains a template video of a straight road from among multiple road template videos linked to the road map, as shown in Figure 6. The second generation unit 14 also obtains a template video of a pedestrian crossing the road from among the pedestrian template videos. The second generation unit 14 then generates a training video by combining the acquired template video of a pedestrian with the acquired template video of a straight road. In this way, the second generation unit 14 can generate training videos more easily than before (compared to creating them from scratch). Note that the template video may include elements that are not associated with any other elements (for example, elements unrelated to the position of a vehicle as indicated in the explanatory text).

[Storage unit 3]
The memory unit 3 is capable of storing the learning data (a pair of learning videos and learning texts: teacher data) generated by the learning data generation system 1. As a result, as the learning data generation system 1 repeatedly generates learning data, the memory unit 3 accumulates multiple pairs of learning texts in various situations (with different elements) and learning videos linked to the learning texts to be written, as shown in Figure 7. In this embodiment, the memory unit 3 is composed of, for example, a semiconductor memory or a hard disk. Note that the memory unit 3 may also be provided by the learning data generation system 1.

[Tuning Section 4]
The tuning unit 4 fine-tunes the language model using the learning data stored in the memory unit 3. The tuning unit 4 may be provided in the vehicle or in the learning data generation system 1. Alternatively, the tuning unit 4 may be provided in another device independent of the vehicle and the learning data generation system 1.

[Effects of the learning data generation system 1 (model learning system 100)]
The learning data generation system 1 described above automatically generates learning data (a pair of learning videos and learning texts: teacher data) specialized for the specific domain of remote monitoring by evaluating each element of an explanatory text in a predetermined format. Therefore, the learning data generation system 1 makes it possible to create a loop (model learning system 100) in which the language model 2 is automatically trained using the newly generated learning data, and the accuracy of text generation automatically improves. As a result, the effort of manually generating a large amount of learning data can be eliminated. In other words, the learning data generation system 1 makes it possible to easily train a language model.

Furthermore, according to the model learning system 100, the estimation accuracy of the language model (the accuracy of the content of the generated explanatory text) gradually improves.

<Training data generation method S1 (model training method S100)>
The following describes in detail a learning data generation method according to another embodiment of the present invention.

[Flow of Model Learning Method S100]
As shown in Figure 8, the training data generation method S1 according to this embodiment is part of the model learning method S100. The model learning method S100 includes the training data generation method S1 as well as a tuning step S2.

[Flow of training data generation method S1]
The learning data generation method S1 includes an acquisition step S11, an evaluation step S12, a first generation step S13, and a second generation step S14. The learning data generation method S1 according to this embodiment further includes a selection step S15.

(Acquisition step S11)
In the initial acquisition step S11, explanatory text is obtained from the language model 2. The acquisition of explanatory text may be performed using the learning data generation system 1 described above, or using other systems or devices. As described above, the language model 2 according to this embodiment outputs multiple types of explanatory text when a single captured video is input. Therefore, in the acquisition step S11 according to this embodiment, multiple types of explanatory text are obtained from the language model 2.

(Selection step S15)
After obtaining the explanatory text, the process moves to the selection step. In the selection step S15, the explanatory text that has the greatest impact on the vehicle's future driving is selected from among several types of explanatory texts. In this embodiment, the selection unit 15 makes the selection using a trained model. The selection of explanatory texts may be performed using the training data generation system 1 described above, or it may be performed using other systems or devices. Note that if the language model 2 generates only one explanatory text for each captured video, the training data generation method S1 does not need to include the selection step S15.

(Evaluation step S12)
After obtaining or selecting the explanatory text, the process moves to the evaluation step. In evaluation step S12, the estimation accuracy of the language model 2 is evaluated for each of the multiple elements that make up the explanatory text. The evaluation of estimation accuracy may be performed using the learning data generation system 1 described above, or it may be performed using other systems or devices.

(First generation step S13)
After evaluating the estimation accuracy of language model 2, the process moves to the first generation step. In the first generation step S13, training texts for training the language model are generated based on the evaluation results of the estimation accuracy. The training texts may be generated using the training data generation system 1 described above, or they may be generated using other systems or devices.

(Second generation step S14)
After generating the training text, the process moves to the second generation step. In the second generation step S14, training videos are generated based on the training text. The training videos may be generated using the training data generation system 1 described above, or they may be generated using other systems or devices.

(Tuning Step S2)
After generating the training data (a set of training text and training video), the process moves to tuning step S2. In tuning step S2, the language model 2 is fine-tuned using the training data. The fine-tuning of the language model 2 may be performed using the model learning system 100 described above, or it may be performed using other systems or devices.

[Effects of the training data generation method S1 and the model training method S100]
The learning data generation method S1 described above automatically generates learning data specific to the domain of remote monitoring by evaluating each element of an explanatory text in a predetermined format. Therefore, with learning data generation method S1, it is possible to create a loop in which the language model is automatically trained using the newly generated learning data, and the accuracy of text generation automatically improves. As a result, the effort of manually generating a large amount of learning data can be eliminated. In other words, with learning data generation method S1, it becomes possible to easily train a language model.

Furthermore, according to the model learning method S100, the estimation accuracy of the language model (the accuracy of the content of the generated explanatory text) gradually improves.

The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included within the technical scope of the present invention.

For example, the memory unit 3 may be provided by the learning data generation system 1.

For example, each part of the learning data generation system 1 is a program that causes the computer to function as each part, and each part can be realized by a learning data generation program that causes the computer to function as each part.
In this case, each component includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the above-mentioned learning data generation program. Each component is realized by executing the above-mentioned learning data generation program using this control device and storage device.
The above-described learning data generation program may be recorded on one or more computer-readable recording media, not temporary ones. Each unit may or may not have such recording media. In the latter case, the program may be supplied to each unit via any wired or wireless transmission medium.
Furthermore, some or all of the functions of each part can be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each part are formed is also included in the scope of this invention. In addition, it is also possible to realize the functions of each part by, for example, a quantum computer.

〔summary〕
A learning data generation system according to embodiment 1 of the present invention comprises: an acquisition unit that, upon input of captured video footage obtained from a camera mounted on a vehicle, acquires explanatory text from a language model that outputs explanatory text in a predetermined format describing the situation of the vehicle; an evaluation unit that evaluates the estimation accuracy of the language model for each of the multiple elements constituting the explanatory text; a first generation unit that generates learning text for the language model to learn based on the evaluation result of the estimation accuracy; and a second generation unit that generates learning video showing the content of the learning text, to be learned by the language model together with the learning text, based on the learning text.

In the learning data generation system according to aspect 2 of the present invention, the elements may be configured such that they include at least the vehicle's driving conditions, as described in aspect 1 above.

The learning data generation system according to embodiment 3 of the present invention may, in embodiment 2 described above, further include, at least, the position of the vehicle and the vehicle's driving intention, and at least one of the objects shown in the captured video, the position of the objects, the movement of the objects, and the current movement of the vehicle.

In the learning data generation system according to aspect 4 of the present invention, in any of aspects 1 to 3 described above, the evaluation unit may be configured to calculate the correct answer rate for each element by comparing the explanatory text with the correct answer text using a second language model different from the language model.

In the learning data generation system according to aspect 5 of the present invention, the first generation unit may be configured to generate the learning text in which the elements with low correct answer rates are weighted, as described in aspect 4 above.

In the learning data generation system according to embodiment 6 of the present invention, the first generation unit may be configured to generate the learning text in the same format as the explanatory text, as described in embodiment 5 above.

In the learning data generation system according to embodiment 7 of the present invention, the second generation unit may be configured to generate the learning video using a pre-prepared template video, as described in any of embodiments 1 to 6 above.

The learning data generation system according to aspect 8 of the present invention may be configured such that, in aspect 7 described above, the template video includes a video of the road at the location of the vehicle as indicated in the explanatory text.

The learning data generation system according to aspect 9 of the present invention may be configured such that, in any one of aspects 1 to 8 described above, the acquisition unit, upon receiving the captured video, acquires multiple types of explanatory texts from a language model that outputs multiple types of explanatory texts, each describing a different driving situation; and the selection unit selects the explanatory text from among the multiple types that has the greatest impact on the future driving of the vehicle; and the evaluation unit evaluates the estimation accuracy of the language model using the selected explanatory text.

The learning data generation program according to embodiment 10 of the present invention is configured to cause a computer to execute the following: an acquisition process to acquire explanatory text from a language model that outputs explanatory text in a predetermined format describing the situation of a vehicle when captured video footage obtained from a camera mounted on a vehicle is input; an evaluation process to evaluate the estimation accuracy of the language model for each of the multiple elements constituting the explanatory text; a first generation process to generate learning text for the language model to learn based on the evaluation results of the estimation accuracy; and a second generation process to generate learning video showing the content of the learning text, to be learned by the language model together with the learning text.

A learning data generation method according to aspect 11 of the present invention includes: an acquisition step in which a computer acquires an explanatory text from a language model that outputs an explanatory text in a predetermined format describing the situation of a vehicle when an image captured by a camera mounted on the vehicle is input; an evaluation step in which the computer evaluates the estimation accuracy of the language model for each of the multiple elements constituting the explanatory text; a first generation step in which the computer generates a learning text for the language model to learn based on the evaluation result of the estimation accuracy; and a second generation step in which the computer generates a learning video showing the content of the learning text, to be learned by the language model together with the learning text.

100 Model Learning System 1 Learning Data Generation System 11 Acquisition Unit 12 Evaluation Unit 13 First Generation Unit 14 Second Generation Unit 15 Selection Unit 16 Second Language Model 17 Second Memory Unit 18 Third Memory Unit 2 Language Model 3 Memory Unit 4 Tuning Unit S100 Model Learning Method S1 Learning Data Generation Method S11 Acquisition Step S12 Evaluation Step S13 First Generation Step S14 Second Generation Step S15 Selection Step S2 Tuning Step

Claims (11)

When video footage captured by a camera mounted on the vehicle is input, an acquisition unit acquires the explanatory text from a language model that outputs explanatory text in a predetermined format describing the status of the vehicle.
An evaluation unit evaluates the estimation accuracy of the language model for each of the multiple elements that constitute the explanatory text,
A first generation unit generates training texts for training the language model based on the evaluation results of the estimation accuracy,
A second generation unit generates a learning video showing the content of the learning text, which is to be trained on the language model together with the learning text, based on the learning text.
Equipped with,
A system for generating training data. The aforementioned elements include at least the driving conditions of the vehicle,
The learning data generation system according to claim 1. The aforementioned element is
This includes at least the location of the vehicle and the intention of the vehicle to travel,
The definition further includes at least one of the following: an object visible in the captured video, the position of the object, the movement of the object, and the current state of the vehicle.
The learning data generation system according to claim 2. The evaluation unit calculates the correct answer rate for each element by comparing the explanatory text with the correct answer text using a second language model different from the language model.
The learning data generation system according to claim 1. The first generation unit generates the training text in which the elements with low correct answer rates are weighted.
The learning data generation system according to claim 4.
The first generation unit generates the learning text in the same format as the explanatory text.
The learning data generation system according to claim 5.
The second generation unit generates the learning video using a pre-prepared template video.
The learning data generation system according to claim 1. The template video includes a video of the road at the location of the vehicle as indicated in the explanatory text.
The learning data generation system according to claim 7. When the captured video is input, the acquisition unit acquires multiple types of explanatory texts from a language model that outputs multiple types of explanatory texts, each containing different descriptions of driving conditions.
The system includes a selection unit that selects the explanatory document from among several types of explanatory documents that has the greatest impact on the future driving of the vehicle,
The evaluation unit evaluates the estimation accuracy of the language model using the selected explanatory text.
A learning data generation system according to any one of claims 1 to 8. On the computer,
When video footage captured by a camera mounted on the vehicle is input, an acquisition process is performed to obtain the explanatory text from a language model that outputs an explanatory text in a predetermined format that describes the status of the vehicle.
For each of the multiple elements that constitute the explanatory text, an evaluation process is performed to evaluate the estimation accuracy of the language model,
Based on the evaluation results of the estimation accuracy, a first generation process generates training texts for training the language model,
A second generation process generates a learning video showing the content of the learning text, which is to be trained on the language model along with the learning text, based on the learning text.
To execute
A program for generating training data. The computer, upon receiving video footage captured by a camera mounted on the vehicle, receives an input from a language model that outputs a descriptive document in a predetermined format describing the vehicle's condition. This includes an acquisition step of obtaining the descriptive document from the language model.
An evaluation step in which the computer evaluates the estimation accuracy of the language model for each of the multiple elements that constitute the explanatory text,
A first generation step in which the computer generates training text for the language model to be trained based on the evaluation result of the estimation accuracy,
A second generation step in which the computer generates a learning video showing the content of the learning text, based on the learning text, to be used to train the language model together with the learning text;
including,
Method for generating training data.