JP7850196B2 - Evaluation system, evaluation program, and evaluation method - Google Patents

Description

This invention relates to an evaluation system, an evaluation program, and an evaluation method.

In recent years, with the legalization of "Level 4" autonomous driving, demonstration experiments of autonomous vehicles are being conducted throughout Japan. In autonomous driving, a vast amount of data is collected from the vehicle and used for the vehicle's AI to perceive events, make future predictions, plan driving routes, and improve (learn) the AI. For example, Non-Patent Document 1 below discloses the current status of such demonstration experiments of autonomous vehicles.

"Autonomous Driving Demonstrations Become Active Domestically and Internationally! What are the Challenges in Data Management?", [online], October 15, 2020, Autonomous Driving Lab, [Accessed July 7, 2023], Internet <URL: https://jidounten-lab.com/u_data-management-1>

Traditionally, evaluating language models (multimodal AI) that output descriptive text based on traffic-related images has involved comparing the text generated by the language model with the correct text according to predetermined evaluation metrics (word-based metrics such as BLEU and BERTscore, and context-based metrics such as sentenceBERT). However, none of the evaluation metrics used conventionally were optimized from the perspective of remote monitoring of autonomous driving.

To solve the above problems, an evaluation system according to one aspect of the present invention is an evaluation system for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input. The evaluation system comprises: a first evaluation unit that outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators; a second evaluation unit that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, constructed by machine learning based on a traffic-related perspective; and a third evaluation unit that outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value. The second evaluation unit inputs a correct answer sentence, which is text that best represents the content to be read from the image, and the text itself to the second language model. Based on a first indicator calculated by the second language model based on the content of the correct answer sentence and a second indicator calculated based on the content of the text, the second evaluation unit outputs a numerical value indicating the degree of deviation between the first indicator and the second indicator as the second evaluation value.

Furthermore, another aspect of the present invention relates to an evaluation program for evaluating the performance of a first language model that, upon input of an image related to traffic, outputs a text describing the content of the image. The evaluation program involves a computer performing: a first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators; a second evaluation process that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, constructed by machine learning based on a traffic-related perspective; and a third evaluation process that outputs the evaluation result of the first language model based on the first and second evaluation values. In the second evaluation process, the computer is instructed to input the correct answer text (a text that best represents the content to be read from the image) and the text itself to the second language model. The computer then outputs a numerical value indicating the degree of deviation between the first and second indicators, calculated by the second language model based on the content of the correct answer text and the content of the text, as the second evaluation value. A computer-readable recording medium storing a learning data generation program also falls within the scope of the present invention.

Furthermore, another aspect of the present invention relates to an evaluation method for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input. This method includes: a first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators; a second evaluation step in which the computer outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic-related perspective; and a third evaluation step in which the computer outputs the evaluation result of the first language model based on the first and second evaluation values. In the second evaluation step, the computer inputs the correct answer text, which is text that best represents the content to be read from the image, and the text itself to the second language model. The computer then outputs a numerical value indicating the degree of deviation between the first and second indicators, calculated by the second language model based on the content of the correct answer text and the content of the text, as the second evaluation value.

This is a block diagram showing an example of the functional configuration of an evaluation system according to an embodiment of the present invention. This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. This flowchart shows an example of a flow of an evaluation method according to an embodiment of the present invention.

<Evaluation System 100>
Hereinafter, an evaluation system 100 according to one embodiment of the present invention will be described in detail.

[Evaluation targets of the evaluation system]
The evaluation system 100 is a system for evaluating the performance of the first language model M1. The first language model M1 to be evaluated is a trained model constructed by machine learning using training data consisting of a set of traffic-related images and a sentence describing the content of the image. The traffic-related images are images obtained by cameras mounted on autonomous vehicles. The images may be of the outside or the inside of the vehicle. The images may also be videos or still images. When a new traffic-related image is input to the first language model M1, it outputs a sentence describing the content of the image.

[Configuration of the evaluation system]
The evaluation system 100 evaluates the performance of the first language model M1 based on the text output by the first language model M1. As shown in Figure 1, the evaluation system 100 comprises a first evaluation unit 1, a second evaluation unit 2, and a third evaluation unit 3.

[First Evaluation Department 1]
The first evaluation unit 1 outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators. The first evaluation unit 1 according to this embodiment includes a word evaluation unit 11, a context evaluation unit 12, and a calculation unit 13.

(Word evaluation unit 11)
The word evaluation unit 11 outputs a first numerical value based on the number of words in a given text that match the words in the corresponding correct answer text. The correct answer text is a sentence that describes the content of an image related to traffic input to the first language model M1, and is generated by means other than the first language model M1 (e.g., a person). In other words, the correct answer text is a sentence that shows the content that should be most easily read from the image. The correct answer text may be stored in a memory unit (not shown) or it may be generated each time an evaluation is performed. In this embodiment, the word evaluation unit 11 outputs a first numerical value using an evaluation index such as "BLEU" or "BERTscore". The first numerical value output by the word evaluation unit 11 in this embodiment is a number greater than 0 and less than or equal to 1.

(Contextual evaluation unit 12)
The context evaluation unit 12 outputs a second numerical value based on the similarity between the context of the sentence and the context of the correct answer sentence corresponding to that sentence. The word evaluation unit 11 according to this embodiment outputs a second numerical value using an evaluation index such as "sentenceBERT". The second numerical value output by the context evaluation unit 12 according to this embodiment is a number greater than 0 and less than or equal to 1.

(Calculation unit 13)
The calculation unit 13 calculates a first evaluation value based on the first and second numerical values. The first evaluation value is a statistical value of the first and second numerical values. The "statistical value" includes summation value, weighted summation value, mean, weighted mean, etc. The summation value is the sum of the first evaluation value and the second evaluation value. The mean is the average of the first evaluation value and the second evaluation value. The weighted mean is the average of the values obtained by multiplying at least one of the first evaluation value and the second evaluation value by a weighting coefficient. The first evaluation value is a number greater than 0 and less than or equal to 1. In other words, the closer the first evaluation value is to 1, the more accurate the text is evaluated by the first evaluation unit 1.

[Second Evaluation Section 2]
The second evaluation unit 2 uses the second language model M2 to output a second evaluation value indicating the accuracy of the text. The second language model M2 is a different language model from the first language model M1 and is a trained model constructed by machine learning based on the perspective of traffic. The second language model M2 used by the second evaluation unit 2 in this embodiment is a Large Language Model (LLM). Specifically, the second language model M2 consists of at least one of "BestScore,""SentenceBert," and "GPT4." By using the second language model M2 configured in this way, the accuracy of the text (the performance of the first language model M1) can be evaluated from a human perspective. In other words, texts that could not be distinguished by word-based evaluation metrics because the words used are almost the same (for example, "Because the truck went straight, the car stopped suddenly" and "Because the car stopped suddenly, the truck went straight," which are texts with reversed causal relationships) can be distinguished as texts with different content. Furthermore, sentences that could not be distinguished by context-based evaluation metrics (for example, "The truck went straight, so the car came to a sudden stop" and "The truck went straight, so the car came to a slow stop," sentences with nearly identical contexts) can also be distinguished as sentences with different content. Note that the second language model M2 may be a trained model other than a large-scale language model. The second evaluation unit 2 comprises a human perspective evaluation unit 21 and a second calculation unit 22.

(Human-perspective evaluation section 21)
The human perspective evaluation unit 21 inputs a text and the corresponding correct answer text to the second language model M2. The human perspective evaluation unit 21 comprises a first human perspective evaluation unit 211 and a second human perspective evaluation unit 212. The first human perspective evaluation unit 211 inputs the correct answer text to the second language model M2. The second human perspective evaluation unit 212 inputs a text to the second language model M2. The second language model M2, having received the text and the correct answer text, calculates a first index and a second index. The first index is an index based on the content of the correct answer text. The first index calculated by the second language model M2 according to this embodiment is a risk level indicating the degree of danger of the event indicated by the correct answer text. The second index is an index based on the content of the text. The second index calculated by the second language model M2 according to this embodiment is a risk level indicating the degree of danger of the event indicated by the text. The first index and the second index output by the second language model M2 according to this embodiment are numerical values greater than 0 and less than or equal to 1. The first-person perspective evaluation unit 211 then acquires the first index calculated by the second language model M2. The second-person perspective evaluation unit 212 then acquires the second index calculated by the second language model M2.

(Second calculation unit 22)
The second calculation unit 22 calculates a numerical value indicating the degree of discrepancy between the first and second indicators, based on the first and second indicators calculated by the second language model M2. Specifically, the second calculation unit 22 calculates a numerical value by substituting the first and second indicators into the following formula (1). The second calculation unit 22 then outputs the numerical value calculated by the second language model M2 as the second evaluation value. The second evaluation value output by the second calculation unit 22 is a numerical value greater than 0 and less than or equal to 1. In other words, the closer the second evaluation value is to 1, the smaller the discrepancy between the content of the text and the correct answer is (the higher the accuracy of the text), according to the second evaluation unit 2.
Numerical value indicating the degree of deviation = 1.0 - |First indicator - Second indicator| ... (1)

Furthermore, as shown in Figure 2, the second evaluation unit 2 may be configured to receive one or more pairs of example sentences describing traffic-related events and risk levels indicating the degree of danger of the events described in the example sentences, before inputting text into the second language model M2. In this way, by performing fusion learning, the second language model M2 can calculate a more accurate first indicator. As a result, the second evaluation unit 2 can output a more accurate second evaluation value.

Furthermore, as shown in Figure 3, the second evaluation unit 2 may be configured to receive instructions (prompts) from the second language model M2 to evaluate images from a traffic safety perspective, in addition to the correct answer sentence and text. These instructions indicate that the evaluation should be conducted from the perspective of a remote monitor of the autonomous vehicle. In this case, the second evaluation unit 2 outputs a numerical value as the second evaluation value, indicating the degree of deviation between the first and second indicators, based on the instructions received by the second language model M2. This allows the second language model M2 to calculate a more accurate first indicator. As a result, the second evaluation unit 2 can output a more accurate second evaluation value.

Furthermore, the second evaluation unit 2 may be configured to output a second evaluation value using Retrieval Augmented Generation (RAG). In this case, the evaluation system 100 further includes a database 4, as shown in Figure 4. The database 4 stores past texts describing the content of images related to past traffic, with multiple past texts containing different content. These past texts include texts previously output by the first language model M1, texts describing the content of traffic accidents recorded by the police, and texts about traffic accidents reported in the past. When the second evaluation unit 2 inputs text into the second language model M2, it extracts past texts similar in content to the text from among the multiple past texts stored in the database 4. The second evaluation unit 2 then inputs the extracted past texts, along with the text, into the second language model M2. This allows the second language model M2 to calculate a more accurate first index. As a result, the second evaluation unit 2 can output a more accurate second evaluation value.

[Third Evaluation Section 3]
The third evaluation unit 3 outputs an evaluation result of the first language model M1 based on the first evaluation value and the second evaluation value. In this embodiment, the third evaluation unit 3 outputs the product of the first evaluation value and the second evaluation value as the evaluation result. As described above, the first evaluation value and the second evaluation value are both numerical values greater than 0 and less than or equal to 1. Therefore, the evaluation result, which is the product of the first evaluation value and the second evaluation value, is also a numerical value greater than 0 and less than or equal to 1, and the closer it is to 1, the higher the performance of the first language model M1.

[Variations of evaluation system 100]
The first language model M1 may be constructed to output a formatted document. A "formatted document" refers to a document that contains essential elements. These elements include at least the vehicle's driving conditions (vehicle position, vehicle driving intention, objects, object positions, object movements, vehicle's current movements, etc.).
Furthermore, the first language model M1 may output multiple types of sentences, each with different descriptions of the driving conditions, when a single image is input. In this case, the first evaluation unit 1 and the second evaluation unit 2 may be configured to obtain multiple types of sentences from the first language model M1 and select the sentence from among the multiple types of explanatory sentences that has the greatest impact on the vehicle's future driving.

Furthermore, Figure 1 illustrates an evaluation system 100 in which the first language model M1 and the second language model M2 are not included in the configuration. However, at least one of the first language model M1 and the second language model M2 may be included in the evaluation system 100.

Furthermore, Figure 1 illustrates a case where text is directly supplied from the first language model M1 to the first evaluation unit 1 and the second evaluation unit 2. However, the evaluation system 100 may be configured to store the text output by the first language model M1 in a storage unit (not shown). The first evaluation unit 1 and the second evaluation unit 2 may then be configured to indirectly retrieve the text from the storage unit.

[Effects and benefits of evaluation system 100]
For example, the sentence "The truck went straight, so the car stopped abruptly" and the sentence "The car stopped abruptly, so the truck went straight" have reversed cause and effect relationships and are judged as completely different sentences from a human perspective. However, conventional word-based evaluation metrics could not distinguish between the two sentences because they use almost the same words. Similarly, the sentence "The truck went straight, so the car stopped abruptly" and the sentence "The truck went straight, so the car stopped slowly" describe different ways the car stopped and are judged as different sentences from a human perspective in terms of safety. However, conventional context-based evaluation metrics could not distinguish between the two sentences because their contexts are almost the same. In short, conventional methods had low accuracy in evaluating whether language models could correctly describe situations when high-risk events occurred, which is most important for the automation of remote monitoring. Therefore, it was still necessary for humans to verify whether the sentences generated by language models contained the information important for remote monitoring. However, human verification of sentences generated by language models and evaluation of language models are extremely time-consuming and costly. However, the evaluation system 100 described above has a third evaluation unit 3 that outputs an evaluation result for the first language model M1 by taking into account not only the first evaluation value, which indicates the structural accuracy of the text based on conventional evaluation indicators, but also the second evaluation value, which indicates the accuracy of the text as evaluated by the second language model M2 from the perspective of remote monitoring. For this reason, the evaluation system 100 makes it possible to accurately and easily evaluate the first language model M1 (confirming whether the text generated by the first language model M1 has extracted information that is important for remote monitoring).

<Evaluation Method S100>
Hereinafter, an evaluation method S100 according to another embodiment of the present invention will be described in detail.

[Evaluation Method S100 Flowchart]
The evaluation method S100 is a method for evaluating the performance of the first language model M1. As shown in Figure 5, the evaluation method S100 includes a first evaluation step S1, a second evaluation step S2, and a third evaluation step S3.

[First Evaluation Step S1]
In the first evaluation step S1, the computer outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators. The first evaluation value may be calculated using the first evaluation unit 1 of the evaluation system described above, or by other means.

[Second Evaluation Step S2]
In the second evaluation step S2, the computer uses the second language model M2 to output a second evaluation value indicating the accuracy of the text. In the second evaluation step S2, the computer inputs the correct answer and the text into the second language model M2. Also in the second evaluation step S2, the computer outputs a numerical value as the second evaluation value indicating the degree of deviation between the first and second indicators calculated by the second language model M2. The calculation of the second evaluation value may be performed using the second evaluation unit 2 of the evaluation system described above, or by other means.

[Third Evaluation Step S3]
In the third evaluation step S3, the computer outputs the evaluation result of the first language model M1 based on the first evaluation value and the second evaluation value. The output of the evaluation result may be performed using the third evaluation unit 3 of the evaluation system, or by other means.

[Effects of Evaluation Method S100]
The evaluation method S100 described above outputs the evaluation result of the first language model M1 in the third evaluation step S3, taking into account not only the first evaluation value, which indicates the structural accuracy of the text based on conventional evaluation indicators, but also the second evaluation value, which indicates the accuracy of the text as evaluated by the second language model M2 from the perspective of remote monitoring. Therefore, according to the evaluation method S100, the evaluation of the first language model M1 (confirmation of whether the text generated by the first language model M1 has extracted information important for remote monitoring) can be performed accurately and easily, similar to the evaluation system 100 described above.

<Different example>
The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

For example, each part of the evaluation system 100 is a program that causes a computer to function as that part, and can be realized by a learning data generation program that causes the computer to function as that part. In this case, each part is equipped with a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the learning data generation program. Each part is realized by executing each process of the learning data generation program (first evaluation process, second evaluation process, third evaluation process) using this control device and storage device. Note that the evaluation system 100 may be configured so that one computer realizes one of the parts, or one computer may be configured to realize two or more of the parts.

Furthermore, the above-mentioned learning data generation program may be recorded on one or more computer-readable recording media, rather than temporarily. These recording media may or may not be present in each unit. In the latter case, the program may be supplied to each unit via any wired or wireless transmission medium.

Furthermore, some or all of the functions of each component can be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each component are formed is also included within the scope of this invention. In addition, it is also possible to realize the functions of each component using, for example, a quantum computer.

〔summary〕
An evaluation system according to aspect 1 of the present invention is an evaluation system for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, and comprises: a first evaluation unit that outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation unit that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic perspective; and a third evaluation unit that outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein the second evaluation unit is configured to input a correct answer sentence, which is text that shows the content that should be most read from the image, and the text, to the second language model, and outputs a numerical value as the second evaluation value indicating the degree of deviation between the first index and the second index based on a first index based on the content of the correct answer sentence and a second index based on the content of the text calculated by the second language model.

The evaluation system according to embodiment 2 of the present invention may have the following configuration in embodiment 1 above: the first evaluation unit includes a word evaluation unit that outputs a first numerical value based on the number of words in the text that match the number of words in the correct answer sentence; a context evaluation unit that outputs a second numerical value based on the similarity between the context of the text and the context of the correct answer sentence; and a calculation unit that calculates the first evaluation value based on the first and second numerical values.

The evaluation system according to embodiment 3 of the present invention may be configured such that, in embodiment 2 described above, the first evaluation value and the second evaluation value are each numerical values greater than 0 and less than or equal to 1, the first evaluation value is a statistical value of the first numerical value and the second numerical value, and the third evaluation unit outputs the product of the first evaluation value and the second evaluation value as the evaluation result.

The evaluation system according to aspect 4 of the present invention may be configured such that, in any of aspects 1 to 3 described above, the first indicator is a risk level indicating the degree of danger of the event described in the correct answer statement, and the second indicator is a risk level indicating the degree of danger of the event described in the text.

The evaluation system according to aspect 5 of the present invention may be configured such that, in aspect 4 above, the second language model is a large-scale language model, and the second evaluation unit inputs one or more sets of example sentences describing traffic-related events and risk levels indicating the degree of danger of the events described in the example sentences into the second language model before inputting the text into the second language model.

The evaluation system according to aspect 6 of the present invention may be configured such that, in any of aspects 1 to 5 described above, it includes a database that stores multiple past texts with different content describing the content of images related to past traffic, and when the second evaluation unit inputs the text into the second language model, it extracts past texts with similar content to the text from among the multiple past texts, and inputs the extracted past texts together with the text into the second language model.

The evaluation system according to embodiment 7 of the present invention may be configured such that, in any of embodiments 1 to 6 described above, the second evaluation unit receives an instruction from the second language model to evaluate the image from the perspective of traffic safety, in addition to the correct answer sentence and the text, and the second language model outputs a numerical value indicating the degree of deviation between the first index and the second index calculated based on the instruction, as the second evaluation value.

The evaluation system according to aspect 8 of the present invention may be configured such that, in aspect 7 above, the traffic-related image is an image obtained by a camera mounted on an autonomous vehicle, and the instruction is an instruction to evaluate from the perspective of a remote monitor of the autonomous vehicle.

An evaluation program according to aspect 9 of the present invention is an evaluation program for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input. The program comprises: a first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators; a second evaluation process that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, constructed by machine learning based on a traffic-related perspective; and a third evaluation process that outputs the evaluation result of the first language model based on the first and second evaluation values. In the second evaluation process, the computer is instructed to input the correct answer text, which is the text that best represents the content to be read from the image, and the text itself to the second language model. The computer then outputs a numerical value indicating the degree of deviation between the first and second indicators, calculated by the second language model based on the content of the correct answer text and the content of the text, as the second evaluation value.

An evaluation method according to aspect 10 of the present invention is an evaluation method for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, and includes: a first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation step in which the computer outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic-related perspective; and a third evaluation step in which the computer outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein in the second evaluation step the computer inputs the correct answer text, which is text that shows the content that should be most read from the image, and the text itself to the second language model, and the computer outputs a numerical value indicating the degree of deviation between the first index and the second index as the second evaluation value, based on a first index based on the content of the correct answer text and a second index based on the content of the text, calculated by the second language model.

100 Evaluation System 1 First Evaluation Unit 11 Word Evaluation Unit 12 Context Evaluation Unit 13 Calculation Unit 2 Second Evaluation Unit 21 Human Perspective Evaluation Unit 22 Second Calculation Unit 3 Third Evaluation Unit 4 Database M1 First Language Model M2 Second Language Model S100 Evaluation Method S11 First Evaluation Step S12 Second Evaluation Step S13 Third Evaluation Step

Claims (10)

An evaluation system for evaluating the performance of a first language model that, when an image related to traffic is input, outputs text describing the content of the image,
A first evaluation unit outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators,
A second evaluation unit that uses a second language model different from the first language model, which is constructed by machine learning based on the perspective of traffic, and outputs a second evaluation value indicating the accuracy of the text,
A third evaluation unit outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value,
Equipped with,
The second evaluation unit is,
The second language model is given the correct answer sentence, which is the sentence that best represents the content to be read from the image, and the sentence itself.
Based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator calculated by the second language model, a numerical value indicating the degree of deviation between the first indicator and the second indicator is output as the second evaluation value.
Evaluation system. The first evaluation unit is,
A word evaluation unit outputs a first numerical value based on the number of words in the aforementioned text that match the number of words in the aforementioned correct answer text,
A context evaluation unit that outputs a second numerical value based on the similarity between the context of the aforementioned sentence and the context of the aforementioned correct answer sentence,
A calculation unit that calculates the first evaluation value based on the first numerical value and the second numerical value,
including,
The evaluation system according to claim 1. The first evaluation value and the second evaluation value are each a number greater than 0 and less than or equal to 1.
The first evaluation value is a statistical value of the first numerical value and the second numerical value,
The third evaluation unit outputs the product of the first evaluation value and the second evaluation value as the evaluation result.
The evaluation system according to claim 2. The first indicator is a risk level that indicates the degree of danger of the event described in the correct answer statement.
The second indicator is a risk level that indicates the degree of danger of the event described in the preceding text.
The evaluation system according to claim 1. The aforementioned second language model is a large-scale language model,
Before inputting the text into the second language model, the second evaluation unit inputs one or more sets of example sentences describing traffic-related events and risk levels indicating the degree of danger of the events described in the example sentences into the second language model.
The evaluation system according to claim 4. This is a historical document describing the content of images related to past transportation, and it includes a database that stores multiple historical documents with different content.
The second evaluation unit is,
When inputting the aforementioned text into the second language model, past texts that are similar in content to the aforementioned text are extracted from among multiple past texts.
The extracted past text is input into the second language model along with the text.
The evaluation system according to claim 1. The second evaluation unit is,
The second language model is given instructions to evaluate the image from a traffic safety perspective, in addition to the correct answer sentence and the text.
The second language model outputs a numerical value as the second evaluation value that indicates the degree of deviation between the first indicator and the second indicator calculated based on the instructions.
The evaluation system according to claim 1. The aforementioned traffic images are images obtained by cameras mounted on autonomous vehicles.
The aforementioned instruction is an instruction to evaluate the autonomous vehicle from the perspective of a remote monitor.
The evaluation system according to claim 7. An evaluation program for evaluating the performance of a first language model that, when an image related to traffic is input, outputs a text describing the content of the image,
On the computer,
A first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators,
A second evaluation process that uses a second language model different from the first language model, which is constructed by machine learning based on the perspective of traffic, to output a second evaluation value indicating the accuracy of the text,
A third evaluation process outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value,
Make it run,
In the second evaluation process, the computer,
The second language model is given the correct answer sentence, which is the sentence that best represents the content to be read from the image, and the sentence itself.
Based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator calculated by the second language model, a numerical value indicating the degree of deviation between the first indicator and the second indicator is output as the second evaluation value.
Evaluation program. An evaluation method for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input,
A first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators,
A second evaluation step in which a computer uses a second language model, which is different from the first language model and is constructed by machine learning based on the perspective of traffic, to output a second evaluation value indicating the accuracy of the text,
A third evaluation step in which the computer outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value,
Includes,
In the aforementioned second evaluation step,
The computer inputs the correct answer sentence, which is a sentence that best represents the content to be read from the image, and the sentence into the second language model.
The computer outputs a numerical value as the second evaluation value, which indicates the degree of discrepancy between the first indicator and the second indicator, based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator calculated based on the content of the sentence.
Evaluation method.