JP7741051B2 - Program, system, and method for evaluating operational monitoring - Google Patents

Description

The present invention relates to a program, system, and method for evaluating operational monitoring.

Research and development is underway on autonomously operated mobile objects such as autonomous vehicles, as well as technologies for remotely monitoring such autonomously operated mobile objects. For example, Patent Document 1 discloses technology for improving monitoring accuracy in a monitoring system that monitors multiple autonomously operated vehicles by assigning autonomously operated vehicles to each monitor based on the level of workload associated with alarms sent from the autonomously operated vehicles.

Japanese Patent Application Laid-Open No. 2020-052906

However, as autonomous driving services become more widespread, it is predicted that the number of trained operation monitors will increase significantly, and training for these operation monitors will be necessary. Therefore, it is desirable to efficiently evaluate the training of operation monitors.

Therefore, the present invention aims to provide technology that can efficiently evaluate the training of operation monitors who monitor autonomously operable mobile objects.

One aspect of the present invention is a program for evaluating the operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event, and causes a computer to execute a presentation step of presenting the alert and situation assessment information associated with the alert to the person being evaluated, and a reception step of receiving, after the presentation step, input from the person being evaluated regarding the situation assessment and operation assessment.

The present invention provides technology that can efficiently evaluate the training of operation monitors who monitor autonomously operable mobile vehicles.

1 is a diagram illustrating an example of an operation monitoring and evaluation system 1 according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of the hardware configuration of a traffic monitoring and evaluation device 10 and a traffic monitoring device 300 according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a functional block configuration of a traffic monitoring and evaluation device 100 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a functional block configuration of an alert issuing device 200 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a functional block configuration of a traffic monitoring device 300 according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an overview of a process for evaluating operation monitoring according to an embodiment of the present disclosure. FIG. 10 illustrates an example of determining how to process a plurality of alerts. FIG. 10 illustrates an example of determining how to process a plurality of alerts. FIG. 10 is a diagram illustrating an example of a flowchart of an evaluation process for operation monitoring executed in an embodiment of the present disclosure. FIG. 10 is a flowchart illustrating an exemplary operation monitoring evaluation process executed in an embodiment of the present disclosure. 1 is a diagram illustrating an example of an evaluation data creation system 2 according to an embodiment of the present disclosure. 1 is a diagram illustrating a functional block configuration of an evaluation data creation system 2 according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a flowchart of an exemplary evaluation data creation process executed in an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example of an operation monitoring and evaluation device 700 according to an embodiment of the present disclosure. 1 is a diagram illustrating an example of a traffic monitoring and evaluation system 12 according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example of a transportation monitoring and evaluation system 14 according to an embodiment of the present disclosure.

Embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that in each drawing, components with the same reference numerals have the same or similar configurations.

First, we will explain how we arrived at the embodiments of this disclosure described below.

In recent years, autonomous driving has attracted considerable attention around the world. In Japan, too, technological development of autonomous driving systems and the establishment of legal systems are underway, with the goal of launching Level 4 autonomous driving services nationwide by fiscal year 2025. Level 4 autonomous driving allows fully autonomous driving only under specific conditions. In Level 4 autonomous driving environments, the area in which an autonomous vehicle can operate is limited, such as on highways or in urban environments with a specified average speed. Within these limited areas and environments, the driving system fully controls the vehicle. This limited area is also known as the Operation Design Domain (ODD), and Level 4 autonomous driving is expected to be realized in areas such as airports and limited residential areas. Examples of Level 4 autonomous driving services include public transportation such as buses and taxis that operate along designated routes within airports, as well as transportation services such as transport vehicles. Level 4 autonomous driving, limited to the ODD, allows the autonomous vehicle to continue operating without driver intervention even when it encounters difficulties. Furthermore, if the autonomous vehicle encounters a situation that it cannot judge on its own, an alert from the autonomous vehicle will trigger a decision to leave the operation to the operation monitor in charge of a specified area within the ODD. In Level 4 autonomous driving services, for example, an operation monitor who performs remote monitoring is required to take appropriate action in the event of an accident. Operation monitoring requires that the operation monitor be able to grasp the surrounding environment, including the autonomous vehicle's driving status, system operation status, and direction of travel without delay, that the system be able to be activated and deactivated without delay by the operation of the operation monitor, and that the operation monitor be notified if continued autonomous driving is difficult (such as when the ODD deviates, there is a system abnormality, communication is lost, or an accident or disaster occurs).

In a society where autonomous driving services, such as Level 4 autonomous driving services, are rolled out nationwide, it is expected that multiple alerts will occur simultaneously as the number of vehicles in operation increases. It is desirable for operation monitors to be trained in a variety of situations so that they can respond to such events quickly and in cooperation with other operation monitors as appropriate. However, training for a variety of situations would require, for example, an experienced operation monitor to stay close to the trainee and evaluate the training results for each situation, which could result in significant personnel costs for sufficient training of operation monitors.

The inventors therefore considered ways to improve the efficiency of evaluating the training of operation monitors in an autonomous driving society, and came up with the embodiments of the present disclosure.

As described below, an embodiment of the present disclosure relates to an evaluation of operation monitoring of an autonomously operable mobile vehicle configured to issue an alert in response to the occurrence of a predetermined event. In the operation monitoring evaluation of an embodiment of the present disclosure, an alert and situational assessment information associated with the alert are presented to the person being evaluated, and then input regarding the situational assessment and operation judgment from the person being evaluated is accepted. Therefore, according to the operation monitoring evaluation of an embodiment of the present disclosure, the person being evaluated inputs information indicating the situational assessment based on the alert issued in response to the occurrence of a predetermined event on the autonomously operable mobile vehicle and the situational assessment information associated with the alert. The situational assessment information may be information regarding the mobile vehicle itself and/or the environment surrounding the mobile vehicle at the time of the occurrence of the predetermined event, such as information regarding the traffic environment around the autonomously operable mobile vehicle, video images inside and outside the vehicle captured by an imaging device such as a camera installed on the mobile vehicle, and audio information inside and outside the mobile vehicle. Therefore, the information regarding the situational assessment relates to the situation as understood by the person being evaluated based on information regarding the situation inside and outside the mobile vehicle when the predetermined event occurs. Furthermore, according to the operation monitoring evaluation of an embodiment of the present disclosure, the person being evaluated also accepts information indicating the operation judgment to respond to the situational assessment as understood by the person being evaluated at the time of the alert being issued. Information related to operational judgment may be, for example, information on how the assessee judges to respond to a situation that the assessee has grasped. In the evaluation of operational monitoring according to an embodiment of the present disclosure, by accepting information related to situation judgment and operational judgment, for example, the situation judgment and operational judgment can be used to obtain information on whether the assessee correctly grasped the situation based on alerts and situation judgment information, and whether they were able to find an appropriate operational judgment for the grasped situation, and these two items can be evaluated when evaluating the assessee, for example, as described below. Therefore, the evaluation of operational monitoring of the assessee to be trained can be carried out efficiently.

Furthermore, the operation monitoring evaluation method according to an embodiment of the present disclosure may be used, for example, to evaluate operation monitors when training operation monitors for autonomously operable mobile vehicles. When the operation monitoring evaluation method according to an embodiment of the present disclosure is used to evaluate operation monitors during training, for example, performance data from autonomous driving services conducted nationwide (information that can be used as information for situational judgment, such as autonomous vehicle alert information, surrounding traffic conditions, and operation instructions for operation monitors) can be used for training operation monitors. Training scenarios may also be randomly generated using alert information and information for situational judgment (such as the surrounding traffic environment, in-vehicle footage, and data collected by various sensors in the autonomous vehicle). For each generated training scenario, for example, the situational judgment made by an experienced operation monitor in response to an alert, as well as situational awareness point data (data related to the timing at which the situation was recognized and a situational judgment was made), may be accumulated in the performance data. For example, during operation monitoring training, the situational awareness point data may be used to determine the timing at which the operation monitor in training should make a situational judgment, and the operation monitor in training may be notified of this. Similarly, performance data may be compiled to record operational decisions made by experienced operational monitors in response to each alert, such as prioritizing alerts, assigning additional operational monitors when multiple alerts occur, arranging emergency vehicles, and providing external operational instructions.

In training operation monitors, performance data from past operation monitors conducted by experienced operation monitors may be used to make situational judgments based on situational judgment information, which is information on alerts and situational judgment information (situations), in various operation situations, and the trained operation monitors being evaluated may be evaluated on whether they are able to make appropriate operation judgments. Whether they are able to grasp the situation based on the alerts and situations may be determined automatically by utilizing the training system's screen operation and voice recognition technology and comparing it with the situational awareness points included in the scenario.

Furthermore, whether an operational decision is appropriate can be evaluated using statistical distribution and probability in comparison with operational decisions based on performance data by experienced operational monitors linked to alerts and situation understanding. When operational decisions are made based on performance data from operational monitoring by experienced operational monitors, even if two or more alerts occur and an additional operational monitor is assigned to prioritize them, or if two or more operational decisions are made, these can be counted as a single operational decision pattern, making it possible to also evaluate training scenarios in which multiple operational monitors make decisions about complex events in an autonomous driving society. Quantitative evaluation using statistical distribution, probability, etc. can numerically calculate the likelihood of decisions regarding situation understanding, alert prioritization, and the assignment of additional operational monitors, as well as operational decisions based on these, enabling training to be reviewed.

As described above, the operation monitoring evaluation method according to an embodiment of the present disclosure may be used in the evaluation of operation monitors during training for autonomously operated mobile vehicles. As described above, with the rollout of Level 4 autonomous driving services, fully autonomous driving will be possible without driver intervention within limited operational design areas such as airports. Therefore, remote operation monitoring, for example, will be desirable for responding to accidents, etc. By using the operation monitoring evaluation method according to an embodiment of the present disclosure in the evaluation of operation monitors during training for autonomously operated mobile vehicles, the operation monitors being trained can be evaluated efficiently. Furthermore, it becomes possible to perform more objective evaluations of operation monitors. Therefore, it becomes possible to efficiently evaluate and implement training, especially when training a large number of operation monitors. Therefore, it becomes possible to train operation monitors who can respond even when a large number of autonomous vehicles are introduced in a short period of time, such as when Level 4 autonomous driving services are rolled out.

The operation monitoring evaluation program, operation monitoring evaluation system, and operation monitoring evaluation method according to this embodiment will be described below with reference to the drawings.

One aspect of an embodiment of the present disclosure relates to a program for evaluating the operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event. The program of the present disclosure causes a computer to execute a presentation step of presenting an alert and situational assessment information associated with the alert to the person being evaluated, and a reception step of receiving, after the presentation step, input from the person being evaluated regarding the situation assessment and operation assessment.

Another aspect of an embodiment of the present disclosure relates to a system for evaluating operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event. The operation monitoring evaluation system according to an embodiment of the present disclosure comprises an operation monitoring evaluation device, an operation monitoring device, an alert issuing device, and an evaluation database. The evaluation database stores alerts presented to the person being evaluated, situation judgment information associated with the alert, situation judgments made by the person being compared based on the alert and the situation judgment information, alert classification results, and/or operation judgments corresponding to the situation judgments. The alert issuing device issues an alert to the operation monitoring device. The operation monitoring device is configured to execute a presentation step of presenting situation judgment information related to the alert to the person being evaluated in response to the issuance of the alert, and a reception step of receiving inputs, after the presentation step, regarding the person being evaluated's situation judgment, the classification results in which multiple alerts are classified into multiple sub-alert groups, and operation judgments. The operation monitoring evaluation device is configured to execute the steps of causing the alert issuing device to issue an alert based on the alert stored in the evaluation database, providing the operation monitoring device with situation judgment information stored in the evaluation database related to the alert issued from the alert issuing device, and evaluating the operation monitoring of the person being evaluated by comparing the situation judgment, classification results, and/or operation judgment made by the person being compared with the situation judgment, classification results, and/or operation judgment made by the person being evaluated.

Another aspect of an embodiment of the present disclosure relates to a method for evaluating operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event. The operation monitoring evaluation method according to an embodiment of the present disclosure includes a presentation step of presenting an alert and situation assessment information associated with the alert to a person being evaluated, and a receiving step of receiving, after the presentation step, input from the person being evaluated regarding the situation assessment and operation assessment.

<System Configuration>
FIG. 1 is a diagram illustrating an example of a traffic monitoring evaluation system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the traffic monitoring evaluation system 1 according to an embodiment of the present disclosure includes a traffic monitoring evaluation device 100, an alert issuing device 200, and traffic monitoring devices 300-1, 300-2, ..., 300-n (n is a natural number). The traffic monitoring evaluation device 100, the alert issuing device 200, and the traffic monitoring devices 300-1 to 300-n are connected by wire or wirelessly via a network 20, such as the Internet or a telephone network. The traffic monitoring devices 300-1 to 300-n are, for example, traffic monitoring devices used for evaluation. Hereinafter, any of the traffic monitoring devices 300-1 to 300-n will also be referred to as the traffic monitoring device 300. The traffic monitoring evaluation system 1 further includes a database (DB).

The operation monitoring evaluation device 100 may be, for example, an operation monitoring evaluation server, and can provide the operation monitoring device 300 with the data necessary to enable the evaluation of operation monitoring by the person being evaluated, which is performed in the operation monitoring device 300, and can also acquire information entered by the person being evaluated in the operation monitoring device 300. The operation monitoring evaluation device 100 may be composed of, for example, one or more physical servers, or may be composed using a virtual server running on a hypervisor, or may be composed using a cloud server.

The alert transmission device 200 is configured to transmit an alert to the operation monitoring device 300 in response to instructions from the operation monitoring evaluation device 100. The operation monitoring evaluation device 100 acquires information about alerts stored in a database DB, for example, and causes the alert transmission device 200 to transmit an alert to the operation monitoring device 300 based on the acquired information about the alert.

The operation monitoring device 300 is used to monitor an automatically operable mobile object performed by a monitor who is being evaluated. The operation monitoring device 300 receives an alert from the alert issuing device 200, and accepts situation judgment information, which is information on situation judgment materials regarding the status of the automatically operable mobile object at the time the alert is issued, obtained from the database DB by the operation monitoring evaluation device 100, and information on the situation judgment made by the operation monitor based on the alert (hereinafter also referred to as first information). The operation monitoring device 300 also accepts information on the operation judgment corresponding to the situation judgment made by the operation monitor (hereinafter also referred to as second information).

As described above, the database DB stores information used to evaluate operation monitors, and may store, for example, information about alerts issued when an individual (e.g., an experienced operation monitor) who is the subject of comparison in the evaluation performs operation monitoring, information on situation assessment materials regarding the status of the autonomously operable vehicle that was the subject of monitoring at the time the alert was issued, and information about the situation assessment and operation judgment made by the individual who is the subject of comparison.

<Hardware configuration>
FIG. 2 is a diagram illustrating an example of the hardware configuration of the traffic monitoring and evaluation device 100 and the traffic monitoring device 300 according to an embodiment of the present disclosure. The traffic monitoring and evaluation device 100 and the traffic monitoring device 300 include a processor 10a, such as a central processing unit (CPU) or a graphical processing unit (GPU), corresponding to a calculation unit; a memory 10b, such as a hard disk drive (HDD) and/or a solid state drive (SSD); a communication unit (interface) 10c for wired or wireless communication; an input unit 10d for accepting input operations; and an output unit 10e for outputting information. The input unit 10d is, for example, a keyboard, a touch panel, a mouse, and/or a microphone. The output unit 10e is, for example, a display and/or a speaker. Although detailed description is omitted, other devices, such as the alert issuing device 200, may also include at least a portion of the hardware configuration described above for the traffic monitoring and evaluation device 100 and the traffic monitoring device 300.

<Function block configuration>
(Training monitoring and evaluation device)
3 is a diagram illustrating an example of a functional block configuration of the operation monitoring and evaluation device 100 according to an embodiment of the present disclosure. The operation monitoring and evaluation device 100 includes a control unit 102, an acquisition unit 104, a communication processing unit 106, an alert information transmission unit 110, a situation determination information transmission unit 112, a storage unit 120, and an operation monitoring and evaluation unit 140. The control unit 102, the acquisition unit 104, the communication processing unit 106, the alert information transmission unit 110, the situation determination information transmission unit 112, and the operation monitoring and evaluation unit 140 can be realized by the processor 10a of the operation monitoring and evaluation device 100 executing a program stored in the storage unit 120, for example. The storage unit 120 can be realized using the memory 10b included in the operation monitoring and evaluation device 100. The program can be stored in a storage medium. The storage medium storing the program may be a non-transitory computer-readable medium. The non-transitory storage medium is not particularly limited, but may be, for example, a storage medium such as a USB memory or a CD-ROM.

The control unit 102 controls the alert information transmission unit 110, for example, so that an alert is output to the alert issuing device 200 at a predetermined timing. The control unit 102 also controls the situation judgment material information transmission unit 112 so that the situation judgment material included in the situation judgment material information is output to the operation monitoring device 300 at a predetermined timing.

The acquisition unit 104 acquires information related to alerts, situation assessment information (information related to situation assessment materials), information related to situation assessment, and information related to operation judgment from the database DB described below. For example, multiple alerts, situation assessment information, situation assessments, and operation judgments may be acquired, or only one of each may be acquired. The acquisition unit 104 also acquires data related to situation assessments and operation judgments from the operation monitoring device 300, which is information used in operation monitoring evaluations received by the operation monitoring device 300 from the operation monitor who is being evaluated.

The communication processing unit 106 performs various communications between the alert issuing device 200, the operation monitoring device 300, and the database DB. For example, the communication processing unit 106 transmits information about alerts obtained from the database DB to the alert issuing device 200 via the alert information sending unit 110.

The alert information transmission unit 110 obtains information about the alert to be transmitted from the alert transmission device 200 to the operation monitoring device 300 from the database DB as described above, and transmits it to the alert transmission device 200.

The situation assessment information transmission unit 112 transmits information regarding situation assessment information obtained from the database DB to the operation monitoring device 300.

The memory unit 120 includes a collected data DB 122 and an evaluation function DB 124. The collected data DB 122 stores data such as situation judgments, classification results, and operation judgments (described below) input by the operation monitor into the operation monitoring device 300. The evaluation function DB 124 stores data indicating evaluation functions for evaluating the judgment results regarding operation monitoring provided by the operation monitor. As described below, the evaluation function DB 124 stores evaluation functions related to probability and statistical distribution, for example.

The operation monitoring evaluation unit 140 evaluates the operation monitoring performed by the operation monitor, who is the person being evaluated, in the operation monitoring device 300. As described below, in the operation monitoring device 300, the operation monitor inputs a situation judgment, an alert classification result, and an operation judgment based on the presented alert and situation judgment information. The operation monitoring evaluation unit 140 comprises a situation judgment evaluation unit 142, a classification result evaluation unit 144, and an operation judgment evaluation unit 146. The situation judgment evaluation unit 142 evaluates the situation judgment (first information) indicated by the operation monitor, which is input by the operation monitor in the operation monitoring device 300 and transmitted from the operation monitoring device 300 to the operation monitoring evaluation device 100, by comparing it with the situation judgment indicated by the person being compared, obtained from a database DB, for example, using an evaluation function related to the evaluation of the situation judgment stored in the evaluation function DB 124. Similarly, the classification result evaluation unit 144 evaluates the alert classification results (the classification results of alerts into sub-alert groups, described below; hereinafter also referred to as third information) indicated by the traffic monitor, which are input into the traffic monitoring device 300 by the traffic monitor and transmitted from the traffic monitoring device 300 to the traffic monitoring evaluation device 100, by comparing them with the classification results indicated by the comparison target obtained from, for example, a database DB, using an evaluation function related to the evaluation of the classification results stored in the evaluation function DB. Furthermore, the operation judgment evaluation unit 146 evaluates the operation judgments (second information) indicated by the traffic monitor, which are input into the traffic monitoring device 300 by the traffic monitor and transmitted from the traffic monitoring device 300 to the traffic monitoring evaluation device 100, by comparing them with the operation judgments indicated by the comparison target obtained from, for example, a database DB, using an evaluation function related to the evaluation of operation judgments stored in the evaluation function DB 124. As described in detail below, the traffic monitoring evaluation unit 140 may also evaluate the combination of alerts and operation judgments, and the order in which alerts are processed.

Figure 3 also shows the database (DB). As shown in Figure 3, the database (DB) stores information regarding: an alert group A including alerts a1-a4 issued by an autonomously operable vehicle that was the subject of monitoring when the comparison person performed operation monitoring; a situation determination element group S including situation determination elements s1-s5 related to the environment inside and outside the vehicle at the time the alert was issued; a situation determination group C including situation determinations c1-c3 provided by the comparison person; and an operation determination group D including operation determinations d1-d3 provided by the comparison person. The database (DB) may also include information regarding which of sub-alert groups g1 and g2 the alerts a1-a4 included in alert group A are classified into. As shown in Figure 3, for alert group A, for example, sub-alert group g1 may include alerts a1 and a3, and sub-alert group g2 may include alerts a2 and a4. The information regarding sub-alert groups is not limited to this; for example, information regarding which alerts are classified into which sub-alert group may be managed in a table or the like, separate from alert group A.

(Alert sending device)
FIG. 4 is a diagram illustrating an example functional block configuration of the alert issuing device 200 according to an embodiment of the present disclosure. The alert issuing device 200 includes a control unit 202, an acquisition unit 204, a communication processing unit 206, an alert output unit 210, and a storage unit 220. The control unit 202, the acquisition unit 204, the communication processing unit 206, and the alert output unit 210 can be realized by the processor 10a of the alert issuing device 200 executing a program stored in the storage unit 220, for example. The storage unit 220 can be realized using the memory 10b included in the alert issuing device 200. As with the operation monitoring and evaluation device 100, the program can be stored in a storage medium. The storage medium storing the program may be, for example, a computer-readable non-transitory storage medium such as a USB memory or a CD-ROM.

The control unit 202 controls the alert output unit 210, for example, so that an alert is sent to the operation monitoring device 300 at a predetermined timing.

The acquisition unit 204 acquires information about alerts sent to the operation monitoring device 300 from the operation monitoring evaluation device 100.

The communication processing unit 206 performs various communications between the operation monitoring evaluation device 100 and the operation monitoring device 300. For example, the alert output unit 210 sends an alert to the operation monitoring device 300 via the communication processing unit 206.

(Operation monitoring device)
FIG. 5 is a diagram illustrating an example of a functional block configuration of a traffic monitoring device 300 according to an embodiment of the present disclosure. The traffic monitoring device 300 includes a control unit 302, an acquisition unit 304, a communication processing unit 306, a storage unit 320, an input unit 350, and an output unit 360. The control unit 302, the acquisition unit 304, and the communication processing unit 306 can be implemented by the processor 10a of the traffic monitoring device 300 executing a program stored in the storage unit 320, for example. The storage unit 320 can be implemented using the memory 10b included in the traffic monitoring device 300. The program can be stored in a storage medium. The storage medium storing the program may be, for example, a computer-readable non-transitory storage medium such as a USB memory or a CD-ROM.

The control unit 302 controls the alert output unit 362, for example, so that an alert is output at a predetermined timing. The control unit 302 also controls the situation assessment information output unit 364 so that situation assessment information is output at a predetermined timing.

The acquisition unit 304 acquires, for example, information regarding situation judgment materials (situation judgment material information) transmitted from the operation monitoring evaluation device 100.

The communication processing unit 306 performs various communications between the operation monitoring device 300, the alert issuing device 200, etc.

The memory unit 320 stores, for example, the collected data DB 322. The collected data DB 322 stores data such as the situation judgments, classification results, and operation judgments (described below) provided by the operation monitor who is being evaluated.

The output unit 360 includes, for example, an alert output unit 362, a situation judgment information output unit 364, and an evaluation result output unit 366. The output unit 360 may include, for example, a display device. The alert output unit 362 may display an alert, for example, when notifying the user of the issuance of an alert on the operation monitoring device 300 at the time of issuance of the alert. Alternatively, the alert may be configured to be notified to the user by audio, for example. Furthermore, the alert may be configured to be notified to the user by image and audio, for example. The situation judgment information output unit 364 outputs situation judgment information, which is information regarding situation judgment information related to the issued alert, in response to the issuance of an alert. The situation judgment information may also be presented to the user by image and/or audio, for example. For example, if the situation judgment information is image data, it may be output by display, or if the situation judgment information is audio data, it may be output by audio. The evaluation result output unit 366 includes, for example, a situation judgment evaluation result output unit 366a, a classification result evaluation result output unit 366b, and an operation judgment evaluation result output unit 366c. When the operation monitoring evaluation device 100 obtains evaluation results for the situation judgment, classification result, and/or operation judgment indicated by the operation monitor who is the person being evaluated, the evaluation result output unit 366 outputs, for example by displaying, the evaluation results of the operation monitoring performed by the person being evaluated, using the situation judgment evaluation result output unit 366a, the classification result evaluation result output unit 366b, and the operation judgment evaluation result output unit 366c, respectively.

The input unit 350 includes, for example, a situation judgment input unit 352, a classification result input unit 354, and an operation judgment input unit 356. The operation monitor, who is the person being evaluated, inputs the situation judgment, classification results, and/or operation judgment, for example, via the situation judgment input unit 352, the classification result input unit 354, and/or the operation judgment input unit 356.

<Processing procedure>
Referring to FIG. 6 , the process of evaluating operation monitoring according to an embodiment of the present disclosure will be described. FIG. 6 is a diagram illustrating an overview of the process of evaluating operation monitoring according to an embodiment of the present disclosure. The following description will be given using an example in which the operation monitor, who is the person being evaluated, undergoes operation monitoring training and performs an evaluation of the operation monitoring. However, the evaluation of operation monitoring according to an embodiment of the present disclosure may be used for evaluations other than the training of operation monitors. For example, the operation monitor may undergo training to monitor multiple autonomously operable vehicles. For example, operation monitoring data from a comparison subject (e.g., an experienced operation monitor) who has actually monitored multiple autonomously operated vehicles may be used in the training. Furthermore, operation monitoring data from multiple comparison subjects may also be used. For example, multiple operation monitors may be asked to monitor the same group of autonomously operated vehicles in the same environment, and the data related to operation monitoring performed by the multiple operation monitors may be used for training. As described below, data related to operation monitoring performed by multiple operation monitors can be used to statistically confirm which judgment is most likely, even in situations where judgments vary. Therefore, by using data obtained from multiple operation monitors as a comparison subject, the accuracy of the judgment of the person being evaluated can be evaluated.

First, an alert and situation assessment information are presented to the operation monitor (operation monitor A), who is being evaluated (presentation step in this embodiment). That is, an alert is first sent to operation monitor A, who has begun training, at a predetermined timing. For example, the operation monitor, who is being evaluated, uses a personal computer with a display device as the operation monitoring device 300 and monitors the video of multiple autonomous vehicles being monitored, displayed on the display device. For example, assuming the above-mentioned Level 4 autonomous driving service, training is conducted by simultaneously monitoring multiple buses operating autonomously without a driver in a limited area. The operation monitor assumes alerts to be issued by the autonomous vehicles being monitored in training, and issues alerts based on historical information of alerts issued by autonomous vehicles monitored by a comparison person, such as an experienced operation monitor. For example, if the comparison person monitors for 120 minutes and alerts are issued 20, 50, 70, and 100 minutes after the start of monitoring, alerts may also be issued during training at times 20, 50, 70, and 100 minutes after the start of training. The alert may be issued by the alert issuing device 200 described above. The alert issuing device 200 may issue the alert, for example, by image or audio. The operation monitor may also be notified of the issuance of the alert by image and audio. Alternatively, an alert issuing unit may be provided in the operation monitoring device 300, and the alert may be displayed on the display unit of the operation monitoring device 300. When multiple autonomous vehicles are being monitored simultaneously, information regarding the alert may be displayed as an image, such as text or an icon, superimposed on a portion of the entire display screen, for example, on the surveillance video of the autonomous vehicle from which the alert is issued. Alternatively, rather than superimposing the alert on the surveillance video of the autonomous vehicle from which the alert is issued, an alert may be displayed in the center of the display device, for example, or an image related to the alert may be displayed on an edge of the display device where no surveillance video of any autonomous vehicle is displayed. The example shown in FIG. 6 illustrates the simultaneous occurrence of alerts a1, a2, a3, and a4. The alert may be, for example, a vehicle accident alert or an in-vehicle personal injury alert.

Furthermore, situation judgment material information, which is information related to situation judgment material related to the issued alert, is presented to the operation monitor. The example shown in Figure 6 shows a case where situation judgment material information s1, s2, s3, s4, and s5 are presented. The situation judgment material may be, for example, interior or exterior vehicle images captured by an imaging device such as a camera installed in the autonomous vehicle. Furthermore, if the autonomous vehicle is equipped with various sensors, information acquired by the sensors may be used as situation judgment material. For example, if the autonomous vehicle is equipped with an audio sensor, audio information when an alert is issued may be presented as situation judgment material information. In addition, information from a temperature sensor, humidity sensor, and/or temperature and humidity sensor may be presented as situation judgment material information. Alternatively, information such as images, audio, or text from passengers or crew members inside the autonomous vehicle may be presented as situation judgment material information. Information from other vehicles may also be used as situation assessment information. For example, when the autonomous vehicle that issues the alert is included in video captured by an imaging device mounted on the exterior of a vehicle traveling near the autonomous vehicle that issues the alert, information such as video from the other vehicle may be presented as situation assessment information. The situation assessment information may be presented to the trainee, the operation monitor, simultaneously or nearly simultaneously with the issuance of the alert. For example, the situation assessment information may be presented several seconds after the alert is issued. The situation assessment information may also be presented for any length of time. For example, the situation assessment information may be presented for a predetermined period of time, such as several minutes, immediately after the alert is issued. Alternatively, the situation assessment information may be presented until the operation monitor makes some kind of decision. The system may be configured to accept an input from the operation monitor indicating that the information does not need to be presented, and to be able to present the information until the operation monitor indicates that the information does not need to be presented. Furthermore, when multiple situation assessment information is presented, the time at which each information is presented may vary depending on the situation assessment information. The situation assessment information may be information at the moment the alert is issued, or, for example, information before, before, after, or after the alert is issued. For example, if video footage of the inside of a vehicle is presented as information for assessing the situation, video footage spanning four minutes from one minute before to three minutes after the alert is issued may be presented.

Next, input regarding the situation assessment and operation judgment from the operation monitor, who is the person being evaluated, is accepted (the acceptance step in this embodiment). First, the operation monitor makes a situation assessment based on the presented alerts a1-a4 and situation assessment elements s1-s5. A situation assessment may be displayed for all alerts, for example. The example shown in Figure 6 shows a case in which situation assessments c1, c2, c3, and c4 are displayed for the issued alerts a1, a2, a3, and a4, respectively. For example, if alert a1 is an in-vehicle accident alert, and situation assessment element s1, which is a video of the interior of the vehicle at the time alert a1 is issued, shows a passenger inside the vehicle falling, and situation assessment element s2, which is an audio from inside the vehicle, shows the sound of the passenger falling, situation assessment c1 may indicate that a passenger has fallen inside the vehicle based on situation assessment elements s1 and s2.

The operation monitor may then classify the alerts (the classification result receiving step in this embodiment). The example shown in Figure 6 shows a case where alerts a1, a2, a3, and a4 are classified into either sub-alert group g1 or g2. That is, Figure 6 shows a case where alerts a1 and a3 are classified into sub-alert group g1, and alerts a2 and a4 are classified into sub-alert group g2. Alert classification may be based on information such as the content of alerts a1 to a4, situation determination factors s1 to s5, and situation determinations c1 to c4 indicated based on alerts a1 to a4 and situation determination factors s1 to s5. For example, if it is determined that alerts a1 and a3 of alerts a1 to a4 will be handled by the assessee themselves, alerts a1 and a3 may be classified into sub-alert group g1, which includes alerts to be handled by the assessee themselves, and if it is determined that the remaining alerts a2 and a4 will be handled by another operation monitor (operation monitor B), alerts a2 and a4 may be classified into sub-alert group g2, which includes alerts to be handled by the other operation monitor. Alternatively, alerts that are to be handled with priority may be classified into sub-alert group g1, and alerts with a relatively low priority may be classified into sub-alert group g2.

Information regarding the alert classification results (third information) may be input, for example, by the input unit 350 of the operation monitoring device 300. The input information may include, for example, a determination result that two or more alerts can be processed together, that is, that multiple alerts can be processed by a single operation decision. For example, information that alerts a1 and a3 can be processed together may be included. Whether two or more alerts can be processed together is not limited to when classifying the alerts, and may be determined, for example, when making a situation judgment or when making an operation decision, as described below. For example, if it is determined that two alerts can be resolved by arranging for an emergency vehicle once, it may be determined that these two alerts can be processed together.

Furthermore, the operation monitor makes an operation judgment. For example, the operation judgment may be made only for alerts that the assessee determined in a previous process to handle themselves. Figure 6 illustrates a case in which operation judgments d1 and d3 are displayed only for alerts a1 and a3 classified in sub-alert group g1, which the assessee, Operation Monitor A, has determined to handle. Therefore, as shown in Figure 6, operation judgments d2 and d4 may be displayed by another operation monitor (Operation Monitor B) for alerts a2 and a4 classified in sub-alert group g2, which another operation monitor (Operation Monitor B) has determined to handle. The other operation monitor B, who has been requested to handle alerts a2 and a4, may be presented with situation judgment information related to alerts a2 and a4 (e.g., situation judgment information s2 and s4), and may make situation judgments c2 and c4 and operation judgments d2 and d4. The other operation monitor B may also be an assessee (trainee). The requested other operation monitor B may also request the received alerts from another operation monitor, if necessary. The requested other operation monitor B may also provide information to the requesting operation monitor via voice call, etc. The requested other operation monitor B may consider the situation judgment materials s2 and s4 themselves and make situation judgments c2 and c4 and operation judgments d2 and d4, or, for example, may make operation judgments based solely on the situation judgments c2 and c4 made by the requesting operation monitor A, without checking the situation judgment materials s2 and s4. Furthermore, when the other operation monitor B checks the situation judgment material information, the requesting operation monitor A may select the situation judgment materials that the other operation monitor B should check, or the other operation monitor B may select all of the situation judgment materials s1 to s5 as the situation judgment materials to check themselves.

For example, if the alert is a vehicle breakdown alert, the operation decision may include guiding passengers or arranging for a loaner vehicle as an alternative means of transportation, or arranging for maintenance of the broken-down vehicle, such as arranging for a tow truck. If a loaner vehicle is arranged, the number of loaner vehicles may be determined, for example. Also, if the alert is an alert regarding a personal injury accident inside the vehicle, the operation decision may indicate, for example, arranging for an ambulance, or transport to the nearest medical institution or fire station. Alternatively, the operation decision may be to continue operation if the personal injury accident resulted in minor injuries or no injuries, such as if a passenger who fell quickly gets up. Thus, in embodiments of the present disclosure, the operation decision may include a decision that no action is required in response to the alert and that operation can continue.

Next, an evaluation of the operation monitor may be carried out (this embodiment is the evaluation step). The operation monitor being evaluated may be evaluated, for example, by comparing the situation judgment, classification results, and/or operation judgment made by a comparison target, such as an experienced operation monitor, in response to the alerts a1-a4 and the situation judgment materials s1-s5 sent to the operation monitor with the situation judgment, classification results, and/or operation judgment made by the operation monitor based on the same alerts a1-a4 and situation judgment materials s1-s5. In an embodiment of the present disclosure, for example, evaluation may be performed on only one, any two, or all three of the situation judgment, classification results, and operation judgment. Furthermore, these evaluations may be performed simultaneously or separately. For example, in operation monitoring according to an embodiment of the present disclosure, situation assessment, alert classification, and operation judgment may be performed in this order. In this case, the situation assessment may be indicated first, followed by an evaluation of the situation assessment, the alert classification may be evaluated first, and the operation judgment may be indicated, followed by an evaluation of the operation judgment. Alternatively, the situation assessment, alert classification, and operation judgment evaluation may all be performed together after the operation judgment evaluation has been performed.

In an embodiment of the present disclosure, the evaluation of operation monitoring may be performed using statistical distributions and/or probabilities calculated based on the situation judgments, classification results, and/or operation decisions made by the person being compared, as described below. That is, in the evaluation of operation monitoring according to this embodiment, a calculation step may be executed to calculate statistical distributions and/or probabilities calculated based on the situation judgments, classification results, and/or operation decisions made by the person being compared.

First, an example of the evaluation of the situation judgment made by the operation monitor, who is the person being evaluated in an embodiment of the present disclosure (the situation judgment evaluation step in this embodiment) will be described. The situation judgment may be evaluated, for example, based on the probability that the situation judgment made by the operation monitor corresponds to the accuracy rate relative to the situation judgment made by the person being compared. For example, the probability corresponding to the accuracy rate may be evaluated based on the integrated value of the deviation between the situation judgment made by the operation monitor and the situation judgment made by the person being compared. The evaluation function for evaluating the situation judgment may be calculated, for example, as follows:

For example, if the group of alerts that have occurred is A, the group of situational judgment factors that have occurred is S, and the group of situational judgments that can be made is C, the situational judgment made by the operation monitor may be expressed as shown in the following formula (1) by determining which of the situational judgments included in the situational judgment group was judged to have actually occurred when the person being compared monitored the operation.

For example, if the group of possible situation judgments C includes n situation judgments c1, c2, ..., cn, then of the n situation judgments, the situation judgments for which the operation monitor judges that an incident has occurred are expressed as a set of values of 1, and the situation judgments for which the operation monitor does not judge that an incident has occurred are expressed as a set of values of 0. Similarly, the group of situation judgments made by the person to be compared is expressed as a set of values of 1 for the situation judgments for which the operation monitor to be compared judges that an incident has occurred, and 0 for the situation judgments for which the operation monitor judges that an incident has not occurred, as shown in the following formula (2).

The evaluation function can be expressed, for example, by the following equation (3), based on the accumulated value for all situation judgment groups of the difference between the set consisting of {0, 1} indicating whether or not a situation judgment included in situation judgment group C has occurred, as indicated by the operation monitor who is the person being evaluated, and the set consisting of {0, 1} indicating whether or not a situation judgment included in situation judgment group C has occurred, as indicated by the person being compared.

As expressed in Equation (3), the evaluation _E1 of the situation judgment of the operation monitor being evaluated increases as the value approaches 1. For example, if the operation monitor determines that a comparison target, such as an expert, did not occur in a situation judgment that indicated that a comparison target had occurred, the value of Equation (3) decreases. Therefore, the value of probability _E1 calculated by Equation (3) corresponds to the accuracy rate of the situation judgment of the operation monitor being evaluated. In other words, the more situation judgments the operation monitor being evaluated determines that a comparison target did not occur in a situation judgment that indicated that a comparison target had occurred, or the more situation judgments the operation monitor being evaluated determines that a comparison target had occurred in a situation judgment that indicated that a comparison target had not occurred, the smaller the value _E1 of Equation (3), and therefore the lower the evaluation of the operation monitor being evaluated. By evaluating the accuracy rate of situation judgment, it is possible to, for example, evaluate whether the operation monitor being evaluated correctly recognized that a person had fallen and required emergency treatment based on the audio and video inside the autonomous vehicle.

Next, an example of an evaluation of the classification results (classification result evaluation step in this embodiment) in which a traffic monitor, who is the subject of evaluation in an embodiment of the present disclosure, classifies multiple alerts into multiple sub-alert groups will be described. The alert classification results may be evaluated using a statistical distribution calculated based on the classification results provided by a comparison subject, such as an experienced traffic monitor. For example, the classification results provided by multiple comparison subjects may include data representing the probability for each sub-alert group, regarding which of the multiple sub-alert groups into which each alert has been classified that the comparison subject has decided to handle. In this case, the classification results provided by the subject of evaluation may be evaluated using an evaluation function calculated based on the sum of the squared deviations between the classification results provided by the traffic monitor, who is the subject of evaluation, and the classification results provided by the comparison subject. The evaluation function used may include a function based on a chi-squared distribution using the sum of the squared deviations between the classification results provided by the subject of evaluation and the classification results provided by the comparison subject for some or all of the sub-alert groups.

The evaluation function for evaluating the classification results may be found, for example, as follows: For example, similar to the example of the evaluation function for evaluating situation judgment described above, if the group of alerts that has occurred is A, the group of materials for situation judgment that has occurred is S, and the power set of the sub-alert groups is Ag, the classification performed by the operation monitor is expressed by the following formula (4) depending on which of the sub-alert groups the evaluatee himself is responsible for.

That is, as shown in formula (4), the classification results of the alerts indicated by the operation monitor being evaluated are expressed as a set of values, for example, if the power set Ag of sub-alert groups contains j sub-alert groups g1, g2, ..., gj (j is a natural number), then of the j sub-alert groups g1 to gj, those sub-alert groups that the operation monitor being evaluated decided to handle themselves are given a value of 1, and those sub-alert groups that the operation monitor decided not to handle themselves (decided that another operation monitor would handle them) are given a value of 0. Similarly, the power set of the sub-alert groups determined by the comparison subjects is expressed as the probability, for each sub-alert group, based on how many of the multiple operation monitors being compared decided to handle it themselves and how many decided not to handle it themselves, as shown in formula (5) below.

The evaluation function can be expressed, for example, as in the following formula (6), based on the sum of the squared value of the difference between a set consisting of {0, 1} indicating whether or not the operation monitor, who is the person being evaluated, will handle the sub-alert groups included in the power set of the sub-alert groups, and a numerical value based on the ratio (probability) of the number of people who decided to handle each sub-alert group themselves and the number of people who decided not to handle it themselves, as indicated by multiple comparison subjects for each sub-alert group.

As expressed in equation (6), _E2 calculated by equation (6) corresponds to the chi-squared value for which sub-alert groups the operator has decided to handle. Therefore, as expressed in equation (6), the smaller the deviation between the classification results of the operator being evaluated and the comparison operator for each sub-alert group, the smaller the difference between the operator's classification results and those of the comparison operator. Therefore, the smaller the deviation and the smaller the value of equation (6), the higher the evaluation of the operator's classification results. For example, for sub-alert groups that many of the comparison operators have decided to handle themselves, the more sub-alert groups the operator being evaluated has decided to handle themselves, the smaller the value of equation (6).

Referring to Figures 7A and 7B, we will explain in more detail an example of using a chi-squared distribution as an evaluation function. Figures 7A and 7B show an example of a judgment on the power set of sub-alert groups indicated by multiple processors. As shown on the left side of Figure 7A, for example, for five sub-alert groups g1 to g5, multiple comparison subjects judge that they will handle sub-alert groups g1, g2, g3, g4, and g5 with probabilities of 0.6, 0.2, 0.1, 0.05, and 0.05, respectively. When the assessee judges that they will handle sub-alert group g1 themselves, the chi-squared value is calculated as 0.666666667. In the same case, when the assessee judges that they will handle sub-alert group g2 themselves, as shown on the right side of Figure 7A, the chi-squared value is calculated as 4.0. In the example shown in Figure 7A, the chi-squared value is relatively large when the assessee determines that they will handle sub-alert group g2 themselves, compared to when they determine that they will handle sub-alert group g1, which more than half of the comparison subjects determined they would handle themselves. Similarly, as shown in Figure 7B, for example, for five sub-alert groups g11 to g15, if a comparison subject, such as an expert, determines that they will handle sub-alert groups g11, g12, g13, g14, and g15 with probabilities of 0.5, 0.3, 0.1, 0.05, and 0.05, respectively, when the assessee determines that they will handle sub-alert group g11 themselves, the chi-squared value is calculated as 1.0. In the same case, when the assessee determines that they will handle sub-alert group g12 themselves, as shown on the right side of Figure 7B, the chi-squared value is calculated as 2.3333333333. Thus, in the example shown in Figure 7B, when the assessee decides to handle sub-alert group g11, which only approximately half of the subjects judged to handle themselves, the chi-squared value is not larger when the assessee decides to handle g12, compared to the case shown in Figure 7A. As can be seen from the examples shown in Figures 7A and 7B, when the assessee decides to handle a sub-alert group that only a portion of the subjects selected, the chi-squared value varies depending on whether there is a bias in the judgments of the subjects and whether many of the subjects judged to handle the same sub-alert group themselves, or whether there is variation in the judgments of the subjects. In this way, the chi-squared value makes it possible to evaluate judgment errors.

Note that the evaluation of alert classification is not limited to the examples described above or with reference to Figures 7A and 7B. For example, a similar chi-squared distribution may be calculated not only for the evaluator's judgment of whether to handle multiple sub-alert groups, but also for the evaluator's judgment of whether to handle multiple alerts. For example, for m alerts a1, a2, ..., am, the probability of which alerts multiple comparison subjects decided to handle themselves may be expressed, and a chi-squared distribution may be calculated for the evaluator's judgment of which alerts they decided to handle themselves. This method also makes it possible to evaluate the degree of certainty of the alerts selected when there is variation in the judgments of multiple comparison subjects. Furthermore, in some embodiments, the evaluation may be conducted not only for sub-alert groups that the evaluator decided to handle themselves, but also for sub-alert groups that another operation monitor decided to handle, for example. For example, for j sub-alert groups g1, g2, ..., gj, it may be evaluated which of the t+1 total operation monitors, consisting of the person being evaluated and t other operation monitors, decided to handle which sub-alert group. In this case, too, it is possible to evaluate using the magnitude of the chi-squared value. Similarly, it is also possible to evaluate the decisions of other operation monitors regarding multiple alerts that they decided to handle. For example, for m alerts a1, a2, ..., am, it may be evaluated which of the u+1 total operation monitors, consisting of the person being evaluated and u other operation monitors, decided to handle which alert. In this case, it may be evaluated using the magnitude of the chi-squared value.

In this way, the chi-square value _E2 calculated by Equation (6) is equivalent to evaluating the likelihood of the alert judgment and classification of the operation monitor being evaluated. In other words, if the person being evaluated does not decide to handle a sub-alert group that many of the people being compared have decided to handle themselves, the value _E2 of Equation (6) will be large, and the person being evaluated will receive a low rating. On the other hand, if there is variation in the sub-alert groups that multiple people being compared have decided to handle themselves, even if the person being evaluated handles a sub-alert group that a relatively small number of people being compared have decided to handle themselves, the value _E2 of Equation (6) will be relatively small, and the decline in the person being evaluated will be relatively small. In this way, the likelihood of the judgment of the person responsible for handling the alerts can be evaluated.

Next, an example of evaluating the results of the operation judgments made by the operation monitor, who is the person being evaluated in an embodiment of the present disclosure (the operation judgment evaluation step in this embodiment) will be described. The operation judgments may be evaluated, for example, using a statistical distribution calculated based on the operation judgments made by the person being compared. For example, if the operation judgments made by the person being compared include data representing the probability of which of multiple operation judgment candidates each was selected by the multiple people being compared, the evaluation may be performed using an evaluation function calculated based on the value obtained by integrating, for each alert, the squared deviation between the operation judgment made by the operation monitor, who is the person being evaluated, and the operation judgment made by the person being compared. The evaluation function may also include a function based on a chi-squared distribution using the value obtained by integrating, for each alert, the squared deviation between the operation judgment made by the person being evaluated and the operation judgment made by the person being compared.

The evaluation function for evaluating operation decisions may be calculated, for example, as follows: For example, assuming that the group of alerts that have occurred is A, the group of situation judgment materials that have occurred is S, and the group of possible operation decisions is D, the combination ODM of the alerts a _k (k=1, 2, ..., N) included in the alert group A and the operation decisions d _k for the alerts a _k is expressed as in the following formula (7).

An alert group A, which is a set of assigned alerts a _k , and an operation decision group D, which is a set of operation decisions d _k , are expressed by the following formulas (8) and (9), respectively.

Furthermore, in the i-th decision-making, whether or not the operation monitor makes an operation decision d in response to the alert a is represented as 1 or 0, as in equation (10).

Furthermore, the probability that the person being compared will make operational decision d for alert a after determining which alert he or she will be responsible for in response to the alert that has occurred and having already made several operational decisions is expressed as in equation (11).

From the above, the evaluation function for operation judgment is expressed as the following formula (12).

Here, E _3i is expressed as in the following formula (13).

As illustrated here, operation decisions can also be evaluated using the chi-squared distribution. That is, in this example, the distribution of operation decisions made by the operation monitor being evaluated is evaluated against the distribution of operation decisions made by multiple comparison subjects. Just as described above for classifying multiple alerts into multiple sub-alert groups and determining the likelihood of alerts that the operation monitor has decided to handle, the chi-squared value can also be used to evaluate the likelihood of operation decisions made by the operation monitor being evaluated. For example, if multiple comparison subjects make multiple different operation decisions, i.e., there is variation in the operation decisions made by the multiple comparison subjects, it can be evaluated which operation decision the operation monitor being evaluated made against the distribution of operation decisions made by the multiple comparison subjects.

In the above-described example of operational decision evaluation, the combination of an alert and the operational decision made in response to the alert can be evaluated, for example, as shown in Equation (13). Therefore, the evaluation process exemplified above can evaluate the likelihood of an operational decision based on the operational decision made by the comparison target, taking into account which alert the operational decision was made in response to. In the evaluation of operational decisions in the embodiments of the present disclosure, the order in which operational decisions are made (the order in which alerts are processed) can also be evaluated. For example, the order in which operational decisions are made can be evaluated using a statistical distribution such as the chi-square distribution described above, relative to the order in which the comparison target made the operational decisions, in terms of the order in which the evaluation target made the operational decisions (the distribution evaluation step in this embodiment). Furthermore, the combination of alerts and operational decisions made by the comparison target and the order in which the evaluation target made the operational decisions can be evaluated based on the distribution of each alert. For example, the order in which the evaluation target processed the combination of alerts and operational decisions can be evaluated using the distribution of the order in which each combination was processed by a comparison target, such as an expert.

When the chi-squared value is calculated for the distribution of combinations of alerts and operational decisions, for example, if the chi-squared value is high, it can be determined that the likelihood is lower than the judgment made by the person being compared, and that there is a high possibility that the person being evaluated made an incorrect judgment. In this case, by analyzing the squared value of the deviation between the judgment made by the person being compared for each combination of alert and operational decision and the judgment made by the operation monitor being evaluated, it is also possible to estimate which alerts were judged incorrectly.

As described above, the evaluation method in the embodiments of the present disclosure can use an evaluation process that utilizes, for example, probability or statistical distribution. By using probability or statistical distribution in this way to perform an evaluation, it is possible to perform a quick and objective evaluation, thereby efficiently evaluating the monitoring accuracy of the operation monitor being evaluated.

In addition, if the accuracy rate for situational judgment is low, it is inferred that the situation was misjudged. If the situation was misjudged, even if the evaluation value of the subsequent judgment in operation monitoring, such as the classification of multiple alerts into sub-alert groups or the results of operation judgment, is high, it is inferred that this is likely a coincidence. Therefore, if the evaluation value of the situation judgment is low, it may be determined that more training is required. Furthermore, as described above, if the evaluation value is low in the evaluation of the plausibility of operation judgment, it is inferred that the operation judgment was not made in the appropriate permutation of the alerts. In such a case, for example, by sequentially examining the chi-squared value for each combination of alert and operation judgment, if the chi-squared value is significantly high for any alert and/or operation judgment, it can be confirmed that an incorrect judgment was made in that alert and/or operation judgment. For example, subsequent training content can be planned taking into account the type of alert and/or operation judgment.

In the above-described embodiment, an example was given in which the operation monitoring performed by the operation monitor being evaluated is evaluated using the results of operation monitoring previously performed by a comparison target such as an expert. However, the method for evaluating operation monitors in the embodiments of the present disclosure may also be used to evaluate operation monitors monitoring autonomously-operated mobile vehicles in operation, for example. For example, an expert or other comparison target may perform operation monitoring based on alerts issued by autonomously-operated mobile vehicles currently in operation, or information on situational assessment materials related to the conditions inside and outside the vehicle when the alert is issued, while the evaluation target operation monitor performs operation monitoring in parallel, and the evaluation target operation monitor's operation monitoring judgment results may be compared with the evaluation target's operation monitoring judgment results to evaluate the evaluation target's operation monitoring.

That is, for example, the step of presenting the above-mentioned alerts and information for situational assessment to the person being evaluated may include a step of presenting a new alert that occurs in an automatically operable vehicle and newly occurring information for situational assessment that is related to the new alert. Furthermore, the person being compared may be evaluated for their new classification results, situational assessments, and/or operational decisions in response to the new alerts and newly occurring information for situational assessment, with the person being evaluated for their new classification results, situational assessments, and/or operational decisions based on the new alerts and newly occurring information for situational assessment.

In the above-described embodiment, examples have been given of mainly evaluating situational judgment, the classification of alerts into sub-alert groups, and operational judgment, but the evaluation targets do not have to be limited to these. For example, the selection of situational judgment materials included in the situational judgment material information considered in each judgment during operation monitoring may be evaluated. For example, data on situational judgment materials considered by the person being compared may be used to evaluate the situational judgment materials considered by the person being evaluated. By evaluating and analyzing the selection of situational judgment materials, it becomes possible to use them as judgment factors when estimating the cause, for example, if the person being evaluated made an error in judgment.

A method for evaluating operation monitoring according to an embodiment of the present disclosure will be described with reference to Figure 8. Figure 8 is an example flowchart of the evaluation process for operation monitoring executed in an embodiment of the present disclosure.

First, the assessee is presented with an alert and situation assessment information associated with the alert (S802). As described above, in this embodiment, the alert may be issued, for example, by the alert issuing device 200 and displayed by the alert output unit 362 of the operation monitoring device 300. In addition, the situation assessment information may be displayed by the situation assessment information output unit 364 of the operation monitoring device 300.

Next, after S802, input regarding the situation judgment and operation judgment by the person being evaluated is accepted (S804). In this embodiment, the situation judgment and operation judgment may be accepted by input via the situation judgment input unit 352 and operation judgment input unit 356 of the operation monitoring device 300.

An exemplary operation monitoring evaluation method according to an embodiment of the present disclosure will be described with reference to Figure 9. Figure 9 shows a flowchart of an exemplary operation monitoring evaluation process performed in an embodiment of the present disclosure.

First, the person being evaluated is presented with an alert and situation assessment information associated with the alert (S902).

Next, the input of the situation assessment by the person being assessed is accepted (S904). The situation assessment by the person being assessed may be accepted, for example, by input via the situation assessment input unit 352 of the operation monitoring device 300.

Next, the system accepts input of the results of categorizing the multiple alerts into sub-alert groups by the assessee (S906). The categorization results may be accepted, for example, by input via the categorization result input unit 354 of the operation monitoring device 300.

Next, the operator's operation judgment is accepted (S908). The operation judgment may be accepted, for example, by input via the operation judgment input unit 356 of the operation monitoring device 300.

Next, the situation judgment, classification results, and/or operation judgment made by the person being evaluated are evaluated (S910). The evaluation of the situation judgment, classification results, and/or operation judgment made by the person being evaluated may be performed by comparing them with the situation judgment, classification results, and/or operation judgment made in operation monitoring performed by a comparable person, such as an expert. The evaluation of the situation judgment, classification results, and/or operation judgment made by the person being evaluated may be performed by, for example, the situation judgment evaluation unit 142, classification result evaluation unit 144, and/or operation judgment evaluation unit 146 of the operation monitoring evaluation unit 140 of the operation monitoring evaluation device 100.

Next, the evaluation results are output (S912). The evaluation results may be output, for example, by the evaluation result output unit 366 of the operation monitoring device 300. The evaluation results of the situation judgment, classification results, and/or operation judgment by the person being evaluated may be displayed, for example, by the situation judgment evaluation result output unit 366a, classification result evaluation result output unit 366b, and/or operation judgment evaluation result output unit 366c of the operation monitoring device 300, respectively.

<Evaluation data creation process>
Next, a process for creating evaluation data including judgment results provided by a comparison subject, such as an expert, used in the evaluation method for operation monitoring according to an embodiment of the present disclosure will be described with reference to Figures 10 and 11. Figure 10 is a diagram showing an example of a system 2 for creating evaluation data for judgment results provided by a comparison subject, according to an embodiment of the present disclosure. Also, Figure 11 is a diagram showing a functional block configuration of the evaluation data creation system 2 according to an embodiment of the present disclosure.

As shown in FIG. 10, the evaluation data creation system 2 includes an evaluation data creation device 400, multiple operation monitoring devices 500-1, 500-2, ..., 500-n (n is a natural number), and multiple autonomously operable vehicles 600-1, 600-2, ..., 600-l (l is a natural number). The evaluation data creation device 400, operation monitoring devices 500-1 to 500-n, and vehicles 600-1 to 600-l are connected by wire or wirelessly via a network 20, such as the Internet or a telephone network. Hereinafter, any operation monitoring device among the operation monitoring devices 500-1 to 500-n will also be referred to as operation monitoring device 500. Additionally, any mobile device among the mobile devices 600-1 to 600-l will also be referred to as mobile device 600. The evaluation data creation system 2 further includes a database DB2.

The evaluation data creation device 400 may be, for example, an evaluation data creation server, and may collect data from the operation monitoring devices 500-1 to 500-n, in which an operation monitor, such as an expert, who is to be compared, remotely monitors the operation of mobile units 600-1 to 600-l and enters their judgments. The collected data may be stored, for example, in database DB2. The evaluation data creation device may be configured, for example, from one or more physical servers, or may be configured using a virtual server running on a hypervisor, or may be configured using a cloud server. The evaluation data creation device 400 may have a hardware configuration similar to that of the operation monitoring evaluation device 100 of the operation monitoring evaluation system 1 in the embodiment of the present disclosure described above.

Operation monitoring devices 500-1 to 500-n are used, for example, by experienced operation monitors to monitor mobile objects 600-1 to 600-1, such as autonomous vehicles. Each operation monitoring device 500 may monitor, for example, multiple mobile objects 600. For example, operation monitoring device 500-1 may be configured to monitor six mobile objects 600-1 to 600-6, and operation monitoring device 500-2 may be configured to monitor the next six mobile objects 600-7 to 600-12. The number of mobile objects monitored is not limited to this; for example, different operation monitors may monitor different numbers of mobile objects, or multiple operation monitors may monitor the same mobile object or multiple mobile objects in duplicate. As mentioned above, for example, in a level 4 autonomous driving service, it is expected that one monitor will monitor multiple autonomous vehicles. Similarly, when creating evaluation data for operation monitoring according to an embodiment of the present disclosure, each operation monitor may monitor multiple mobile objects. The traffic monitoring device 500 receives alerts from the mobile units 600 and accepts input from the traffic monitor regarding situation assessment and operational decisions when an alert is issued. The traffic monitor makes situation assessment and operational decisions based on situation assessment information transmitted by the mobile units 600 when an alert is issued. As shown in FIG. 10 , alerts from the mobile units 600 and situation assessment information when an alert is issued may be transmitted to each traffic monitoring device 500 via the Internet. Note that while FIG. 10 illustrates an example of an evaluation data creation system 2 equipped with multiple traffic monitoring devices, the number of traffic monitoring devices 500 may be, for example, one. In other words, the evaluation data may be data monitored by a single traffic monitor. The traffic monitoring device 500 may have a hardware configuration similar to that of the traffic monitoring device 300 of the traffic monitoring evaluation system 1 in the above-described embodiment of the present disclosure.

Mobile object 600 is, for example, a mobile object capable of autonomous driving. Mobile object 600 is, for example, a mobile object such as a bus that can be driven autonomously within a limited area, provided that the area is one in which the above-mentioned Level 4 autonomous driving service is provided. When an alert occurs, mobile object 600 transmits the alert to operation monitoring device 500. Mobile object 600 also transmits to operation monitoring device 500, for example, images of the interior and exterior of the vehicle acquired by an in-vehicle camera and an exterior camera, and audio data from inside and outside the vehicle acquired by sound collection devices installed inside and outside the vehicle. The data transmitted to operation monitoring device 500 may be used, for example, as the above-mentioned situation assessment information.

As described above, database DB2 may store, for example, information regarding alerts issued when an operation monitor monitors operation, situation assessment information regarding the status of the autonomously operable vehicle that was the subject of monitoring at the time the alert was issued, and information regarding the situation assessment and operation assessment made by the operation monitor.

Next, the functional block configuration of the evaluation data creation system 2 will be described with reference to Figure 11. As shown in Figure 11, the evaluation data creation device 400 in an embodiment of the present disclosure includes a control unit 402, an acquisition unit 404, a communication processing unit 406, and an evaluation data creation unit 440. The evaluation data creation unit 440 includes a situation judgment data creation unit 442, a classification result data creation unit 444, an operation judgment data creation unit 446, an alert data creation unit 448, and a situation judgment material data creation unit 449. The evaluation data creation device 400 uses the situation judgment data creation unit 442, classification result data creation unit 444, and operation judgment data creation unit 446 to create situation judgment data, classification result data, and operation judgment data to be evaluated when evaluating the operation monitor of the person being evaluated, based on the situation judgment, classification results, and operation judgment results entered into the operation monitoring device 500 by an experienced operation monitor, which are acquired by the acquisition unit 404 via the communication processing unit 406 from the operation monitoring device 500. The evaluation data creation device 400 also acquires from the operation monitoring device 500 the alerts and situation judgment information confirmed by the operation monitor to be compared when making the situation judgment, alert classification, and operation judgment, and creates alert data and situation judgment information by the alert data creation unit 448 and situation judgment information creation unit 449, respectively. The created alert data and situation judgment information are used by the person being evaluated when performing operation monitoring, for example, when the person being evaluated undergoes operation monitoring training. The evaluation data, including the created situation assessment data, operation assessment data, classification result data, alert data, and situation assessment material data, may be stored in database DB2.

As shown in FIG. 11 , the operation monitoring device 500 includes a control unit 502, an acquisition unit 504, a communication processing unit 506, a memory unit 520, an input unit 550, and an output unit 560. The operation monitoring device 500 acquires, for example, alert information issued by the mobile object 600 via the acquisition unit 504 via the communication processing unit 506, and displays the information in the alert output unit 562 of the output unit 560. Furthermore, data such as video and audio at the time the alert was issued is acquired from the mobile object 600 as situation assessment information, and displays the information in the situation assessment information output unit 564. The input unit 550 includes, for example, a situation assessment input unit 552, a classification result input unit 554, and an operation judgment input unit 556. Based on the displayed alert and situation assessment information, the operation monitor inputs a situation assessment, classification result, and/or operation judgment using the situation assessment input unit 552, classification result input unit 554, and/or operation judgment input unit 556. The memory unit 520 stores, for example, a collected data DB 522. The collected data DB 522 may store data such as situation judgments, classification results, and operation judgments provided by the operation monitor who is being evaluated.

As shown in FIG. 11 , the autonomously driven vehicle 600 includes a control unit 602, a communication processing unit 606, an alert issuing unit 630, a situation determination information transmitting unit 650, and a situation acquisition unit 660. When a predetermined event occurs, such as when a passenger is confirmed to have fallen inside the vehicle, the alert issuing unit issues an alert. For example, when an alert is issued, the situation determination information acquired by the situation acquisition unit 660 is transmitted to the operation monitoring device 500 by the situation determination information transmitting unit 650. The situation acquisition unit 660 includes, for example, an imaging unit 662, an audio sensor 664, a temperature sensor 666, and a humidity sensor 668. The imaging unit 662 may include, for example, a camera installed inside the vehicle and a camera installed outside the vehicle, and may be configured to capture images of the situation inside and outside the vehicle. The situation assessment information transmission unit 650 may transmit video and audio data acquired by the situation acquisition unit 660 to the operation monitoring device 500 as situation assessment information.

Figure 11 also shows database DB2. As shown in Figure 11, database DB2 stores operation monitoring data for two operation monitors, T1 and T2. Database DB2 stores information regarding, for example, alert group A, which includes alerts a1-a4 issued by the automatically operable mobile vehicle that was the target of monitoring when operation monitors T1 and T2, who are comparison subjects, respectively performed operation monitoring; situation determination material group S, which includes situation determination materials s1-s5 related to the environment inside and outside the vehicle at the time the alert was issued; situation determination group C, which includes situation determinations c1-c3 indicated by the comparison subjects; and operation judgment group D, which includes any of operation judgments d1-d4 indicated by the comparison subjects. Database DB2 may also include information regarding which of sub-alert groups g1 and g2 alerts a1-a4 included in alert group A were classified. As shown in FIG. 11 , for example, the alert group A stored for operation monitor T1 has sub-alert group g1 including alerts a1 and a3 and sub-alert group g2 including alerts a2 and a4, while the alert group A stored for operation monitor T2 has sub-alert group g1 including alerts a1 and a4 and sub-alert group g2 including alerts a2 and a3. Furthermore, the operation judgment group D stored for operation monitor T1 includes, for example, operation judgments d1, d2, and d3, while the operation judgment group D stored for operation monitor T2 includes, for example, operation judgments d1, d2, and d4. Database DB2 may further store statistical distribution data for data related to situation judgments, classification results, and/or operation judgments entered by multiple operation monitors.

Next, the evaluation data creation process executed in the evaluation data creation system 2 will be described with reference to Figure 12. Figure 12 shows an example flowchart of an exemplary evaluation data creation process executed in an embodiment of the present disclosure.

First, the operation monitoring device 500 receives the alert issued by the vehicle 600 and information on the situation when the alert was issued (step S1202).

Next, the operation monitoring device 500 accepts input of a situation assessment from the operation monitor (step S1204).

Next, the operation monitoring device 500 accepts input from the operation monitor of the results of classifying the alert into sub-alert groups (step S1206).

Next, the operation monitoring device 500 accepts input of an operation judgment from the operation monitor (step S1208).

The operation monitoring device 500 then stores the situation assessment, classification results, and operation judgment (step S1210). For example, the situation assessment, classification results, and operation judgment may be transmitted to the evaluation data creation device 400, and stored in the evaluation database DB2 by the evaluation data creation device 400.

(Modification)
In the above-described embodiment of the present disclosure, the operation monitoring evaluation method has been described with reference to an example in which the operation monitoring evaluation system 1 executes the evaluation method for operation monitoring. However, the operation monitoring evaluation method according to the embodiment of the present disclosure may also be executed solely by the operation monitoring evaluation device. FIG. 13 shows an example of the functional block configuration of the operation monitoring evaluation device 700 in this case. As shown in FIG. 13 , the operation monitoring evaluation device 700 includes not only the components of the operation monitoring device 300 of the above-described operation monitoring evaluation system 1, but also a operation monitoring evaluation unit 740 having the same configuration as the operation monitoring evaluation unit 140 of the operation monitoring evaluation device 100. Therefore, the operation monitoring evaluation device 700 can evaluate the situation judgment, alert classification result, and operation judgment input by the person being evaluated based on the alert issued by the alert output unit 762 and the situation judgment material information displayed by the situation judgment material information output unit 764 using the situation judgment result evaluation unit 742, classification result evaluation unit 744, and operation judgment evaluation unit 746 of the operation monitoring evaluation unit 740 in the operation monitoring evaluation device 700, respectively. Information on operation monitoring previously performed by a comparison target, such as an expert, used in the evaluation may be stored in a database DB3 as shown in FIG. 13 , or may be stored in a storage unit 720 of the operation monitoring evaluation device 700. Similar to the operation monitoring device 300 of the operation monitoring evaluation system 1, the operation monitoring evaluation device 700 may also include a control unit 702, an acquisition unit 704, a communication processing unit 706, a storage unit 720, an input unit 750, and an output unit 760. The storage unit 720 may include a collected data DB 722, the input unit 750 may include a situation judgment input unit 752, a classification result input unit 754, and an operation judgment input unit 756, and the output unit 760 may include an alert output unit 762, a situation judgment information output unit 764, and an evaluation result output unit 766. Furthermore, the evaluation result output unit 766 of the output unit 760 may include a situation judgment evaluation output unit 766a, a classification result evaluation output unit 766b, and an operation judgment evaluation output unit 766c.

In the embodiment of the present disclosure described above with reference to FIG. 1, an example was described in which a single alert transmission device transmits an alert in response to information transmitted from the operation monitoring evaluation device. However, the configuration of the alert transmission device is not limited to this. For example, an alert transmission device may be provided for each operation monitoring device. FIG. 14 shows an example of an operation monitoring evaluation system 12 in this case. As shown in FIG. 14, alert transmission devices 200-1, 200-2, ..., 200-n may be installed for each operation monitoring device 300-1, 300-2, ..., 300-n. Also, with reference to FIG. 14, in the operation monitoring evaluation system 12, an example was described in which the alert transmission device 200 is installed within the same network as the operation monitoring devices 300-1, 300-2, ..., 300-n. However, the alert transmission device 200 does not have to be installed within the same network as the operation monitoring devices 300-1, 300-2, ..., 300-n. FIG. 15 shows another example of an operation monitoring evaluation system 14. For example, as shown in FIG. 15, in the operation monitoring evaluation system 14, the alert sending device 200 is installed in the same network as the operation monitoring evaluation device 100, such as a server, and sends alerts to operation monitoring devices 300-1, 300-2, ..., 300-n via the Internet, for example, to have the operation monitor, who is being evaluated, perform operation monitoring.

Note that the configuration of the alert issuing device is not limited to these. For example, an alert issuing unit may be provided within the operation monitoring device 300, and the operation monitoring device 300 may issue an alert based on information related to the alert sent from the operation monitoring evaluation device 100.

The operation monitoring evaluation method according to the embodiments of the present disclosure described above enables efficient evaluation of operation monitors being trained, such as during training of operation monitors for autonomously operated mobile vehicles. It also enables more objective evaluation of operation monitors. Therefore, it is possible to efficiently evaluate and implement training, especially when training a large number of operation monitors. Therefore, it is possible to train operation monitors who can respond even when a large number of autonomous vehicles are introduced in a short period of time, such as when Level 4 autonomous driving services are deployed. Furthermore, the technology according to each embodiment can contribute to achieving Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization," and Goal 11 of the Sustainable Development Goals (SDGs), "Make cities and human settlements inclusive, safe, resilient and sustainable."

The above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the scope of the present invention. The flowcharts, sequences, elements included in the embodiments, as well as their arrangements, materials, conditions, shapes, sizes, etc., are not limited to those illustrated and can be modified as appropriate. Furthermore, configurations shown in different embodiments can be partially substituted or combined.

(Appendix 1: Operation monitoring and evaluation device)
An operation monitoring evaluation device for evaluating operation monitoring of an automatically operable mobile body,
a memory for storing an alert issued in response to a predetermined event occurring in the automatically operable mobile body, situation determination information associated with the alert, and a classification result of the alert, a situation determination, and/or an operation determination corresponding to the situation determination, indicated by a person to be compared based on the alert and the situation determination information;
an alert sending unit that sends the alert;
a display unit that displays the situation determination information related to the alert;
An input unit,
First information on a situation judgment made by the assessee based on the alert and the situation judgment material information;
Second information about the driving judgment of the assessee corresponding to the situation judgment;
Third information on a classification result in which the assessee classifies the alert into a plurality of sub-alert groups based on the alert and the situation judgment material information;
an input unit for receiving
an evaluation unit that evaluates the operation monitoring of the person being evaluated by comparing the classification result, the situation judgment, and/or the operation judgment indicated by the person to be compared with the first information, the second information, and/or the third information;
An operation monitoring and evaluation device including:

(Appendix 2: Program for preparing evaluation data)
A program for creating evaluation data for evaluating operation monitoring of an automatically operable mobile body, the program comprising:
From the automatically operable mobile body,
an alert that is sent in response to the occurrence of a predetermined event in the autonomously operable mobile object;
situation determination information, which is information regarding the state of the automatically operable mobile body at the time of issuing the alert;
and
From a monitor who monitors the autonomously operable mobile body,
First information about a situation judgment made by the monitor based on the alert and the situation judgment material information;
Second information on the result of the monitor's operational judgment corresponding to the situation judgment;
third information on a classification result in which the monitor classifies the alert into a plurality of sub-alert groups based on the alert and the situation determination material information;
and
A program that executes.

(Appendix 3: Program for evaluating operation monitoring)
A program for evaluating operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event, the program comprising:
A presentation step of presenting the alert and situational assessment information associated with the alert to the assessee;
After the presentation step, a reception step of receiving input regarding the situation judgment and operation judgment by the person being evaluated;
A program that executes.

(Note 4: Alert priority)
the plurality of sub-alert groups include a first sub-alert group and a second sub-alert group;
The classification result by the assessee includes a judgment result that the alerts included in the first sub-alert group are to be processed with priority over the alerts included in the second sub-alert group.
3. The program according to claim 3.

(Appendix 5: Evaluation method for operational decisions)
The operation judgment by the assessee includes information about the combination of the alert and the operation judgment and the order in which the operation judgment is executed,
The evaluating step includes:
a distribution evaluation step of evaluating the distribution of the combination of the alert and the operation decision indicated by the person to be compared and the order in which the operation decision is executed for each alert based on the combination and the order included in the operation decision made by the person to be evaluated;
3. The program according to claim 3.

(Appendix 6: Real-time operation monitoring)
The alert presented to the assessee includes a new alert occurring in the autonomously operable mobile object,
The situation determination information presented to the assessee includes newly occurring situation determination information related to the new alert,
In the evaluation step,
The operation monitoring of the person being evaluated is evaluated by comparing the new situation judgment, classification result, and/or operation judgment made by the person being compared in response to the new alert and the newly occurring situation judgment information with the new situation judgment, classification result, and/or operation judgment made by the person being evaluated based on the new alert and the newly occurring situation judgment information.
3. The program according to claim 3.

1...Travel monitoring and evaluation system, 2...Evaluation data creation system, 10a...Processor, 10b...Memory, 10c...Communication unit, 10d...Input unit, 10e...Output unit, 20...Network, 100, 700...Travel monitoring and evaluation device, 200...Alert sending device, 300, 500...Travel monitoring device, 400...Evaluation data creation device, 600...Mobile object, DB, DB2, DB3...Database

Claims (9)

A program for evaluating operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event, the program comprising:
A presentation step of presenting the alert and situational assessment information associated with the alert to the assessee;
After the presentation step, a reception step of receiving input regarding the situation judgment and operation judgment by the person being evaluated;
Execute
The receiving step includes:
The classification results of the assessee, in which the multiple alerts are classified into multiple sub-alert groups
a classification result receiving step of receiving an input regarding the classification result;
the plurality of sub-alert groups include a first sub-alert group and a second sub-alert group;
The classification result by the person being evaluated includes a judgment result that the alerts included in the first sub-alert group will be handled by the person being evaluated, and the alerts included in the second sub-alert group will be handled by another operation monitor . an evaluation step of evaluating the operation monitoring of the person being evaluated by comparing the situation judgment, the classification result, and/or the operation judgment indicated by the person being compared in response to the alert and the situation judgment information with the situation judgment, the classification result, and/or the operation judgment indicated by the person being evaluated;
The program according to claim 1 , further causing the computer to execute the following: The classification result by the evaluator includes a judgment result that two or more of the alerts are to be processed together.
The program according to claim 1 . The evaluating step includes:
a calculation step of calculating a statistical distribution and/or probability calculated based on the situation judgment, the classification result, and/or the operation judgment indicated by the person to be compared;
The program according to claim 2 . The evaluating step includes:
a situation judgment evaluation step of evaluating the situation judgment by the person being evaluated based on a probability corresponding to a correct answer rate for the situation judgment indicated by the person being compared,
The probability corresponding to the correct answer rate is based on a value obtained by accumulating the deviation between the situation judgment by the person being evaluated and the situation judgment by the person being compared.
The program according to claim 4 . the classification result indicated by the person to be compared includes data expressing a probability for each of the plurality of sub-alert groups regarding which of the sub-alert groups the person to be compared has determined to handle;
The evaluating step includes:
a classification result evaluation step of evaluating the classification result by the person to be evaluated using an evaluation function calculated based on a value obtained by accumulating the squared values of the deviations between the classification result by the person to be evaluated and the classification result indicated by the person to be compared;
The evaluation function includes a function based on a chi-square distribution using a value obtained by accumulating the squared deviation between the classification result by the person being evaluated and the classification result indicated by the person being compared for the sub-alert group,
The program according to claim 4 . The operation decisions indicated by the persons to be compared include data representing probabilities for each of a plurality of operation decision candidates regarding which of the plurality of operation decision candidates each of the persons to be compared selected, and
The evaluating step includes:
an operation judgment evaluation step for evaluating the operation judgment of the person being evaluated using an evaluation function calculated based on a value obtained by accumulating the squared value of the deviation between the operation judgment of the person being evaluated and the operation judgment indicated by the person being compared, for the alert;
The evaluation function includes a function based on a chi-square distribution using a value obtained by accumulating the squared value of the deviation between the operation judgment by the person being evaluated and the operation judgment indicated by the person being compared for the alert,
The program according to claim 4 . A system for evaluating operation monitoring of an autonomously operable mobile object configured to issue an alert in response to the occurrence of a predetermined event,
An operation monitoring and evaluation device;
An operation monitoring device;
an alert sending device;
an evaluation database;
The evaluation database stores the alerts presented to the person being evaluated, situational judgment information associated with the alerts, situational judgments made by the person being compared based on the alerts and the situational judgment information, classification results of the alerts, and/or operational judgments corresponding to the situational judgments;
the alert transmission device transmits the alert to the operation monitoring device,
In response to the issuance of the alert, the operation monitoring device
A presentation step of presenting the situation judgment material information related to the alert to the assessee;
After the presentation step, a reception step is performed to receive inputs regarding the situation judgment by the person being evaluated, a classification result in which the plurality of alerts are classified into a plurality of sub-alert groups, and the operation judgment.
is configured to be executable,
The operation monitoring and evaluation device
causing the alert issuing device to issue the alert based on the alert stored in the evaluation database;
providing the operation monitoring device with the situation determination information stored in the evaluation database and related to the alert transmitted from the alert transmitting device;
an evaluation step of evaluating the operation monitoring of the person being evaluated by comparing the situation judgment, the classification result, and/or the operation judgment indicated by the person being compared with the situation judgment, the classification result, and/or the operation judgment indicated by the person being evaluated;
configured to be able to execute
system. 1. A method for evaluating operation monitoring of an autonomously operable vehicle configured to issue an alert in response to the occurrence of a predetermined event, comprising:
A presentation step of presenting the alert and situational assessment information associated with the alert to the assessee;
After the presentation step, a reception step of receiving input regarding the situation judgment and operation judgment by the person being evaluated;
Including ,
The receiving step includes:
The classification results of the assessee, in which the multiple alerts are classified into multiple sub-alert groups
a classification result receiving step of receiving an input regarding the classification result;
the plurality of sub-alert groups include a first sub-alert group and a second sub-alert group;
A method in which the classification result by the person being evaluated includes a judgment result that the alerts included in the first sub-alert group will be handled by the person being evaluated, and the alerts included in the second sub-alert group will be handled by another operation monitor .