JP7803897B2 - Self-service fueling monitoring system - Google Patents

Description

The present invention relates to a system for monitoring self-service refueling at gas stations.

At conventional gas stations, a supervisor (employee) inside the station building would monitor the behavior of customers (fuel dispensers) filling up their tanks visually or via a surveillance monitor, and then operate a control device to issue permission to fill up. In recent years, development has progressed on systems that use AI (Artificial Intelligence) to monitor surveillance camera footage, and these systems are beginning to be applied to monitoring self-service gas dispensers.

Here, the following are examples of prior art in the technical field related to this invention. For example, Patent Document 1 discloses an invention in which a monitoring device analyzes camera footage to determine abnormal behavior by a user, and controls vehicle refueling via user operation.

International Publication No. 2021/015256

Development of systems that analyze camera footage based on a predetermined learning model and determine whether or not to grant permission to refuel is progressing, but there is a problem in that the accuracy of the judgment deteriorates due to changes in the surrounding environment and over time. For this reason, to maintain judgment accuracy, it is necessary to manually verify the success or failure of the judgment, collect camera footage where the judgment fails as learning data, and periodically re-run machine learning to update the learning model. However, verifying the success or failure of the judgment at each gas station and selecting and collecting footage where the judgment fails as learning data places a heavy burden on the person performing this task.

The present invention was made in consideration of the above-mentioned conventional circumstances, and aims to enable the efficient collection of learning data for retraining the learning model used to monitor self-service refueling.

In order to achieve the above object, the present invention is configured as follows.
In other words, a self-service refueling monitoring system according to one embodiment of the present invention is characterized by comprising a monitoring device that uses a learning model that has learned the specific behavior of the refueler to detect specific behavior from video data and make a judgment as to whether or not to allow refueling, a terminal device that accepts operations to allow refueling, and a learning device that tags the video data as normal behavior if the judgment result by the monitoring device as to whether or not to allow refueling does not match the operation content on the terminal device.

Here, the terminal device can be configured to accept an operation to specify an area to be cut out from the video data, and the learning device can be configured to tag the area cut out from the video data as normal behavior.

In another aspect of the present invention, a self-service refueling monitoring system includes a monitoring device that detects predetermined behavior from video data using a learning model that has learned the predetermined behavior of refuelers; a terminal device that accepts an operation to specify the type of abnormal behavior; and a learning device that, if the detection results from the monitoring device do not match the abnormal behavior corresponding to the operation content accepted by the terminal device, tags the video data with the abnormal behavior corresponding to the operation content.

Here, the terminal device can be configured to accept an operation to specify an area to be cut out from the video data, and the learning device can be configured to tag the area cut out from the video data with abnormal behavior.

The present invention makes it possible to efficiently collect learning data for retraining the learning model used to monitor self-service refueling.

1 is a diagram showing an example of the configuration of a self-service refueling permission system included in a self-service refueling monitoring system according to an embodiment of the present invention. 1 is a diagram showing an example of the configuration of a determination accuracy maintenance system included in a self-service refueling monitoring system according to one embodiment of the present invention. 2 is a diagram showing an example of a display on a terminal device of the self-service refueling permission system of FIG. 1. FIG. 10 is a diagram illustrating an example of a processing flow for collecting learning data according to the first embodiment. FIG. 11 is a diagram illustrating an example of a processing flow for collecting learning data according to the second embodiment. FIG. 11 is a diagram illustrating an example of a processing flow for collecting learning data according to the third embodiment. FIG. 11 is a diagram illustrating an example of a processing flow for collecting learning data according to the third embodiment. FIG. 13 is a diagram illustrating an example of a processing flow for collecting learning data according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of a processing flow for collecting learning data according to the fifth embodiment. FIG. 3 is a diagram showing an example of a processing flow by the determination accuracy maintenance system of FIG. 2 .

A self-service refueling monitoring system according to one embodiment of the present invention will be described below with reference to the drawings. The self-service refueling monitoring system according to one embodiment of the present invention has two subsystems: a self-service refueling permission system 100 configured as shown in FIG. 1, and a determination accuracy maintenance system 200 configured as shown in FIG. 2.

The self-service refueling permission system 100 is a system that determines whether or not to permit self-service refueling based on camera footage, and is installed at each gas station. The self-service refueling permission system 100 includes a monitoring device 110, a terminal device 120, a control device 130, a monitoring camera 140, a sensor 150, and a server 160.

The surveillance camera 140 captures images of the fuel dispenser (user) filling up the self-service tank and outputs the captured image data (camera footage) to the monitoring device 110. The surveillance camera 140 only needs to be installed so that it can capture images of the fuel dispenser's actions when filling up the self-service tank; for example, it is installed in a position that overlooks the area including the area between the meter and the vehicle from above.

Sensor 150 detects sound, heat, odor, light, oil leaks, and other signals generated by the actions of the person filling up the tank at the self-service facility, and outputs the detection results to monitoring device 110. Examples of sensors 150 include sensors that detect when the fuel nozzle is removed from the fuel dispenser. Sensor 150 is installed at or near the fuel dispenser (for example, in a parking area where the vehicle to be filled up is parked).

The monitoring device 110 is installed in a location separate from the metering machine (for example, inside the gas station building). The monitoring device 110 has an image analysis device 111, an interface 112, and a wireless device 113. The image analysis device 111 analyzes camera footage using an analysis program that uses a pre-set learning model, detects certain actions that the fuel dispenser may take, and determines whether or not to permit self-service refueling (permit/deny). The wireless device 113 communicates wirelessly with the terminal device 120. The interface 112 is interposed between the monitoring camera 140, sensor 150, control device 130, image analysis device 111, and wireless device 113. The monitoring device 110 is, for example, a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs related to each function of the present invention.

The terminal device 120 is a device that accepts operations from a monitor (employee) regarding whether or not to permit self-refueling during self-refueling. The terminal device 120 has a control unit 121, a communication unit 122, a display unit 123, and an operation unit 124. The control unit 121 comprehensively controls the operation of each unit of the terminal device 120. The communication unit 122 communicates wirelessly with the monitoring device 110. The display unit 123 displays various information including the determination results by the monitoring device 110. The operation unit 124 accepts various operations including operations regarding whether or not to permit self-refueling. The display unit 123 and the operation unit 124 may be integrated into a touch panel. The terminal device 120 is, for example, a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs related to each function of the present invention.

The terminal device 120 receives and displays the detection results of specified actions and the determination results of whether refueling is permitted from the monitoring device 110. The monitor refers to the detection results of specified actions and the determination results of whether refueling is permitted displayed on the terminal device 120, decides whether to permit or prohibit (disallow) refueling, and inputs an operation regarding whether to permit self-refueling. Here, for example, when an operation to permit refueling is input, a refueling permission signal is sent from the terminal device 120 to the metering machine, which then activates the pump inside the metering machine and makes refueling possible. Here, we will explain an example in which the terminal device 120 is a portable terminal such as a tablet that can be carried by the monitor (employee), but it may also be a stationary terminal installed (fixed) in a predetermined location.

Figure 3 shows an example of a monitoring screen displayed on the terminal device 120. The monitoring screen 300 in Figure 3 has a lane status display section 310, a live video display section 320, a detection history display section 330, a detected video display section 340, abnormal behavior confirmation buttons 351-354, and a refueling permission button 360.

The lane status display unit 310 displays the status of each fueling lane at the gas station (waiting to be fueled, being fueled, etc.), the type of fuel being fueled, the amount of fuel being fueled, and other order details. The live video display unit 320 displays live (real-time) camera footage of the active fueling lane the fueler is visiting (if there are multiple active fueling lanes, a fueling lane selected from among them). The detection history display unit 330 displays the history of the fueler's actions detected from camera footage of the active fueling lane. The detection video display unit 340 displays the camera footage at the time the fueler's actions were detected (i.e., the camera footage capturing the fueler's actions).

The abnormal behavior confirmation buttons 351-354 are operated by the monitor when they confirm, through camera footage or visual observation, that a fuel dispenser is engaging in abnormal behavior that should not be engaged in, and are provided for each type of abnormal behavior. In this example, the abnormal behavior detection buttons include a cigarette possession confirmation button 351 that is operated when it is confirmed that the fuel dispenser is carrying cigarettes; a portable can confirmation button 352 that is operated when it is confirmed that a portable can is being carried; a risky behavior confirmation button 353 that is operated when it is confirmed that the fuel dispenser is engaging in dangerous behavior (e.g., filling up with fuel together with multiple people); and a nozzle half-insertion confirmation button 354 that is operated when it is confirmed that the fuel dispenser is not fully inserted into the fuel filler opening. In this example, a partially inserted fuel filler nozzle indicates that the fuel filler nozzle is not inserted properly, and indicates behavior by the fuel dispenser that should be determined to be abnormal. The fuel fill permission button 360 is operated when the monitor confirms, through camera footage or visual observation, that self-service fueling should be permitted.

The control device 130 is installed, for example, near the weighing machine, and controls the operation of the weighing machine based on control signals transmitted from the monitoring device 110 in response to operation of the terminal device 120. For example, when the terminal device 120 receives an operation from the monitor to permit refueling, it controls the weighing machine so that refueling is carried out in response to operation by the refueler. Also, when the terminal device 120 receives an operation from the monitor to prohibit (not permit) refueling, it controls the weighing machine to prohibit (suspend) refueling. Note that the control signal in response to operation of the terminal device 120 is not limited to being transmitted from the monitoring device 110 to the control device 130, but may also be transmitted from the terminal device 120 to the control device 130.

Like the monitoring device 110, the server 160 is installed in a location separate from the weighing machine (for example, inside a gas station building or on the cloud). The server 160 has an image database 161, a learning model management unit 162, and a communication unit 163. The image database 161 stores learning data for retraining a learning model that detects predetermined behaviors from camera footage. The learning data stored in the image database 161 is transmitted to the judgment accuracy maintenance system 200. The learning model management unit 162 updates the analysis program used in the image analysis device 111 of the monitoring device 110 based on the learning model distributed from the judgment accuracy maintenance system 200. The communication unit 163 communicates with the judgment accuracy maintenance system 200. The server 160 is realized, for example, by a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs for realizing each function related to the present invention.

The determination accuracy maintenance system 200 is used to maintain the determination accuracy of the self-service refueling permission system 100, and is installed at a base (e.g., a management center) connected to each gas station via a network line. The determination accuracy maintenance system 200 may be installed inside the gas station building rather than at a remote base. The determination accuracy maintenance system 200 is equipped with a learning device 210 that retrains the learning model used in the self-service refueling permission system 100 at each gas station.

The learning device 210 has a communication unit 211, an image database 212, a machine learning unit 213, and a control unit 214. The communication unit 211 communicates with the self-service refueling permission system 100 of each gas station. The image database 212 stores learning data collected from the self-service refueling permission system 100 of each gas station. The machine learning unit 213 performs machine learning (relearning) based on the learning data stored in the image database 212, and regenerates a learning model to be used in the self-service refueling permission system 100 of each gas station. The control unit 214 controls the re-learning by the machine learning unit 213. The learning model generated by the machine learning unit 213 is distributed to the self-service refueling permission system 100 of each gas station. The learning device 210 is realized, for example, by a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs to realize each function of the present invention.

The learning model may be a general-purpose learning model that is commonly used by multiple gas stations, or a dedicated learning model that is used only at a specific gas station. A general-purpose learning model can be generated based on learning data collected from the self-service refueling permission systems 100 of the multiple gas stations that use it. A dedicated learning model can be generated based on learning data collected from the self-service refueling permission system 100 of the specific gas station that uses it. These learning models may be provided separately for each type of behavior to be detected, or one learning model may be configured to be able to detect multiple types of behavior.

The main features of the self-service refueling monitoring system in this example are that it includes a monitoring device 110 that uses a learning model that has learned predetermined actions that a refueler may perform when self-service refueling to detect predetermined actions from camera footage of the self-service refueling process; a terminal device 120 that accepts operations from the monitor regarding whether or not to allow self-service refueling during self-service refueling; and a learning device 210 that has an image database 212 that stores camera footage as learning data for re-training the learning model for predetermined actions when a predetermined action is determined to have been missed or falsely detected by comparing the detection results of the predetermined action by the monitoring device 110 with the operations performed on the terminal device 120.

In other words, the self-service refueling monitoring system of this example is configured to automatically recognize when a predetermined behavior has been missed or mistakenly detected by comparing the detection results of the predetermined behavior by the monitoring device 110 with the operation details on the terminal device 120, and to store camera footage of when a predetermined behavior has been missed or mistakenly detected in the image database 212 as learning data for re-learning. In this way, camera footage of when a predetermined behavior has been missed or mistakenly detected can be collected as learning data for re-learning, based on the operations performed by the monitor as part of their normal duties (operations related to whether or not to allow self-service refueling). As a result, the learning model used to monitor self-service refueling can be efficiently re-trained. Furthermore, because the judgment of the monitor at each gas station is reflected in the collection of learning data, even if a learning model is prepared for each gas station, it becomes easy to re-train a learning model appropriate for that gas station.

The operation of collecting learning data will be described below with reference to several examples.
[First Example of Learning Data Collection]
4 shows an example of a processing flow for collecting learning data according to the first embodiment. A user (fuel dispenser) intending to perform self-service refueling gets into a vehicle and arrives at a gas station, parks the vehicle in a parking area near a meter, gets out of the vehicle, and begins refueling. At this time, a surveillance camera 140 captures the actions of the fuel dispenser during self-service refueling and transmits the camera footage to a monitoring device 110. The monitoring device 110 analyzes the camera footage using an analysis program based on a preset learning model and detects certain actions that the fuel dispenser may perform.

For example, the monitoring device 110 performs an analysis of camera footage using a learning model to detect whether a fuel dispenser is carrying cigarettes (step S101). It also performs image analysis using a learning model to detect whether a portable fuel can has been brought in (step S102). It also performs image analysis using a learning model to detect risky behavior by fuel dispensers (e.g., filling up with multiple people) (step S103). It also performs image analysis using a learning model to detect the insertion state of the fuel nozzle into the fuel filler opening (fully inserted/partially inserted) (step S104).

Next, the monitoring device 110 determines whether or not to permit refueling (permit/deny) based on the results of the above detection process (steps S101 to S104) (step S105). For example, if possession of cigarettes, carrying a portable can, or risky behavior is not detected and the fully inserted fuel nozzle is detected, it is determined that refueling may be permitted. Alternatively, if possession of cigarettes, carrying a portable can, risky behavior, or the partially inserted fuel nozzle is detected, or if the fully inserted fuel nozzle is not detected, it is determined that refueling is not permitted. Data on the results of the above detection process and the determination process are transmitted to the terminal device 120.

The terminal device 120 displays the data received from the monitoring device 110 and accepts operations from the monitor (employee) regarding whether or not to allow refueling. For example, if the monitor decides to allow refueling, the monitor operates the refueling permission button 360 on the monitoring screen 300 in Figure 3. On the other hand, if the monitor confirms any abnormal behavior and decides not to allow refueling, the monitor operates one of the abnormal behavior confirmation buttons 351-354 on the monitoring screen 300 in Figure 3.

Next, the monitoring device 110 compares the result of its own refueling permission determination with the result of the refueling permission determination made by the monitor (operation related to refueling permission), and determines whether they match (step S106). If the result of the refueling permission determination made by the monitoring device 110 matches the result of the refueling permission determination made by the monitor (step S106: Yes), the monitoring device 110 transmits comparison result data recording that the detection was successful to the server 160 (step S107).

On the other hand, if the refueling permission determination result made by the monitoring device 110 does not match the refueling permission determination result made by the monitor (step S106: No), the monitoring device 110 identifies the image analysis results of the incorrect determination and classifies them as "missed detection" or "false detection" (step S108), and transmits the comparison result data recording the detection failure to the server 160 along with the video of the incorrect determination and the classification result (step S109).

The server 160 stores the data received from the monitoring device 110 in the image database 161 and transmits it to the learning device 210 of the judgment accuracy maintenance system 200 as needed or periodically. The learning device 210 stores the data received from the server 160 in the image database 212 and performs re-learning of the learning model when certain conditions are met.

In this way, in the first embodiment, the detection results of the specified behavior are compared with the operation details on the terminal device 120 to automatically recognize when a specified behavior has been missed or mistakenly detected, and the camera footage is identified when a specified behavior has been judged to have been missed or mistakenly detected. This makes it possible to efficiently collect camera footage when a specified behavior has been missed or mistakenly detected, and to re-learn it. Note that in the above explanation, behavior detection through video analysis is performed as a step prior to granting permission for refueling, but behavior detection may also be performed after refueling is permitted, and refueling may be stopped if abnormal behavior such as possession of cigarettes, carrying a portable can, or dangerous behavior is detected.

[Second Example of Learning Data Collection]
5 shows an example of a processing flow for collecting learning data according to the second embodiment. In the second embodiment, the explanation focuses on a learning model for detecting the insertion state (fully inserted/partially inserted) of a fuel nozzle relative to a fuel filler neck. The monitoring device 110 waits until the sensor 150 detects that the fuel filler neck has been removed from the metering machine (step S201). Upon detecting that the fuel filler neck has been removed from the metering machine, the monitoring device 110 analyzes the camera image based on the learning model and determines the insertion state of the fuel filler neck relative to the fuel filler neck (step S202).

If a certain amount of time has passed since detecting that the fuel filler nozzle has been removed, and the fuel filler nozzle is not detected as being fully or partially inserted into the fuel filler neck (Step S202: Not Detected), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120, and has the monitor confirm the insertion status of the fuel filler nozzle (Step S203). If the terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been fully inserted (e.g., pressing the fuel fill permission button 360) (Step S204: OK), the monitoring device 110 determines that full nozzle insertion was not detected, and stores the camera image at that time as positive learning data for full nozzle insertion (Step S205). On the other hand, if the terminal device 120 receives an operation indicating that the fuel nozzle has been partially inserted (for example, pressing the nozzle partially inserted confirmation button 354) (step S204: NG), the monitoring device 110 determines that the nozzle has not been partially inserted and stores the camera image at that time as positive learning data for fully inserted nozzle (step S205).

Furthermore, if the monitoring device 110 detects that the fuel filler nozzle is fully inserted into the fuel filler neck within a certain period of time after detecting that the fuel filler nozzle has been removed (Step S202: Nozzle Fully Inserted), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120, and prompts the monitor to confirm the insertion status of the fuel filler nozzle (Step S207). If the terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been fully inserted (e.g., pressing the Refueling Permission button 360) (Step S207: OK), the monitoring device 110 determines that the full insertion of the nozzle was accurately detected. On the other hand, if the terminal device 120 receives an operation indicating that the fuel filler nozzle has been partially inserted (e.g., pressing the Nozzle Partial Insertion Confirmation button 354) (Step S204: NG (Partial Insertion)), the monitoring device 110 determines that this is a false detection of full nozzle insertion and a missed detection of partial nozzle insertion, and stores the camera image at that time as positive learning data for partial nozzle insertion (Step S205). Furthermore, if the terminal device 120 receives an operation indicating that a different state (for example, the fuel nozzle is not inserted) has been confirmed (step S207: NG (different)), the monitoring device 110 determines that this is a false detection of the nozzle being fully inserted or partially inserted, and stores the camera images from that time as negative learning data for the nozzle being fully inserted and negative learning data for the nozzle being partially inserted (step S208).

Furthermore, if the monitoring device 110 detects that the fuel filler nozzle is partially inserted into the fuel filler neck within a certain period of time after detecting that the fuel filler nozzle has been removed (Step S202: Nozzle partially inserted), the monitoring device 110 notifies the monitor by displaying a message on the terminal device 120, prompting the monitor to confirm the insertion status of the fuel filler nozzle (Step S209). If the terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been partially inserted (e.g., pressing the Nozzle Partial Insertion Confirmation button 354) (Step S209: OK), the monitoring device 110 determines that the partial nozzle insertion was correctly detected. On the other hand, if the terminal device 120 receives an operation indicating that the fuel filler nozzle has been fully inserted (e.g., pressing the Refueling Permission button 360) (Step S209: NG (Fully Inserted)), the monitoring device 110 determines that this is a false detection of the nozzle being partially inserted and a missed detection of the nozzle being fully inserted, and stores the camera image at that time as positive learning data for fully inserted nozzle (Step S210). Furthermore, if the terminal device 120 receives an operation indicating that a different state (for example, the fuel nozzle is not inserted) has been confirmed (step S209: NG (different)), the monitoring device 110 determines that this is a false detection of the nozzle being fully inserted or partially inserted, and stores the camera images from that time as negative learning data for the nozzle being fully inserted and negative learning data for the nozzle being partially inserted (step S210).

The positive learning data and negative learning data for the insertion state of the fuel filler nozzle collected by the monitoring device 110 as described above are stored in the server 160 and are sent as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the positive learning data and negative learning data for the insertion state of the fuel filler nozzle received from the server 160 in the image database 212, and performs re-learning of the learning model for detecting the insertion state of the fuel filler nozzle when predetermined conditions are met.

In this way, the second embodiment is configured to distinguish between camera footage in the event of a missed detection and camera footage in the event of a false detection and collect that footage separately. That is, camera footage in the event of a missed detection of a predetermined action (in this example, the fuel nozzle being fully inserted or partially inserted) is stored as positive learning data showing what the predetermined action looks like when it is correct. Furthermore, camera footage in the event of a false detection of a predetermined action is stored as negative learning data showing what the predetermined action looks like when it is incorrect. In this way, by storing camera footage in the event of a missed detection and camera footage in the event of a false detection separately, it becomes possible to retrain a learning model that can detect predetermined actions with higher accuracy.

[Third Example of Learning Data Collection]
6A and 6B show example processing flows for collecting learning data according to the third embodiment. The example processing flow of FIG. 6A relates to the collection of learning data for a learning model for detecting abnormal behavior that should not be performed by a fuel pump operator. The example processing flow of FIG. 6B relates to the collection of learning data for a learning model for detecting normal behavior that should be performed by a fuel pump operator. Examples of abnormal behavior include carrying cigarettes, carrying a portable can, risky behavior, and partially inserting the fuel nozzle. Examples of normal behavior include fully inserting the fuel nozzle.

First, with reference to FIG. 6A, learning data collection for a learning model for detecting abnormal behavior will be described.
The monitoring device 110 analyzes the camera footage based on the learning model to determine whether or not abnormal behavior is present (step S301). If no abnormal behavior is detected (step S301: No), the monitoring device 110 prompts the monitor to take an action regarding whether or not to permit refueling. If the terminal device 120 subsequently accepts an operation to permit refueling (e.g., pressing the refueling permission button 360) (step S302: OK), the monitoring device 110 determines that the failure to detect abnormal behavior was appropriate. On the other hand, if the terminal device 120 does not accept the operation to permit refueling but instead accepts another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S302: NG), the monitoring device 110 determines that abnormal behavior was missed and stores the camera footage at that time as positive learning data for abnormal behavior (step S303).

Furthermore, if abnormal behavior is detected by image analysis based on the learning model (Step S301: Yes), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take action regarding whether or not to allow refueling (Step S304). If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (Step S305: OK), the monitoring device 110 determines that abnormal behavior was detected incorrectly and stores the camera footage at that time as negative learning data for abnormal behavior (Step S306). On the other hand, if the terminal device 120 does not accept an action to allow refueling but instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S305: NG), the monitoring device 110 determines that the detection of abnormal behavior was appropriate.

Next, with reference to FIG. 6B, learning data collection for a learning model for detecting normal behavior will be described.
The monitoring device 110 analyzes the camera footage based on the learning model to determine whether normal behavior is present (step S401). If normal behavior is detected (step S401: Yes), the monitoring device 110 prompts the monitor to take an action regarding whether to permit refueling. If the terminal device 120 subsequently accepts an operation to permit refueling (e.g., pressing the refueling permission button 360) (step S402: OK), the monitoring device 110 determines that the detection of normal behavior was appropriate. On the other hand, if the terminal device 120 does not accept the operation to permit refueling but instead accepts another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S402: NG), the monitoring device 110 determines that normal behavior is a false positive and stores the camera footage at that time as negative learning data for normal behavior (step S403).

Furthermore, if normal behavior is not detected by image analysis based on the learning model (step S401: No), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take action regarding whether or not to allow refueling (step S404). If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (step S405: OK), the monitoring device 110 determines that normal behavior was not detected and stores the camera image at that time as positive learning data for normal behavior (step S406). On the other hand, if the terminal device 120 does not accept an action to allow refueling but instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S405: NG), the monitoring device 110 determines that not detecting normal behavior was appropriate.

The learning data for normal and abnormal behavior collected by the monitoring device 110 as described above is stored in the server 160 and is sent to the learning device 210 of the judgment accuracy maintenance system 200 as needed or periodically. The learning device 210 stores the learning data for normal and abnormal behavior received from the server 160 in the image database 212, and re-trains the learning model for detecting normal behavior and the learning model for detecting abnormal behavior when certain conditions are met.

In this way, in the third embodiment, learning data for normal behavior and learning data for abnormal behavior are collected according to different standards. This makes it possible to retrain a learning model that detects normal behavior with higher accuracy and a learning model that detects abnormal behavior with higher accuracy.

[Fourth Example of Learning Data Collection]
7 shows an example of a process flow for collecting learning data according to the fourth embodiment. The process flow example in FIG. 7 relates to the collection of learning data for a learning model that has learned multiple types of abnormal behavior that should not be performed by fuel pump drivers. The multiple types of abnormal behavior include, for example, carrying cigarettes, carrying a can, risky behavior, and partially inserting the fuel nozzle.

The monitoring device 110 analyzes the camera footage based on the learning model and determines whether or not multiple types of abnormal behavior are present (Step S501). If none of the multiple types of abnormal behavior are detected (Step S501: No), the monitoring device 110 prompts the monitor to take an action regarding whether or not to allow refueling. If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (Step S502: OK), the monitoring device 110 determines that not detecting abnormal behavior was appropriate. On the other hand, if the monitor does not allow refueling because he or she has observed one of the multiple types of abnormal behavior (Step S502: NG), the terminal device 120 accepts an action to specify the type of abnormal behavior that was observed (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S503). At this time, the monitoring device 110 determines that the type of abnormal behavior specified by the monitor has been missed, and stores the camera footage from that time as positive learning data for that type of abnormal behavior (step S504).

Furthermore, if one of multiple types of abnormal behavior is detected by image analysis based on the learning model (step S501: Yes), the type of abnormal behavior detected is displayed on the terminal device 120 to notify the monitor and prompt the monitor to take action regarding whether or not to allow refueling (step S505). If the terminal device 120 subsequently receives an action to allow refueling (e.g., pressing the refueling permission button 360) (step S506: OK), the monitoring device 110 determines that the type of abnormal behavior detected was a false positive and stores the camera image at that time as negative learning data for that type of abnormal behavior (step S507).

On the other hand, if the monitor does not allow refueling because he or she has observed one of several types of abnormal behavior (Step S506: NG), the terminal device 120 accepts an operation to specify the type of abnormal behavior observed (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S508). As a result, if the type of abnormal behavior detected does not match the type of abnormal behavior specified (Step S509: No), the monitoring device 110 determines that there was a detection miss for the specified type of abnormal behavior and stores the camera footage at that time as positive learning data for that type of abnormal behavior (Step S510). On the other hand, if the type of abnormal behavior detected matches the type of abnormal behavior specified (Step S509: Yes), the monitoring device 110 determines that the type of abnormal behavior detected was appropriate.

The learning data for each of the multiple types of abnormal behavior collected by the monitoring device 110 as described above is stored in the server 160 and is sent to the learning device 210 of the judgment accuracy maintenance system 200 as needed or periodically. The learning device 210 stores the learning data for each of the multiple types of abnormal behavior received from the server 160 in the image database 212, and re-trains the learning model for detecting each of the multiple types of abnormal behavior when certain conditions are met.

In this way, the fourth embodiment is configured to collect learning data by distinguishing between multiple types of abnormal behavior. This makes it possible to retrain a learning model that detects each of the multiple types of abnormal behavior with higher accuracy.

In the fourth embodiment, one learning model is configured to detect multiple abnormal behaviors, but multiple learning models may be configured to detect different abnormal behaviors. In this case, in S504, the abnormal behavior specified by the monitor in S503 is stored as positive learning data for the learning model. In S507, the abnormal behavior detected in S505 is stored as negative learning data for the learning model. In S510, since the abnormal behavior detected in S505 was a false positive, the abnormal behavior detected in S505 is stored as negative learning data for the learning model. Furthermore, since the abnormal behavior specified by the monitor in S508 was not detected, the abnormal behavior specified by the monitor in S508 is also missed, so the abnormal behavior is stored as positive learning data for the learning model.

[Fifth Example of Learning Data Collection]
Fig. 8 shows an example of a process flow for collecting learning data according to the fifth embodiment. The process flow example in Fig. 8 relates to the collection of learning data for a learning model that has learned normal behavior that a fuel dispenser should perform and abnormal behavior that a fuel dispenser should not perform. For example, such a learning model may detect a fully inserted nozzle as a normal behavior and a partially inserted nozzle as an abnormal behavior.

The monitoring device 110 analyzes the camera footage based on the learning model to determine whether normal or abnormal behavior is present (Step S601). If neither normal nor abnormal behavior is detected (Step S601: Not Detected), the monitoring device 110 prompts the monitor to take an action regarding whether to permit refueling (Step S602). If the terminal device 120 subsequently accepts an action to permit refueling (e.g., pressing the refueling permission button 360) but instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S603: NG), the monitoring device 110 determines that abnormal behavior was not detected and stores the camera footage at that time as positive image data for abnormal behavior (Step S604). On the other hand, if the terminal device 120 accepts an action to permit refueling (Step S603: NG), the monitoring device 110 determines that normal behavior was not detected and stores the camera footage at that time as positive image data for normal behavior (Step S605).

Furthermore, if abnormal behavior is detected by image analysis based on the learning model (Step S601: Abnormal Behavior), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take action regarding whether or not to allow refueling (Step S606). If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (Step S607: OK), the monitoring device 110 determines that this is a missed detection of normal behavior and a false positive detection of abnormal behavior, and stores the camera image at that time as positive image data for normal behavior (Step S608). On the other hand, if the terminal device 120 does not accept an action to allow refueling but instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S607: NG), the monitoring device 110 determines that the detection of abnormal behavior was appropriate.

Furthermore, if normal behavior is detected by image analysis based on the learning model (Step S601: Normal Behavior), the monitoring device 110 prompts the monitor to take an action regarding whether or not to allow refueling. If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (Step S609: OK), the monitoring device 110 determines that the detection of normal behavior was appropriate. On the other hand, if the terminal device 120 does not accept an action to allow refueling but instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (Step S609: NG), the monitoring device 110 determines that this is a false positive detection of normal behavior and a missed detection of abnormal behavior, and stores the camera footage from that time as positive image data for abnormal behavior (Step S610).

The learning data for each of the normal and abnormal behaviors collected by the monitoring device 110 as described above is stored in the server 160 and is sent to the learning device 210 of the judgment accuracy maintenance system 200 as needed or periodically. The learning device 210 stores the learning data for each of the normal and abnormal behaviors received from the server 160 in the image database 212, and re-trains the learning model for detecting each of the normal and abnormal behaviors when certain conditions are met.

In this way, in the fifth embodiment, learning data for normal behavior and learning data for abnormal behavior are collected according to different standards. This makes it possible to retrain a learning model that detects normal behavior and abnormal behavior with higher accuracy.

Figure 9 shows an example of a processing flow by the learning device 210 of the judgment accuracy maintenance system 200 in Figure 2. The example processing flow in Figure 9 focuses on the execution of re-learning of the learning model, and is executed after collecting learning data using the method shown in Figures 4 to 8. The following describes an example of processing subsequent to the processing flow shown in Figure 4 (first embodiment).

The learning device 210 stores the comparison result data (detection success/detection failure), erroneous judgment images, and classification results (detection failure/false positive detection) received from each gas station server 160 in the image database 212 (step S701). The control unit 214 of the learning device 210 aggregates the data stored in the image database 212 and determines whether the conditions for starting re-learning have been met. In this example, the conditions for starting re-learning are: [Condition 1] The detection failure rate (= number of detection failures/total number of comparison results) is equal to or greater than a first threshold (step S702); [Condition 2] The false detection rate (= number of false detections/total number of comparison results) is equal to or greater than a second threshold (step S703); [Condition 3] An operation instructing the start of re-learning has been received from the administrator (step S704); and [Condition 4] A predetermined amount of time has passed since the previous learning (step S705). If any of [Condition 1] to [Condition 4] is satisfied, the control unit 214 causes the machine learning unit 213 to start relearning of the learning model (step S706). On the other hand, if none of [Condition 1] to [Condition 4] is satisfied, relearning of the learning model is not started.

The machine learning unit 213 performs machine learning (relearning) based on the learning data accumulated in the image database 212, and regenerates a learning model to be used in each gas station's self-service refueling permission system 100. The learning data is created by an operator visually checking and tagging the video data stored as positive learning data and the video data stored as negative learning data. Alternatively, an observer may generate learning data by pressing the abnormal behavior confirmation buttons 351-354 or the refueling permission button 360 in Figure 3 to tag the corresponding abnormal behavior or normal behavior. This reduces the time required for annotation. The learning model generated by the machine learning unit 213 is distributed to each gas station's self-service refueling permission system 100 via the communication unit 211. The server 160 of each gas station's self-service refueling permission system 100 provides the learning model received from the learning device 210 of the judgment accuracy maintenance system 200 to the learning model management unit 162. The learning model management unit 162 updates the analysis program of the image analysis device 111 of the monitoring device 110 based on the distributed learning model. By updating the learning model used to monitor self-service refueling in this way, it is possible to prevent the accuracy of judgment from deteriorating due to changes in the surrounding environment or over time.

In the explanation so far, the monitoring device 110 performs a process to identify camera footage when a behavior is determined to have been missed or falsely detected by comparing the detection results of a specified behavior with the operation content on the terminal device 120. However, this process may also be performed by another device (e.g., the terminal device 120, the server 160, etc.), and the entity performing the process does not matter. However, if the above process is performed by a device located separately from the internal network of each gas station (e.g., a device on the determination accuracy maintenance system 200 side), there is a concern that the network load will increase, so it is preferable that the above process be performed by a device located on the internal network of each gas station.

The learning model may be common to all gas stations, or a dedicated learning model may be prepared for each gas station. For example, a common learning model may be used for gas stations with a standard layout and camera placement, while a dedicated learning model may be used for gas stations with a unique layout and camera placement.

The image data to be tagged may also be specified by the observer via the monitoring screen of Figure 3. For example, the observer specifies an area to be cut out from the live video 320 and presses one of the abnormal behavior confirmation buttons 351-354 or the refueling permission button 360. This observer's operation causes the image data of the cut-out area to be tagged with the abnormal behavior or normal behavior corresponding to the pressed button.

The above describes embodiments of the present invention, but these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take on a variety of other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and their modifications are included within the scope and spirit of the invention described in this specification, etc., and are also included in the invention described in the claims and their equivalents.

In addition, the present invention can be provided not only as the devices described above or as systems composed of these devices, but also as methods executed by these devices, programs for implementing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable format.

The present invention can be used in a system that monitors self-service refueling at gas stations.

100: Self-service refueling permission system, 110: Monitoring device, 111: Image analysis device, 112: Interface, 113: Wireless device, 120: Terminal device, 121: Control unit, 122: Communication unit, 123: Display unit, 124: Operation unit, 130: Control device, 140: Monitoring camera, 150: Sensor, 160: Server, 161: Image database, 162: Learning model management unit, 163: Communication unit, 200: Judgment accuracy maintenance system, 210: Learning device, 211: Communication unit, 212: Image database, 213: Machine learning unit, 214: Control unit, 300: Monitoring screen, 310: Lane status display unit, 320: Live video display unit, 330: Detection history display unit, 340: Detection video display unit, 351 to 354: Abnormal behavior confirmation button, 360: Refueling permission button

Claims (2)

a monitoring device that uses a learning model that has learned about the action of inserting a fuel nozzle into a fuel filler opening as a predetermined action of a fuel dispenser, detects the predetermined action from video data after it has been detected that the fuel nozzle has been removed from the metering machine , and determines whether or not to permit fuel dispense;
a terminal device that accepts a refueling permission operation that permits refueling;
a control device that sets the metering machine to a state where fuel can be supplied in response to the fuel supply permission operation received by the terminal device;
A self-service refueling monitoring system characterized by comprising a learning device that, if the judgment result by the monitoring device regarding whether refueling is permitted does not match the operation content on the terminal device, judges that the full insertion of the refueling nozzle into the refueling opening has been missed, and tags normal behavior with the video data and stores it as learning data for re-learning the learning model of the specified behavior. the terminal device accepts an operation to designate an area to be cut out from the video data;
The self-service refueling monitoring system according to claim 1 , wherein the learning device tags the region extracted from the video data as normal behavior.