JP7789474B2 - Self-service fuel management system - Google Patents

Description

The present invention relates to a self-service refueling management system, a control method for a self-service refueling management system, and a self-service refueling station.

At conventional self-service gas stations, staff in the control room of the gas station building would visually or via a monitoring monitor monitor the customer's refueling behavior, safety, and risky behavior, and issue permission to refuel using a control device (see, for example, Patent Documents 1 and 2).

Japanese Patent Application Laid-Open No. 2004-269018 Japanese Patent Application Publication No. 10-264995

At the conventional self-service gas stations mentioned above, staff in the control room would visually or via a monitor monitor monitor customers' refueling behavior, safety, and risky behavior, and issue permission to refuel using a control device. However, if it is difficult for staff to check visually or via a monitor, they would have to go to the site to check, which is time-consuming.

In such cases, it is conceivable to use a monitoring device (AI robot) to determine whether or not the user is behaving abnormally from captured images, but in cases where there is little so-called training data or the user accidentally behaves irregularly, the monitoring device alone may not be able to make an accurate determination even if the behavior is not abnormal.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a self-service refueling management system, a control method for a self-service refueling management system, and a self-service refueling station that can easily determine whether a user is behaving abnormally by having at least a monitoring device (AI robot) determine whether a user is behaving abnormally from captured images, and by having store staff review the information (images, etc.) that served as the basis for that determination, thereby reducing the effort required of store staff to conduct such verification.

A self-service fueling system according to one aspect of the present invention comprises:
means for capturing an image of a user refueling;
A means for determining whether the behavior of the user is abnormal, normal, or unknown based on the captured image;
A means for stopping or prohibiting refueling when it is determined that the user's behavior in each refueling process is abnormal or unknown;
a management terminal that displays an image that serves as the basis for the determination;
The present invention is characterized by comprising:

In the self-service refueling management system,
The management terminal may have a means for displaying, in addition to the image on which the determination is based, a live image of the area surrounding the image on which the determination is based.

In the self-service refueling management system,
The control terminal can be operated to stop or prohibit fuel supply.

In the self-service refueling management system,
When the refueling process is stopped or prohibited, a means for canceling the stop or prohibition by operating the management terminal may be provided.

In the self-service refueling management system,
The system may also have a means for determining whether the user's actions are abnormal, normal, or unknown for each refueling procedure, and displaying a history of the determined user's actions on the management terminal.

In the self-service refueling management system,
The judgment means may use training data from an image database created by machine learning to make a judgment as to whether the image is abnormal, normal, or unknown, and may update the image database by machine learning using the images judged to be abnormal or normal once the judgment is complete.

A gas station according to one aspect of the present invention includes:
means for capturing an image of a user refueling;
A means for determining whether the behavior of the user is abnormal, normal, or unknown based on the captured image;
A means for stopping or prohibiting refueling when it is determined that the user's behavior in each refueling process is abnormal or unknown;
a management terminal that displays an image of the basis for the determination;
The present invention is characterized by comprising:

With the self-service refueling management system of the present invention, the monitoring device determines abnormal behavior by the user, and store staff can easily check the results of the determination to determine whether or not there is an abnormality, thereby reducing the amount of work required for store staff to check inside the gas station.

FIG. 1 is a diagram showing an example of the configuration of a self-service refueling management system 100 in a self-service refueling station according to this embodiment. FIG. 2 is a diagram showing an example of the configuration of the display unit H of the management terminal (mobile terminal) X of the self-service refueling management system 100 shown in FIG. FIG. 3 is a conceptual diagram showing an example of the overall flow of a control method for self-service refueling in the self-service refueling management system 100 shown in FIG. FIG. 4 is a conceptual diagram showing another example of the overall flow of the control method for self-service refueling in the self-service refueling management system 100 shown in FIG. FIG. 5A is a flow diagram showing an example of a specific flow at the time of refueling in the self-service refueling management system 100 shown in FIG. 1 . FIG. 5B is a flowchart following FIG. 5A, showing an example of a specific flow at the time of refueling in the self-service refueling management system 100 shown in FIG. FIG. 5C is a flowchart following FIG. 5B, showing an example of a specific flow at the time of refueling in the self-service refueling management system 100 shown in FIG.

[Self-service fueling management system]
Here, Fig. 1 is a diagram showing an example of the configuration of a self-service refueling management system 100 in a self-service refueling station according to this embodiment. Also, Fig. 2 is a diagram showing an example of the configuration of a display unit H of a management terminal (mobile terminal) X of the self-service refueling management system 100 shown in Fig. 1.

In this invention, self-service refueling refers to refueling a vehicle performed by a customer themselves through their own operations. For example, the customer stops the vehicle in a parking area, selects the type and amount of fuel to be refueled, removes the oil gun (nozzle) from the dispenser, inserts the oil gun into the fuel filler opening, starts refueling, finishes refueling, returns the oil gun to the dispenser, and leaves the store, all of which are performed solely by the customer after confirmation by a store employee or other person.

The self-service refueling management system 100 shown in FIG. 1 is configured to, for example, permit a user (refueler) MW to operate a dispenser D to refuel a vehicle W, and then carry out refueling. The types of fuel dispensed at the self-service refueling station include gasoline, diesel, LPG, and other oils.

1, the self-service refueling management system 100 includes a monitoring device Y ( determination means and display means), a control terminal (portable terminal) X, a control device SSC (refueling stop/prohibition), a monitoring camera C (imaging means), a sensor S, a dispenser D, and a server Z. Note that in the drawings and the following description, an example of a portable terminal X such as a tablet carried by a store clerk MX will be described as one form of the control terminal, but the same explanation applies when the control terminal is a terminal installed (fixed) in a predetermined location.

[Surveillance Camera]
In addition, the surveillance camera C is a means for capturing images of the user refueling, and for example, as shown in Figure 1, it captures at least the refueling action performed by the user MW, and outputs the image data obtained by capturing the image to the surveillance device Y.

The surveillance camera C includes, for example, a dispenser D camera that captures at least the area around the dispenser D and acquires image data, and a stopping area camera that captures at least the stopping area where the vehicle W being refueled by the dispenser D stops and acquires image data.

Note that, here, the "surroundings of the dispenser" refers to the range in which the dispenser body can be monitored and the range around the image that served as the basis for the determination.

It may also include a camera that captures images of the fuel filler opening of the vehicle W.

[sensor]
Furthermore, the sensor S is configured to detect at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling or actions of the user MW, for example, as shown in Figure 1, and output the detection results to the monitoring device Y.

The sensor S is installed, for example, near the dispenser D and/or in the stopping area where the vehicle W to be refueled by the dispenser D stops.

Note that the "stopping area" mentioned above refers to the area where a vehicle stops to refuel, and the image capture includes the entire vehicle and the area where a user would normally move to refuel.

[Monitoring device]
The monitoring device Y also has a determining means for determining whether the user's behavior is abnormal, normal, or unclear based on the captured image, and a means for displaying on the management terminal X the image that is the basis for the determination.

For example, as shown in Figure 1, the dispenser D and the control device SSC are installed separately. In particular, the monitoring device Y is located, for example, in a building within the self-service gas station.

The monitoring device Y monitors the refueling of the vehicle W using the dispenser D operated by the user MW, and determines whether the user MW is behaving abnormally (as described below), normally, or is unable to be determined, based on the results of analyzing image data obtained by capturing at least the user MW's behavior.

In addition, this monitoring device Y is configured to determine whether the current user MW is behaving abnormally, for example, by using a machine-learned judgment model that uses behavioral image data relating to images of past users' actions of refueling a vehicle W using a dispenser D, which have been acquired in the past, and that has been classified as abnormal or normal behavior by past users as training data. Once the judgment is complete, the image on which the judgment was made is registered in the image database Z1 and becomes part of the training data.

In particular, the monitoring device Y calculates the degree of similarity between, for example, the behavioral image data relating to the behavioral image of the user MW using the dispenser D to refuel the vehicle W obtained this time and the behavioral image data determined to be abnormal behavior from the previously described classified data, and determines whether the user MW is behaving abnormally based on the calculation results.

The "approximation" calculation can be performed using any known method. For example, it can be performed as described in Japanese Patent No. 4701100. Specifically, still portions are removed from the video, and a feature vector is calculated from the moving portions of the video using cubic higher-order local autocorrelation features. Alternatively, the feature vector can be calculated using optical flow. These feature vectors are statistically processed and displayed as features. One method is to compare these features with features of normal behavior, and determine that the closer the values are, the higher the degree of approximation. Several thresholds can then be set for the difference in values, and a judgment can be made that a value within this range is normal, a value within this range is unknown, and a value outside this range is abnormal. Alternatively, a multilayer neural network, as described in Japanese Patent Application Laid-Open No. 2019-96179, can be used to determine user behavior.

Furthermore, the monitoring device Y is configured to determine whether the user MW is engaging in abnormal behavior based on the analysis results of the image data and the detection results detected by the sensor S, for example, as shown in Figure 1.

This determination is generally made for each refueling process. Here, the refueling process refers to the actions a user takes at a self-service gas station to refuel, such as parking the vehicle in a parking area, selecting the type and amount of fuel to refuel, removing static electricity, removing the oil gun from the dispenser, inserting the oil gun's pipe into the fuel filler opening, starting refueling, completing refueling, removing the oil gun's pipe from the fuel filler opening, returning the oil gun to the dispenser, and leaving the station. Specifically, refueling begins when oil is dispensed from the oil gun's pipe by operating the lever on the oil gun.

Furthermore, regarding the aforementioned monitoring device Y's determination of normal, abnormal, or unknown, for example, if more than half of the oil gun's pipe is inserted into the fuel filler neck, it is determined to be "normal," if the tip of the oil gun's pipe is removed from the fuel filler neck, it is determined to be "abnormal," and if only the tip of the oil gun's pipe is inserted into the fuel filler neck, it is determined to be "unknown."

In addition, abnormal behaviors can be assessed not just for each step, but also for the overall risk of behaviors such as smoking.

The monitoring device Y then uses, for example, signal data (big data) previously acquired by the sensor S to determine whether the current user MW is behaving abnormally based on a comparison with data on abnormal user behavior in the past.

The image used to determine the user MW's behavior at this time is sent from monitoring device Y to management terminal X and displayed. A live image of user MW may also be displayed on management terminal X, but this control may be performed by monitoring device Y.

As shown in Figure 1, this monitoring device Y includes, for example, an image analysis device Y1, an interface Y2, and a wireless device Y3.

The image analysis device Y1 then analyzes image information of the user MW's behavior at least before and/or after the start of refueling, as shown in Figure 1, acquired by the surveillance camera C, and determines whether the user MW is behaving abnormally.

More specifically, this image analysis device Y1 is equipped with an AI robot (artificial intelligence), and uses behavioral image data relating to images of past user operations of refueling a vehicle W using a dispenser D to determine whether the current user MW is behaving abnormally, using a machine-learned judgment model as training data to determine whether the current user MW is behaving abnormally. Once the judgment is complete, the image on which the judgment was made is registered in the image database Z1 and becomes part of the training data.

Furthermore, wireless device Y3 is configured to communicate wirelessly with at least mobile terminal X, for example, as shown in Figure 1.

Furthermore, as shown in FIG. 1, the interface Y2 is interposed between the surveillance camera, the sensor S, the control device SSC, the image analysis device Y1, and the wireless device Y3.

[server]
1, the server Z is configured to perform machine learning on the classified data, which is obtained by classifying behavioral image data relating to an image of a past user's operation of refueling a vehicle W using a dispenser D, as training data, by judging the behavioral image data as abnormal or normal behavior of the past user, as shown in FIG. 1. The server Z is also configured to update the program of the above-mentioned judgment model.

This server Z includes, for example, an image database Z1 and an image recognition and machine learning unit Z2, as shown in Figure 1.

The image database Z1 is configured to store behavioral image data relating to images of past user operations to refuel a vehicle W using a dispenser D, which were previously acquired by the monitoring device Y.

In addition, the image recognition and machine learning unit Z2 uses the previously described behavioral image data, which has been classified as abnormal or normal behavior of the user in the past, as training data for machine learning, and updates the program of the previously described judgment model.

Classification data for an image database can be not only the registration of multiple images themselves, but also the numerical classification of user behavior, categorizing it as abnormal, normal, or unknown.

[Administrative terminal]
As mentioned above, in this embodiment, as one form of management terminal, an example of a mobile terminal X such as a tablet carried by a store clerk MX will be described, but the same explanation will be given in the case where the management terminal is a terminal installed (fixed) at a predetermined location.

The mobile terminal X, which is the management terminal, displays the image that served as the basis for determining whether the behavior was abnormal, unknown, or normal. It may also display live images. Here, live images refer to images of the area surrounding the image that served as the basis for the determination.

For example, as shown in FIG. 1, information acquired by monitoring device Y (such as image data acquired from monitoring camera C and detection results from sensor S, the judgment results of monitoring device Y, and the images on which the judgment is based) is acquired from monitoring device Y, output, and displayed, and is operated by store attendant MX who is monitoring the refueling of vehicle W.

This mobile terminal X is configured to receive information including the determination result from the monitoring device Y via wireless communication, for example, and to transmit information (instructions) corresponding to the operation of the store clerk MX to the monitoring device Y. Note that this mobile terminal X may also be configured to receive information including the determination result from the monitoring device Y via wired communication, and to transmit information (instructions) corresponding to the operation of the store clerk MX to the monitoring device Y.

Furthermore, this mobile terminal X may be configured to transmit information (instructions) corresponding to operations performed by the store clerk MX to the control device SSC as necessary.

In particular, this mobile terminal X is provided separately from the monitoring device Y and the control device SSC, as shown in FIG. 1. In particular, this mobile terminal X is operated by a store clerk MX at a remote location connected to the monitoring device Y via wireless communication or wireless communication.

Here, as shown in FIG. 1, the mobile terminal X includes, for example, a display unit H, an operation unit A, a communication unit T, a notification unit E, and a control unit CNT.

The display unit H of the mobile terminal X is configured to display predetermined information, such as the information acquired by the monitoring device Y described above and the judgment results of the monitoring device Y. This display unit H is, for example, a touch panel. Note that the display unit H of the mobile terminal X may be configured to display the same content as that displayed on the display unit G1 of the control device SSC described below.

The display unit H of this mobile terminal X includes, for example, a lane control status display unit CV, a live image display unit LV, a judgment image (image used as the basis for the judgment) display unit JV, a history display unit LG, and a button display unit SW, as shown in Figure 2.

The lane control status display unit CV displays a dispenser mark CV1 indicating each dispenser D and the fueling position CV2 for each lane.

The display of each refueling position CV2 also includes the control status of the dispenser D in each lane (whether it is waiting for refueling or on standby, the type of fuel, the amount of fuel to be refueled, and other order details, etc.).

In addition, the live image display unit LV is configured to display live images based on image data obtained by capturing images of the user MW's actions in order to relay the status of fueling to the vehicle W.

In addition, the judgment image display unit JV is configured to display judgments and judgment images based on image data obtained by capturing images of the user MW's behavior, which are the basis for the judgment results of the monitoring device Y.

In addition, the history display unit LG is configured to display the history (determination time, event log) of the user MW's actions for each refueling process as determined by the monitoring device Y.

The button display unit SW also displays buttons such as a "Confirm" button, "Allow" button, "Stop" button, and "Cancel Stop" button, which correspond to the operation unit A used by the store clerk MX to input operations.

In this way, for example, mobile terminal X acquires information acquired by monitoring device Y and the results of the monitoring device Y's judgment from monitoring device Y and displays them on the history display unit LG, and also displays live images (around the image data) based on image data obtained by capturing images of user MW's actions in order to relay the status of refueling of vehicle W on the live image display unit LV.

Furthermore, the mobile terminal X is configured to display a judgment image based on image data obtained by capturing images of the user MW's behavior, which serves as the basis for the judgment result of the monitoring device Y, on the judgment image display unit JV.

In addition, the mobile terminal X acquires from the monitoring device Y the history (event log) of the details of the user MW's behavior as determined by the monitoring device Y, and displays the determined image on the determination image display unit JV and also on the history display unit LG.

In addition, the mobile terminal X is configured to display on the history display unit LG the details of the user MW's behavior newly determined by the monitoring device Y, so that the details are added to the history (event log including determination time LG1, determination details LG2, determination result LG3, etc.).

In addition, the mobile terminal X acquires from the monitoring device Y the time at which the monitoring device Y determines whether the user MW is behaving abnormally, and displays this information on the history display unit LG along with the history (event log) of the content of the user MW's behavior as determined by the monitoring device Y.

In this way, store clerk MX can more accurately determine whether or not there is an abnormality by checking the evaluation image and history displayed on the display unit H of mobile terminal X.

Furthermore, the operation unit A of the mobile terminal X is used by the store clerk MX to input operations, as shown in FIG. 1, for example. This operation unit A is a touch panel portion (button display unit SW) that displays buttons for various operations and selections, such as a "Confirm" button, an "Allow" button, and a "Stop" button, for operation by the store clerk MX. In this case, the touch panel also functions as the display unit H, as described above.

In addition, the communication unit T of the mobile terminal X is configured to transmit and receive data such as predetermined information and commands via wireless communication with the monitoring device Y and/or the control device SSC.

Furthermore, the notification unit E of the mobile terminal X is configured to notify the store clerk MX of predetermined information. This notification unit E is configured to output a predetermined buzzer sound. Alternatively, the notification unit E may correspond to the part of the display unit H that outputs predetermined information.

The control unit CNT of the mobile terminal X is configured, for example, by a computer such as a CPU, and controls the operation of the display unit H, operation unit A, notification unit E, and communication unit T to perform the necessary processing of the mobile terminal X.

[Control device]
In addition, the control device SSC is a means for stopping or prohibiting refueling when the monitoring device Y determines that the user's behavior is abnormal or unknown or when a stop signal or prohibition signal is sent by operation from the management terminal X, and is a means for canceling the stop or prohibition by operation from the management terminal X.

Here, stopping refueling means that the supply of fuel currently being refueled will be prevented when a stop signal is issued during refueling, and prohibition means that the supply of fuel will be prevented before refueling begins.

For example, as shown in FIG. 1, the system controls the refueling of vehicle W using dispenser D operated by user MW in accordance with at least the determination results of monitoring device Y and the operation of mobile terminal X by store clerk MX.

For example, the control device SSC may be configured to refuel the vehicle W using the dispenser D in response to an operational input by the store clerk MX to the mobile terminal X instructing permission for refueling by the user MW.

On the other hand, the control device SSC is configured to suspend (prohibit) refueling of the vehicle W using the dispenser D in response to the determination result of the monitoring device Y and operation input by the store clerk MX to the mobile terminal X instructing the user MW to prohibit or stop refueling.

When the monitoring device Y determines that the user MW is behaving normally, the control device SSC allows the user to refuel the vehicle W using the dispenser D. For safety reasons, a system may be added in which a store clerk issues permission.

The "normal actions" of the user MW mentioned above include, for example, the user MW ordering refueling, the user MW removing static electricity, inserting the nozzle of the dispenser D into the fuel filler opening of the user MW's vehicle W, and other normal actions required for refueling.

The control device SSC is configured to stop or prohibit refueling of the vehicle W using the dispenser D if the monitoring device Y determines that the user MW is behaving abnormally or in an undeterminable manner during each refueling process.

The results of the monitoring device Y's judgment, information on the operation input of the mobile terminal X, and other information (such as image data captured by the monitoring camera C and the detection results of the sensor S) are transmitted to this control device SSC via the interface Y2 of the monitoring device Y. However, the necessary information may also be transmitted from the mobile terminal X to this control device SSC, for example.

Here, as shown in Figure 1, the control device SSC includes, for example, a display unit G1, an operation unit G2, a control unit G3, and a notification unit G4.

The display unit G1 of the control device SSC is configured to display predetermined information, such as the information acquired by the monitoring device Y described above and the judgment results of the monitoring device Y. This display unit G1 is, for example, a touch panel. As described above, the display unit G1 of the control device SSC may be configured to display the same information (image) as the display unit H of the mobile terminal X.

The operation unit G2 of the control device SSC is used by the store clerk MX to input operations, as shown in FIG. 1, for example. This operation unit G2 is a touch panel that displays buttons for various operations and selections, such as a "Confirm" button, an "Allow" button, and a "Stop" button, which the store clerk MX can operate. In this case, the touch panel also functions as the display unit G2 of the control device SSC, as described above.

In addition, the notification unit G4 of the control device SSC is configured to notify the store clerk MX of specified information. This notification unit G4 is configured to output a specified buzzer sound. Alternatively, this notification unit G4 may correspond to the part of the display unit G1 that outputs specified information.

The control unit G3 of the control device SSC is configured, for example, by a computer such as a CPU, and controls the operation of the display unit G1, operation unit G2, and notification unit G4 to perform the necessary processing of the control device SSC.

[Overall flow concept of the control method for self-service refueling in the self-service refueling management system]
Here, an example of the concept of the flow of the overall control method for self-service refueling in the self-service refueling management system 100 having the configuration and functions described above will be described.

Figure 3 is a conceptual diagram showing an example of the overall flow of the control method for self-service refueling in the self-service refueling management system 100 shown in Figure 1.

First, as shown in FIG. 3, for example, a user MW enters the self-service gas station in a vehicle W, parks the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and begins refueling (step S1).

At this time, the monitoring camera C begins capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S begins detecting at least one of the following that occurs in conjunction with the refueling or action performed by the user MW: sound, temperature, odor, light, or oil leakage, and outputs the detection results to the monitoring device Y.

Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data captured by the monitoring camera C of the user MW's behavior and the detection results of the sensor S (step S2).

If the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the user MW completes refueling (step S4).

On the other hand, if monitoring device Y determines in step S2 that user MW is behaving abnormally, the control device SSC prohibits user MW from refueling based on this determination result (step S5). At this time, an alarm is displayed on the management terminal on clerk MX's mobile terminal X, along with an image showing the basis for the determination. Then, clerk MX checks the image and visually confirms whether or not the behavior is abnormal. If the behavior is abnormal, the control device SSC again prohibits refueling of vehicle W using dispenser D based on operational input by clerk MX, who has checked the image, on mobile terminal X (or control device SSC). If the visual inspection reveals no problems, refueling is resumed (step S6).

Furthermore, if the monitoring device Y is unable to determine the user MW's behavior in step S2, the control device SSC prohibits the user MW from refueling based on this determination result (step S7), and the store clerk confirms the image that served as the basis for the determination. Thereafter, based on the operation input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC again prohibits or permits refueling of the vehicle W using the dispenser D (step S8), and the user MW completes refueling.

Figure 4 is a conceptual diagram showing another example of the overall flow of the control method for self-service refueling in the self-service refueling management system 100 shown in Figure 1. Note that the flow of steps S1 to S6 shown in Figure 4 is similar to the flow of steps S1 to S6 shown in Figure 3, but steps S11 to S13 differ from the steps shown in Figure 3.

That is, for example, as shown in FIG. 4, a user MW arrives at the self-service gas station in a vehicle W, stops the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and then begins refueling (step S1).

At this time, the monitoring camera C begins capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S begins detecting at least one of the following that occurs in conjunction with the refueling action performed by the user MW: sound, temperature, odor, light, or oil leakage, and outputs the detection results to the monitoring device Y.

Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data captured by the monitoring camera C of the user MW's behavior and the detection results of the sensor S (step S2).

If the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the user MW completes refueling (step S4).

On the other hand, if the monitoring device Y determines in step S2 that the user MW is behaving abnormally, the control device SSC stops the user MW from refueling based on this determination result (step S5), and the image displayed on the mobile terminal X as the basis for the determination is displayed on the display unit H. Thereafter, based on the operational input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC again stops or allows refueling of the vehicle W using the dispenser D (step S6).

Furthermore, if the monitoring device Y is unable to determine the user MW's behavior in step S2, the store clerk MX determines whether the user MW is behaving abnormally based on image data of the user MW's refueling behavior displayed on the display H of the mobile terminal X or on visual inspection (step S11). Based on the operation input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC allows the vehicle W to be refueled using the dispenser D (step S12), and the user MW completes refueling (step S13).

In this way, according to the control flow of the self-service refueling management system 100 shown in Figures 3 and 4 above, the monitoring device (AI robot) Y determines whether the user MW is behaving abnormally, and the store clerk can easily determine whether the user is behaving abnormally by checking the information (images, etc.) that served as the basis for the determination, thereby reducing the effort required of the store clerk to check. Furthermore, while the monitoring device (AI robot) performs control, the store clerk MX can determine the user MW's abnormal behavior as needed, allowing for more accurate decisions regarding refueling.

In particular, if the monitoring device Y determines that the user MW is taking normal actions for refueling, the control device SSC will allow the user MW to refuel based on this determination result, and the clerk MX can complete the refueling process for the user MW (steps S2 to S4).

In particular, by having monitoring device (AI robot) Y determine abnormal behavior by user MW, risk factors that go unnoticed by store staff are reduced, enabling safer operation of self-service gas stations.

It also has the effect of expanding the range of staff placement that is acceptable, while satisfying regulations under the Fire Service Act, which require staff to be on standby in a location where they can visually monitor the refueling point.

[Specific flow of refueling in the self-service refueling management system 100]
Next, an example of a specific flow of the self-service refueling management system 100 during refueling will be described, which specifically implements the concept of the control flow of the self-service refueling management system 100 shown in FIGS. 3 and 4 described above.

Here, Figure 5A is a flow diagram showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Figure 1. Also, Figure 5B is a flow diagram following Figure 5A, showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Figure 1. Also, Figure 5C is a flow diagram following Figure 5B, showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Figure 1. Note that the following explanation will focus on the operation of the mobile terminal X operated by the store clerk MX in the self-service refueling management system 100, but the operation of the control device SSC will also be explained in a similar manner.

For example, user MW arrives at the self-service gas station in vehicle W, parks vehicle W in a stopping area adjacent to dispenser D, gets out of the vehicle, and begins refueling. At this time, surveillance camera C begins capturing images of at least the refueling actions performed by user MW, and outputs the captured image data to monitoring device Y.

Then, for example, as shown in FIG. 5A, mobile terminal X displays "Waiting" in the lane control status display section CV2 of display section H (step S101 in FIG. 5A).

Then, the mobile terminal X sequentially switches and displays the live images of each lane on the live image display unit LV of the display unit H at predetermined time intervals (step S102 in Figure 5A).

Then, monitoring device Y determines whether user MX has completed the fuel supply setting for dispenser D in the lane (step S103 in Figure 5A).

If, in step S103, the monitoring device Y determines that the user MX has not completed the fueling setting for the dispenser D in the lane in question, the process returns to step S102, and the mobile terminal X causes the live image display unit LV of the display unit H to sequentially switch between displaying live images of each lane at predetermined intervals.

On the other hand, if in step S103 monitoring device Y determines that user MX has completed the fuel supply setting for dispenser D in the lane in question, mobile terminal X determines whether it has acquired from monitoring device Y the determination result and information regarding whether user MX has touched the static electricity removal sheet of dispenser D to complete static electricity removal (step S104 in Figure 5A).

Then, after obtaining from monitoring device Y the determination result and information regarding whether or not the static electricity removal has been completed in step S104, mobile terminal X obtains the determination result and information regarding whether or not user MX has removed the fuel nozzle from dispenser D in the lane (step S105 in Figure 5A).

Then, when the mobile terminal X acquires from the monitoring device Y the determination result and information that the user MX has removed the fuel nozzle from the dispenser D in the lane, it displays the determination result and information (determination image, determination time, etc.) on the history display section LG and determination image display section JV of the display unit H (step S106 in Figure 5A).

Then, the mobile terminal X switches the display of the refueling position CV2 corresponding to the lane in the lane control status display unit CV from "Waiting" to "Waiting for refueling," and displays a live image of the lane in the live image display unit LV (step S107 in Figure 5A).

Then, mobile device X notifies store clerk MX via notification unit E (step S108 in Figure 5A).

In response to this notification from notification unit E, store clerk MX can easily determine whether or not there is an abnormality by checking the evaluation image and history displayed on the history display unit LG, evaluation image display unit JV, and live image display unit LV of the display unit H of mobile terminal X, thereby reducing the amount of work required by the store clerk.

Then, as shown in FIG. 5B, for example, the mobile terminal X determines whether the refueling location CV2 corresponding to the lane in the lane control status display section CV of the display section H has been selected by the user MX (step S109 in FIG. 5B).

Then, if the mobile terminal X determines that the refueling location CV2 corresponding to the lane in the lane control status display section CV of the display section H has been selected by the user MX's operation, it stops the notification section E from notifying the store clerk MX (step S110 in Figure 5B).

Then, the mobile terminal X selects and displays (e.g., displays with a thick black frame) the fueling location CV2 corresponding to the lane selected by the user MX's operation in the lane control status display section CV of the display section H (step S111 in Figure 5B).

After that, when user MX begins inserting the nozzle of dispenser D into vehicle W, monitoring device Y determines whether user MX's insertion of the nozzle of dispenser D into vehicle W is an abnormal behavior.

Then, the mobile terminal X determines whether it has received from the monitoring device Y the determination result and information regarding whether the user MX has touched the static electricity removal sheet on the dispenser D and completed static electricity removal (step S112 in Figure 5B).

Then, when the mobile terminal X acquires the determination result and information regarding whether static electricity removal has been completed from the monitoring device Y in step S112, it overwrites and displays the determination image information from the determination result and information regarding whether static electricity removal has been completed in the determination image display unit LV, and also displays the determination time, determination content, etc. in the history display unit LG (step S113 in Figure 5B).

This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the evaluation image and history displayed on the history display unit LG and evaluation image display unit JV of the display unit H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

Then, the store clerk MX operates the permission button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S114 in FIG. 5B), which causes the control device SSC to activate the pump of the dispenser D (step S115 in FIG. 5B). This allows the user MW to start refueling the vehicle W using the dispenser D. The monitoring device Y then sequentially determines whether the user MW is behaving abnormally after refueling has begun.

Then, the mobile terminal X cancels the selection display (e.g., display with a thick black frame) of the refueling location CV2 corresponding to the lane in the lane control status display section CV of the display section H (step S116 in Figure 5B).

Then, the mobile terminal X sequentially switches and displays the live images of each lane on the live image display section LV of the display section H (step S117 in Figure 5B).

Then, as shown in FIG. 5C, for example, if the monitoring device Y does not detect any abnormal behavior by the user MW after refueling has begun (step S119 in FIG. 5C), the mobile terminal X confirms that the nozzle of the dispenser D is hooked on the nozzle hook (step S120 in FIG. 5C), causing the control device SSC to stop driving the pump of the dispenser D (step S121 in FIG. 5C). This ends the refueling of the vehicle W by the user MW using the dispenser D.

Then, the mobile device X erases the judgment image, judgment result, and information corresponding to that lane from the display unit H (step S122).

On the other hand, if the monitoring device Y determines in step S119 that the user MW is behaving abnormally after refueling has begun, and the abnormal behavior is that the nozzle has become detached from the fuel filler cap (step S123 in Figure 5), the control device SSC will stop driving the pump of the dispenser D (step S124 in Figure 5C).

Then, the mobile terminal X displays a live image of the lane in the live image display unit LV, overwrites the information on the judgment image in the judgment image display unit LV, and additionally displays the judgment time, judgment content, etc. in the history display unit LG (step S125 in Figure 5C).

This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the evaluation image and history displayed on the history display unit LG and evaluation image display unit JV of the display unit H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

After that, the store clerk MX, having confirmed that there is no problem with the user MX's behavior, operates the confirmation button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S126 in Figure 5C), which causes the display of "Confirmed" to be added to the history display unit LG, indicating that the store clerk MX has confirmed the transaction (step S127 in Figure 5C).

After that, the store clerk MX, having confirmed that there is no problem with the user MX's behavior, operates the stop release button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S128 in FIG. 5C), which causes the control device SSC to restart the pump of the dispenser D (step S129 in FIG. 5C). This allows the user MW to resume refueling the vehicle W using the dispenser D, and the process returns to the previously described step S119.

On the other hand, in the above-mentioned step S123, if the monitoring device Y determines that the abnormal behavior of the user MW after the start of refueling is something other than the nozzle being detached from the fuel filler cap (for example, user MX smoking), the mobile terminal X displays a live image of the lane in the live image display unit LV, overwrites the information on the judgment image in the judgment image display unit LV, and additionally displays the judgment time, judgment content, etc. in the history display unit LG (step S130 in Figure 5C).

This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the evaluation image and history displayed on the history display unit LG and evaluation image display unit JV of the display unit H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

After that, the store clerk MX, having confirmed that there is no problem with the user MX's behavior, operates the confirmation button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S131 in FIG. 5C), which causes the display of "Confirmed" to be added to the history display unit LG, indicating that the store clerk MX has confirmed the behavior (step S132 in FIG. 5C). Then, the process returns to the above-mentioned step S119, and the user MW continues to refuel the vehicle W using the dispenser D.

As described above, with the self-service refueling management system according to one embodiment of the present invention, the display of the mobile device displays images captured by a surveillance camera, information on the judgment made by the monitoring device (AI robot), and the judgment results. This allows the store attendant to determine whether or not there is an abnormality by checking not only the captured image but also the judgment made by the monitoring device (AI robot), thereby reducing the amount of checking required by the store attendant and further increasing safety within the gas station.

Furthermore, store staff can compare the live image (current) captured by the surveillance camera on the display of mobile device X with the judgment image (past) used for the monitoring device's judgment, making it easy to determine whether or not there are any abnormalities.

Furthermore, as mentioned above, the monitoring device (AI robot) Y determines and controls abnormal behavior of the user MW, and, if necessary, the store clerk MX determines and controls abnormal behavior of the user MW, thereby enabling labor savings and effective use of personnel.

In particular, by having monitoring device (AI robot) Y determine abnormal behavior by user MW, risk factors that go unnoticed by store staff are reduced, enabling safer operation of self-service gas stations.

It also has the effect of expanding the range of staff placement that is acceptable, while satisfying regulations under the Fire Service Act, which require staff to be on standby in a location where they can visually monitor the refueling point.

In addition, in the examples shown in Figures 5A and 5B above, if the judgment result indicates abnormal behavior and the nozzle is detached from the fuel filler neck, the pump will be stopped. However, this is not limited to this. If the judgment result is unclear, the pump may be stopped if the store clerk does not press the confirmation button after an alarm or image has been displayed to notify the store clerk, or if the store clerk does not press the confirmation button after a predetermined time has elapsed since the warning.

While several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their variations are within the scope of the invention and its equivalents as defined in the claims, as well as the scope and spirit of the invention.

100 Self-service refueling management system Y Monitoring device Y1 Image analysis device Y2 Interface Y3 Wireless device X Mobile terminal MX Clerk H Display unit A Operation unit T Communication unit E Notification unit CNT Control unit C Surveillance camera S Sensor D Dispenser Z Server SSC Control device G1 Display unit G2 Operation unit G3 Control unit G4 Notification unit W Vehicle MW User (fuel dispenser)
CV Lane control status display section LV Live image display section JV Judgment image display section LG History display section SW Button display section

Claims (6)

an imaging means for imaging a user filling up with fuel in a fueling lane;
a determining means for determining whether the behavior of the user is abnormal, normal, or unknown based on the image obtained by the imaging means;
prohibition means for stopping or prohibiting refueling when the determination means determines that the user's behavior for each refueling process is abnormal or unclear;
a portable management terminal that displays not only the image of the basis of the judgment but also a live image of the area surrounding the image that is the basis of the judgment ;
Including,
The self-service refueling management system is characterized in that the portable management terminal displays the judgment result of the judgment means based on the image obtained by the imaging means, indicating whether the user's behavior is abnormal, normal, or unknown. The self-service refueling management system described in claim 1, characterized in that the mobile management terminal displays an operation section that can stop or prohibit refueling on the screen that displays the live image and the judgment results. The self-service refueling management system described in any one of claims 1 and 2 further comprises a means for canceling the refueling process by operating the mobile management terminal if the refueling process is stopped or prohibited. The self-service refueling management system described in any one of claims 1 to 3 is characterized in that it has a means for determining whether a user's actions are abnormal, normal, or unknown for each refueling procedure, and displaying a history of the determined user actions on the mobile management terminal. The self-service fueling management system described in any one of claims 1 to 4, characterized in that the judgment means uses a machine-learned judgment model using data from an image database classified as abnormal or normal as training data to judge whether the image is abnormal, normal, or unknown, and registers the image used for the judgment in the image database. an imaging means for imaging a user filling up with fuel in a fueling lane;
a determining means for determining whether the behavior of the user is abnormal, normal, or unknown based on the image obtained by the imaging means;
prohibition means for stopping or prohibiting refueling when the determination means determines that the user's behavior for each refueling process is abnormal or unclear;
a portable management terminal that displays not only the image of the basis of the judgment but also a live image of the area surrounding the image that is the basis of the judgment ;
Including,
the mobile management terminal displays the result of the determination made by the determination means based on the image obtained by the imaging means, whether the behavior of the user is abnormal, normal, or unknown;
A gas station using a system characterized in that refueling can be stopped or prohibited by operating the mobile management terminal.