JP7372038B2 - Image analysis system and method - Google Patents

Description

The present invention relates to an image analysis system and an image analysis method.

BACKGROUND ART Gas stations that provide self-service refueling are conventionally known. According to laws and regulations, supervisors who are qualified as hazardous materials handlers are permanently stationed at such gas stations. The supervisor remotely controls whether refueling is possible while checking the refueling operations of the refueling person through live images of the refueling space from the gas station office.

In addition, Patent Document 1 states, ``The color tone of the fuel nozzle is identified by imaging the vicinity of the fuel filler opening of the vehicle, and based on the identified color tone of the fuel filler nozzle, the oil type of the fuel filler nozzle matches the oil type of the vehicle. If it is determined that the oil type is not suitable, a warning that the oil type is unsuitable is issued.''

Japanese Patent Application Publication No. 2009-179103

Patent Document 1 does not disclose warnings other than "unsuitable oil type based on the color tone of the oil supply nozzle".

However, there are other acts of misconduct (acts that deviate from safety or error prevention norms) that must be judged by the supervisor when determining whether or not to refuel.
In general, supervisors must constantly watch live video of the refueling space and be alert to various fraudulent activities at the same time. Therefore, there was a problem in that the burden on the supervisor was heavy.

Therefore, the present invention provides an image analysis technique for determining whether or not refueling is possible.

In order to solve the above problems, one of the representative image analysis systems of the present invention includes a video acquisition unit that acquires a video of a refueling space, and a learning model that performs machine learning to detect fraudulent acts from a group of videos of the refueling space. The system includes a detection unit that detects fraudulent activity in the video acquired by the video acquisition unit using the video acquisition unit, and a determination unit that generates information regarding whether or not refueling is possible based on the detection result of the fraudulent activity.

The present invention enables image analysis to determine whether refueling is possible.
Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

It is a diagram showing a self-service gas station GS. 1 is a block diagram of an image analysis system 100. FIG. FIG. 2 is a diagram showing the basic configuration of a learning model. FIG. 3 is a diagram illustrating an example of preprocessing of video information. FIG. 3 is a diagram showing a learning data set of refueling positions. FIG. 3 is a diagram showing a learning data set for static electricity removal behavior. FIG. 3 is a diagram showing a learning data set for nozzle insertion. FIG. 3 is a diagram showing a learning data set of the number of guests. FIG. 3 is a diagram illustrating a learning data set for refueling vehicles other than vehicles. FIG. 3 is a diagram showing a learning data set for caution-required situations. FIG. 3 is a diagram showing a learning data set for region segmentation of car numbers. It is a flowchart (1/2) showing the operation of the image analysis system 100. 2 is a flowchart (2/2) showing the operation of the image analysis system 100. It is a figure explaining the example of detection of the refueling position of a vehicle. FIG. 3 is a diagram illustrating an example of detecting static electricity removal behavior. FIG. 6 is a diagram illustrating an example of detecting the insertion status of a refueling nozzle. FIG. 3 is a diagram illustrating an example of detecting the number of guests. FIG. 3 is a diagram illustrating an example of detection of external refueling of a vehicle, dangerous/unsafe acts, etc. FIG. 3 is a diagram showing a display screen of a monitoring computer or a handheld terminal. It is a figure explaining the image analysis after refueling permission. 5 is a diagram showing main parts of an image analysis system 500. FIG. 5 is a flowchart showing the operation of the image analysis system 500.

Embodiments of the present invention will be described below with reference to the drawings.

[Configuration description of Example 1]
FIG. 1 is a diagram showing a self-service gas station GS.
In the figure, a plurality of fueling lanes (No1 to NoX) are installed at the gas station GS. A refueling device 10 is arranged in each of these refueling lanes. The refueling device 10 is provided with three systems of refueling nozzles corresponding to three types of oil (regular, high-octane, and light oil). These three systems of refueling nozzles are color-coded so that the oiler can easily distinguish between the oil types.

For example, a refueling nozzle that refuels with regular gasoline is color-coded "red." The refueling nozzle that supplies high-octane gasoline is color-coded "yellow." The refueling nozzle that supplies light oil is color-coded "green."

On the other hand, an image acquisition unit 11 is installed in each refueling lane so as to acquire an image of the refueling space with as few blind spots as possible. For example, the video acquisition unit 11 includes a camera A1 (for behavior analysis) that photographs the behavior of a refueling person from a bird's-eye view position or a bird's-eye view position, and a camera B1 (for vehicle number analysis) that photographs a vehicle's license plate from an oblique side direction. It consists of

FIG. 2 is a block diagram of an image analysis system 100 for monitoring refueling.
In the figure, an image analysis system 100 includes a video acquisition unit 11, a detection unit 12 (including learning models 200 to 260), a determination unit 13, a refueling history 14, a monitoring computer 15, an alarm 16, an alarm 16a, and a handheld terminal. 17, a wireless LAN 18, an interface section 19, a fuel supply control device 20, and a network switch 21.

Here, the detection unit 12, determination unit 13, and refueling history 14 are realized by the functions of an image processing server 22 with an AI learning function.

The image processing server 22 is configured as a computer equipped with a CPU (Central Processing Unit), memory, and the like as hardware. Part or all of this hardware may be replaced by a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or the like. Further, part or all of the hardware may be concentrated or distributed on servers on a network and placed in the cloud.

Next, the flow of signals within the image analysis system 100 will be explained.
The video acquisition unit 11 is connected to the network switch 21 via a LAN cable (PoE power supply). The video acquisition unit 11 receives power supply from the network switch 21 via a LAN cable. Further, the video information output from the video acquisition unit 11 is supplied to the network switch 21 via a LAN cable.

The network switch 21 switches the video information (live video) supplied from the video acquisition unit 11 for each fueling lane for each fueling lane and outputs it to the detection unit 12 .

The detection unit 12 analyzes the video information using the learning models 200 to 260 to detect fraudulent activities in the refueling space. The determination unit 13 generates information regarding refueling availability based on the detection result of fraudulent activity.

The information regarding refueling availability generated by the determination unit 13 is transmitted via the network switch 21 to the monitoring personal computer 15 in the office of the gas station GS. The monitoring personal computer 15 uses the monitor screen and the alarm 16 to alert the resident supervisor based on the information regarding whether refueling is possible or not.

Additionally, an alarm 16a is also installed in each fueling lane. These alarm devices 16a are connected to the monitoring personal computer 15 via a network switch 21. The monitoring computer 15 provides voice guidance and alerts to the refueling person via the alarm 16a.

Further, the monitoring personal computer 15 is connected to a fuel supply control device 20 via a network switch 21 and an interface section 19.

The refueling control device 20 controls the refueling stop of the refueling device 10, the amount of refueling, and the amount of discharge by controlling the refueling valve and refueling pressure in the refueling device 10 for each refueling lane.

This refueling device 10 also communicates with a monitoring personal computer 15 and an image processing server 22 via a network switch 21, and transmits information such as billing and settlement at the time of refueling, selected oil type and amount of refueling. The refueling history 14 records information on the history related to refueling.

Note that the supervisor can perform fuel supply monitoring and fuel supply control similar to the monitoring computer 15 via a handheld terminal 17 such as a tablet. A plurality of wireless LANs 18 are distributed and arranged so that this handheld terminal 17 covers communication within the premises of the gas station GS.

Note that the learning models 200 to 260 included in the detection unit 12 are used for image analysis such as the next fraudulent act and car number recognition.
・Learning model 200: Misbehavior of not stopping correctly at the refueling position ・Learning model 210: Misbehavior of not removing static electricity ・Learning model 220: Misbehavior of not correctly inserting the refueling nozzle ・Learning model 230: Unnecessary number of people staying Fraudulent behavior/learning model 240: Fraudulent behavior/learning model of refueling something other than a vehicle 250: Fraudulent behavior/learning model regarding caution-required situations 260: Recognition of vehicle number Machine learning and use of these learning models 200 to 260 will be described later. do.

[Basic configuration of learning models 200 to 260]
First, the basic configuration common to the learning models 200 to 260 will be explained.

FIG. 3 is a diagram showing the basic configuration of learning models 200 to 260.
In the figure, image information of the refueling space is input to a preprocessing section 31. The preprocessing unit 31 performs preprocessing on the video information in accordance with the content of the image analysis.

For example, the preprocessing unit 31 performs analysis area setting (cropping, resizing, bird's-eye transformation, viewpoint transformation, distortion correction, padding before convolution, etc.), difference comparison (background difference, contour extraction, etc.), vector analysis (motion vector detection, motion tracking), signal processing (feature extraction, shape extraction, edge enhancement, gradation correction, dynamic range expansion/reduction, level normalization, noise suppression, moiré removal, white balance adjustment, color processing, color reduction processing, monochrome) (e.g.) and other pre-treatments.

For example, in the learning model that performs "image analysis for the presence or absence of an object" shown in FIG. Perform image processing such as cropping.

In addition, in the learning model that performs "image analysis regarding refueling nozzles, refueling destinations, etc." shown in FIG. Determine. Further, the preprocessing unit 31 detects the target area of the refueling nozzle or the refueling destination by vector analysis of the movement of the refueling nozzle (or the hand of the refueling person), and crops the target area.

Furthermore, in the learning model that performs "image analysis related to vehicle number" shown in FIG. Crop. Furthermore, if necessary, keystone correction or the like may be applied to the license plate, or color reduction processing may be performed to remove the background color of the license plate.

In addition, in the type of learning model that performs "image analysis related to human behavior" shown in FIG. Detects the target area, such as inside, and crops the target area.

The video information preprocessed by the preprocessing unit 31 is input to the input layer 32 in the learning models 200 to 260. The video information in the input layer 32 is processed through a convolution layer/activation function 33 and a pooling layer 34, and is converted into a feature map consisting of neurons that are activated according to local features of the image space. The convolution layer/activation function 33 and pooling layer 34 are multilayered by the number of layers suitable for image analysis.

The neuron values of the feature map that have passed through the multi-layered convolution layer/activation function 33 and pooling layer 34 are serialized and then input to the neural network 35 . The neural network 35 is configured by connecting (for example, fully connecting) neurons with a number of layers suitable for image analysis. An output layer 36 at the final end of the neural network 35 outputs a value indicating the result of image analysis (likelihood, etc.).

[Machine learning of learning models 200 to 260]
The neurons in the convolution layer/activation function 33 and the neural network 35 described above have a weight array w and a bias b. These weight array w and bias b are sequentially updated by repeating machine learning (error backpropagation method, etc.) using a learning data set, and a determination result that is in line with the learned teacher data is output.

This machine learning is performed as a preparation before the system starts operating. Furthermore, after the system starts operating, actual surveillance video is automatically collected as training images, and the supervisor's judgments regarding the surveillance video are used as training data, so that machine learning is performed in accordance with reality.

5 to 11 are diagrams showing examples of learning data sets (set groups of training images and teacher data) used in such machine learning, classified by type of image analysis. Each of these learning data sets will be explained below.

・Machine learning of "fraudulent act of not stopping correctly at the refueling position" for the learning model 200 Figure 5 shows a learning dataset for performing machine learning on the correct stopping position of the vehicle during refueling (hereinafter referred to as "refueling position"). 200A-C are shown. The training images of the learning datasets 200A to 200C are preprocessed by the preprocessing unit 31 of the learning model 200 (a process of cropping the vicinity of the refueling position as a target area, preferably a process of converting the stop frame into a rectangle by bird's-eye view conversion). is the video information that has already been implemented.

In the same figure, the group of training images of the learning data set 200A is video information of a state in which the vehicle is stopped without protruding from the stop frame of the refueling space, and the refueling port of the vehicle is in the direction of the refueling device 10. . Teacher data of "refueling position OK" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 200B is video information of a state in which the vehicle is stopped and protrudes from the stop frame of the refueling space. Teacher data of "refueling position NG" is assigned to these training image groups.

Furthermore, the group of training images of the learning data set 200C is video information in a state where the fuel filler port of the vehicle is on the opposite side from the fuel filler 10. Teacher data of "refueling position NG (or refueling port facing backward NG)" is assigned to these groups of training images.

Using the learning data sets 200A to 200C shown in FIG. 5, machine learning is performed to determine whether or not the vehicle is stopped at the refueling position. A learning model 200 is obtained that detects fraud on the opposite side.

- Machine learning of "fraudulent act of not removing static electricity" for learning model 210 FIG. 6 shows learning data sets 210A to 210C for performing machine learning of fraudulent act of not removing static electricity. The training images of the learning data sets 210A to 210C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 210 (processing of cropping the vicinity of the static electricity removal sheet as a target area).

In the figure, the group of training images of the learning data set 210A is video information of a refueling person touching a static electricity removal sheet with his bare hands (not wearing gloves or the like). Teacher data of "static electricity removal OK" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 210B is video information of a state in which the refueling person is not in contact with the static electricity removal sheet. Teacher data of "static electricity removal NG" is assigned to these groups of training images.

Furthermore, the group of training images of the learning data set 210C is video information of a state in which a refueling person is in contact with a static electricity removal sheet through a glove or handkerchief (insufficient discharge). Teacher data of "static electricity removal NG (or glove wearing NG)" is assigned to these training image groups.

Using the learning data sets 210A to 210C shown in FIG. 6, machine learning is performed to determine whether or not a refueling person has taken action to remove static electricity. A learning model 210 is obtained that detects a fraudulent act in which static electricity is removed.

- Machine learning of "fraudulent act of not correctly inserting the refueling nozzle" for the learning model 220 FIG. 7 shows learning data sets 220A to 220C for machine learning of the fraudulent act of incorrectly inserting the refueling nozzle. The training images of the learning data sets 220A to 220C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 220 (processing of cropping the vicinity of the refueling port detected by the movement vector of the refueling nozzle as a target area). It is.

In the figure, the group of training images of the learning data set 220A is video information of a state in which the refueling nozzle is correctly inserted into the refueling port. Teacher data of "nozzle insertion OK" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 220B is video information in which the fuel nozzle is not inserted into the fuel filler port. Teacher data indicating "nozzle insertion NG" is assigned to these groups of training images.

Furthermore, the group of training images of the learning data set 220C is video information in which the refueling nozzle is not inserted all the way into the refueling port and is floating or tilted. Teacher data indicating "nozzle insertion NG" is assigned to these groups of training images.

A learning model that detects fraudulent acts in which the refueling nozzle is not correctly inserted into the refueling port by performing machine learning on whether or not the refueling nozzle is correctly inserted into the refueling port using the learning data sets 220A to 220C shown in FIG. 220 is obtained.

- Machine learning of "fraudulent behavior in which an unnecessary number of people stay" for the learning model 230 FIG. 8 shows learning data sets 230A to 230C for machine learning of fraudulent behavior related to the number of guests staying. The training images of the learning data sets 230A to 230C are preprocessed by the preprocessing unit 31 of the learning model 230 (processing of cropping the surroundings of the vehicle as a target area, preferably processing of converting the stop frame into a rectangle by bird's-eye conversion). This is video information that has been completed.

In the same figure, the group of training images of the learning data set 230A is video information in a state where the number of people staying is the same as the number required for refueling (for example, one refueler). Teacher data of "number of guests allowed" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 230B is also video information in a state where the number of people staying is the same as the number required for refueling (for example, about two people, a refueler and an advisor). Teacher data of "number of guests allowed" is assigned to these groups of training images.

Furthermore, the group of training images of the learning data set 230C is video information of a state in which the number of people staying exceeds a predetermined number required for refueling. Teacher data of "No number of guests staying" is assigned to these groups of training images.

By performing machine learning on the number of people staying in a refueling space using the learning data sets 230A to C shown in FIG. 8, a learning model 230 that detects fraudulent acts in which the number of people in the refueling space exceeds a predetermined number is obtained.

- Machine learning of "fraudulent act of refueling something other than a vehicle" for the learning model 240 FIG. 9 shows learning data sets 240A to 240C for machine learning of the fraudulent act of refueling something other than a vehicle. The training images of the learning datasets 240A to 240C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 240 (processing to crop the refueling nozzle insertion destination and reach range as the target area). .

In the same figure, the group of training images of the learning data set 240A is video information of a state in which a vehicle is being refueled. Teacher data of "vehicle refueling OK" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 240B is video information of a state in which a vehicle other than a vehicle (kerosene tank, plastic tank, portable can, etc.) is being refueled. Teacher data of "no refueling to vehicles other than vehicles" is assigned to these groups of training images.

Furthermore, the group of training images of the learning data set 240C is video information of refueling a vehicle (kerosene tank, plastic tank, portable can, etc.) mounted in the trunk of a vehicle. Teacher data of "no refueling to vehicles other than vehicles" is assigned to these groups of training images.

Using the learning data sets 240A to 240C shown in FIG. 9, a learning model 240 for detecting fraudulent acts of refueling other than vehicles can be obtained by performing machine learning on whether the refueling destination is a vehicle or not.

- Machine learning of "fraudulent acts related to cautionary situations" for the learning model 250 FIG. 10 shows learning data sets 250A to 250C for machine learning of fraudulent acts related to cautionary situations. The training images of the learning data sets 250A to 250C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 250 (processing of cropping the range where the caution-required situation occurs as a target area).

In the figure, the groups of training images of learning data sets 250A and 250B are video information in which one of the following types of caution-required situations (or a combination of multiple types of caution-required situations) has occurred.

(1) Use of open flames, (2) Cigarettes, (3) Lighters, (4) Matches, (5) Smoking implements, (6) Smoking, (7) Ignition, (8) Spark generation, (9) High temperature objects, ( 10) Flammable materials, (11) Combustible materials, (12) Explosive materials, (13) Dangerous materials, (14) Carrying cans, (15) Kerosene tanks, (16) Carelessness of talking or staring at screens, (17) Inattention due to mobile phones, (18) Inattentiveness due to mobile devices, (19) Presence of children outside the vehicle, (20) Looking away while refueling, (21) Conversation while refueling, (22) Falls of refueling person, etc. , (23) Disputes among refueling personnel, (24) Disturbances in the refueling space, (25) Losing or forgetting to close the refueling cap, (26) Abnormal behavior such as being drunk, falling down, crouching, or hiding something, ( 27) Careless behavior, (28) Dangerous behavior Teacher data of "Caution Required Situation NG" is assigned to these groups of training images.

On the other hand, the group of training images of the learning data set 250C is video information obtained by removing the "situation requiring attention" from the training images of the learning data sets 250A and 250B. Teacher data of "OK, no situation requiring attention" is assigned to these groups of training images.

Using the learning data sets 250A to 250C shown in FIG. 10, a plurality of learning models 250 that respectively detect fraudulent acts related to situations requiring attention are obtained by performing machine learning on fraudulent acts related to situations requiring attention.

- Machine learning of "vehicle number recognition" for learning model 260 FIG. 11 shows a learning data set 260A for region segmentation of vehicle numbers. The training images of this learning dataset 260A are processed by the preprocessing unit 31 of the learning model 260 (processing of cropping the license plate area, preferably converting the license plate into a rectangle, and normalizing the image size of the license plate. This is video information that has already been processed (processing to do so).

Area labels (land code, vehicle type code, usage code, serial designation number code, etc.) for dividing the training images into areas by code type are assigned to these groups of training images. The area label here is given in accordance with the license plate display laws and regulations of the country or region where the image analysis system 100 is used.

By performing machine learning using such a learning data set 260A, a learning model 260 that divides the vehicle number of a license plate into image regions according to code type is obtained.

Further, this learning model 260 reads each code of the car number as letters, numbers, and symbols by processing the segmented image area with a learning device for character recognition.

[Operation description of image analysis system 100]
Next, the operation of the image analysis system 100 performed using the learning models 200 to 260 that have undergone machine learning will be explained.

12 and 13 are flowcharts showing the operation of the image analysis system 100.
14 to 18 are diagrams for explaining the state of image analysis.
FIG. 19 is a diagram showing a display screen of the monitoring personal computer 15 and the handheld terminal 17.
The operation of image analysis will be described below in the order of step numbers shown in FIGS. 12 and 13.

Step S11: Cameras A1 and B1 (video acquisition unit 11) arranged for each refueling lane output video information (video) of the refueling space. This video information is input to the detection unit 12 in the image processing server 22 via the network switch 21.

Step S12: The detection unit 12 outputs video information to the learning model 200. The preprocessing unit 31 in the learning model 200 performs preprocessing on this video information (processing of cropping the vicinity of the vehicle's refueling position as a target area, preferably processing of converting the stop frame into a rectangle by bird's-eye conversion). . The preprocessed video information is processed through a learning model 200 that performs machine learning to determine whether the vehicle is correctly stopped at the refueling position. The learning model 200 outputs the likelihood of a fraudulent act in which the vehicle does not stop at the refueling position. Alternatively, the likelihood of a fraudulent act in which the vehicle is in the opposite direction and the fuel filler port is not on the fuel supply device 10 side is output.

FIG. 14 is a diagram illustrating an example of detecting the refueling position of a vehicle.
The more the vehicle stops in the correct position and orientation, the lower the likelihood of fraud. Conversely, the more the vehicle deviates from its correct refueling position or the more the vehicle is turned in the wrong direction, the higher the likelihood of fraud.

Step S13: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 200. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S14. In other cases, the determination unit 13 moves the operation to step S15.

Step S14: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring personal computer 15, handheld terminal 17, and alarm device 16a. The monitoring computer 15, the handheld terminal 17, and the alarm 16 can monitor and display information such as "fraudulent behavior in which the vehicle does not stop at the refueling position" or "fraudulent behavior in which the vehicle is facing the wrong direction and the refueling port is not on the side of the refueling device 10." The supervisor will be notified by voice notification or alarm lighting. The alarm device 16a placed in the refueling lane also issues audio notifications and lights up an alarm, warning the refueling person of ``unauthorized behavior in which the vehicle does not stop at the refueling position'' or ``the vehicle is facing the wrong direction and the refueling port is on the side of the refueling device 10.'' Report non-existent fraudulent activities. After these notification operations, the determination unit 13 returns the operation to step S12.

Step S15: The detection unit 12 outputs video information to the learning model 210. The preprocessing unit 31 in the learning model 210 performs preprocessing (processing of cropping the vicinity of the static electricity removal sheet as a target area) on this video information. The preprocessed video information is processed through a learning model 210 that performs machine learning to determine whether static electricity has been removed. The learning model 210 outputs the likelihood of cheating without static electricity removal.

FIG. 15 is a diagram illustrating an example of detecting static electricity removal behavior.
If the refueler's hand touches the static electricity removal sheet even for a moment, the likelihood of fraudulent activity is reduced. However, if an intervening object such as a glove or handkerchief whose color is different from the skin color is detected, the likelihood of fraudulent activity increases.

Step S16: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 210. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S17. In other cases, the determination unit 13 moves the operation to step S18.

Step S17: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring personal computer 15, handheld terminal 17, and alarm device 16 notify the supervisor of "unauthorized acts of not removing static electricity" by displaying on the monitor, giving voice notification, or lighting an alarm. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueling person of ``misconduct of not removing static electricity.'' After these notification operations, the determination unit 13 returns the operation to step S15.

Step S18: The detection unit 12 outputs video information to the learning model 220. The preprocessing unit 31 in the learning model 220 performs preprocessing (processing of cropping the vicinity of the fuel filler port detected by vector tracking of the fuel filler nozzle as a target area) on this video information. The preprocessed video information is processed through a learning model 220 that performs machine learning to determine whether the fuel nozzle is correctly inserted into the fuel filler port. The learning model 220 outputs the likelihood of cheating by not correctly inserting the refueling nozzle.

FIG. 16 is a diagram illustrating an example of detecting the insertion status of the refueling nozzle.
The more correctly the refueling nozzle is inserted into the refueling port, the lower the likelihood of fraud. The more the refueling nozzle floats or tilts, the higher the likelihood of fraud.

Step S19: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 220. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S20. In other cases, the determination unit 13 moves the operation to step S21.

Step S20: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring personal computer 15, handheld terminal 17, and alarm device 16a. The monitoring computer 15, handheld terminal 17, and alarm device 16 notify the supervisor of "unauthorized act of not correctly inserting the refueling nozzle into the refueling port" through monitor display, audio notification, and lighting of an alarm. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueling person of ``misconduct by not correctly inserting the refueling nozzle into the refueling opening.'' After these notification operations, the determination unit 13 returns the operation to step S18.

Step S21: The detection unit 12 outputs video information to the learning model 230. The preprocessing unit 31 in the learning model 230 performs preprocessing (a process of cropping the area around the vehicle as a target area, preferably a process of converting the stop frame into a rectangle by bird's-eye view conversion) on this video information. The preprocessed video information is processed through a learning model 230 that performs machine learning to determine the number of people who are unnecessary for refueling. The learning model 230 outputs the likelihood of a fraudulent act in which the number of people exceeds a predetermined number and an unnecessary number of people stay in the refueling space.

FIG. 17 is a diagram illustrating an example of detection of a person other than the refueler.
If there are enough people to refuel, the likelihood of fraud is low. As the number of non-fuelers increases, the likelihood of fraud increases.

Step S22: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 230. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S23. In other cases, the determination unit 13 moves the operation to step S24.

Step S23: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring computer 15, hand-held terminal 17, and alarm device 16 notify the supervisor of ``unauthorized behavior in which an unnecessary number of people stay for refueling'' through monitor display, audio notification, and alarm lighting. The alarm device 16a placed in the refueling lane also provides audible notification and lights up an alarm to notify refueling personnel of ``a fraudulent act in which an unnecessary number of people stay for refueling.'' After these notification operations, the determination unit 13 returns the operation to step S21.

Step S24: The detection unit 12 outputs video information to the learning model 240. The preprocessing unit 31 in the learning model 240 performs preprocessing (processing of cropping the range that the refueling nozzle can reach as a target area) on this video information. The preprocessed video information is processed through a learning model 240 that performs machine learning to determine whether the refueling destination is a vehicle (including a motorcycle). The learning model 240 outputs the likelihood of a fraudulent act in which the refueling destination is somewhere other than a vehicle.

FIG. 18 is a diagram illustrating an example of detecting refueling of a vehicle other than a vehicle.

If the vehicle is refueled normally, the likelihood of fraud is low. If a refueling location other than the vehicle is detected, the likelihood of fraudulent activity increases.

Step S25: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 240. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S26. In other cases, the determination unit 13 moves the operation to step S27.

Step S26: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring computer 15, hand-held terminal 17, and alarm device 16 notify the monitor of "unauthorized acts of refueling other than vehicles" through monitor display, audio notification, and alarm lighting. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueler of ``unauthorized acts of refueling other than vehicles.'' After these notification operations, the determination unit 13 returns the operation to step S24.

Step S27: The detection unit 12 outputs video information to the learning model 250. The preprocessing unit 31 in the learning model 250 performs preprocessing on this video information (processing such as cropping a range where a caution-required situation occurs as a target area). The pre-processed video information is processed through a learning model 250 that has machine learned the cautionary situation. The learning model 250 outputs the likelihood of cheating regarding the cautionary situation.

Step S28: The determination unit 13 generates information regarding refueling availability based on the likelihood of fraud output by the learning model 250. If it is determined that refueling is not possible here, the determination unit 13 moves the operation to step S29. In other cases, the determination unit 13 moves the operation to step S30.

Step S29: The determination unit 13 transfers the information regarding the impossibility of refueling to the monitoring personal computer 15, handheld terminal 17, and alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the monitoring person of the "unauthorized act related to the situation requiring attention" through a monitor display, audio notification, or lighting of an alarm. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueling person of ``illegal activity related to a situation requiring attention.'' After these notification operations, the determination unit 13 returns the operation to step S27.

Step S30: The detection unit 12 outputs video information to the learning model 260. The preprocessing unit 31 in the learning model 260 performs preprocessing on this video information (a process of cropping the license plate area, preferably a process of converting the license plate into a rectangle and normalizing the image size of the license plate). Implement. The preprocessed video information is processed through a learning model 260 that performs machine learning on car number reading. The learning model 260 outputs the result of reading the car number.

Step S31: The detection unit 12 determines the color of the refueling nozzle used for refueling based on the video information of the refueling space, and acquires information about the type of oil to be refueled. Note that information may be acquired from the oil supply device 10 or the like as the oil type selected for purchase by the oil supply device 10 as the oil type to be refueled.

Step S32: The determination unit 13 checks the vehicle number read in step S30 against the refueling history 14. If the same vehicle number is found in the refueling history 14, the determining unit 13 acquires the type of oil that was refueled in the past from the refueling history 14.

Step S33: If the oil type information acquired in step S31 is different from the oil type that was refueled in the same vehicle in the past, the determination unit 13 moves the operation to step S34. On the other hand, when refueling with the same type of oil as in the past or when the vehicle number is not found in the refueling history 14, the determination unit 13 moves the operation to step S35.

Step S34: The determination unit 13 transfers the information regarding the unsuitability of the oil type to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring personal computer 15, handheld terminal 17, and alarm device 16 notify the supervisor that "refueling is not possible due to unsuitable oil type" through a monitor display, audio notification, or lighting of an alarm. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueler that "refueling is not possible due to inappropriate oil type." After these notification operations, the determination unit 13 returns the operation to step S31.

Step S35: The determination unit 13 queries the vehicle number read in step S30 to an external registration database via a network such as the Internet, and obtains registration information on the oil type and vehicle type. (If permission can be obtained in advance, such as as a regular customer, if possible, obtain oil type and vehicle type registration information from a vehicle inspection registration information service, etc.)

Step S36: If the oil type information acquired in step S31 does not match the registered information, the determination unit 13 moves the operation to step S37. On the other hand, when refueling with an oil type that matches the registered information or when the vehicle number is not found in the registration database, the determination unit 13 shifts the operation to step S38.

Step S37: The determination unit 13 transfers the information regarding the unsuitability of the oil type to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring personal computer 15, handheld terminal 17, and alarm device 16 notify the supervisor that "refueling is not possible due to unsuitable oil type" through a monitor display, audio notification, or lighting of an alarm. The alarm device 16a placed in the refueling lane also issues an audio notification and lights up an alarm to notify the refueler that "refueling is not possible due to inappropriate oil type." After these notification operations, the determination unit 13 returns the operation to step S31.

Step S38: The determination unit 13 determines whether or not the car number read in step S30 is a car number for a rental car (including car sharing) (for example, in Japan, if the usage code of the car number is "wa" or "re", The vehicle number of the rental car is determined.

Step S39: In the case of a rental car, since the refueling person uses the vehicle temporarily, there is a high probability that the refueling person will use the wrong type of oil. Therefore, when the determining unit 13 detects the car number of the rental car, it determines that there is a "possibility of incorrect oil type" and moves the operation to step S40. If it is not the car number of a rental car, the determination unit 13 moves the operation to step S41.

Step S40: The determination unit 13 transmits the possibility of an incorrect oil type to the monitoring computer 15, handheld terminal 17, and alarm device 16a. The monitoring computer 15, hand-held terminal 17, and alarm device 16 notify the monitor that "there is a risk of incorrect oil type" through a monitor display, audio notification, or lighting of an alarm. The alarm device 16a disposed in the refueling lane also provides audible notification and lights up an alarm to notify the refueling person that there is a "possibility of using the wrong type of oil." After these notification operations, the determination unit 13 returns the operation to step S41.

Step S41: In the case of a vehicle with a small fuel tank, such as a motorcycle, the nozzle is inserted at an angle or shallowly, so full tank detection (fuel stopper operation) by the fuel nozzle may be inaccurate. Therefore, when the determination unit 13 detects the vehicle number of a vehicle with a small fuel tank, such as a motorcycle, the determination unit 13 suppresses the amount of fuel discharged and the amount of fuel supplied to avoid fuel spillage.

Step S42: At the self-service gas station GS, a supervisor who is a hazardous materials handler monitors refueling performed by a refueling person and gives permission for refueling, in accordance with laws and regulations.
FIG. 19 is a diagram showing the screen display of the monitoring personal computer 15 and handheld terminal 17 used by this supervisor.
In the figure, the screen display is composed of multiple windows such as a live display area 301, an information field 302, an analysis display area 303, a confirmation required button 304, and an operation area 305.
In the live display area 301, live images of each refueling lane shot by the image acquisition unit 11 are simultaneously displayed on multiple screens.
The information column 302 includes the vehicle number detected in step S30, the registration information inquired in step S35, the refueling history 14 data inquired in step S32, information on the customer's billing decision, and the past information of the vehicle and the refueling person. A list of fraudulent activities will be displayed.

The analysis display area 303 displays, as analysis results by the learning models 200 to 260 described above, images of important parts related to the impossibility of refueling, a situational explanation of the analysis contents, and countermeasures against fraudulent acts.
The analysis display area 303 may display the analysis results for the refueling lane selected by the monitor through screen selection in the live display area 301.
Further, analysis results for each of a plurality of fueling lanes may be automatically switched and displayed at predetermined intervals.
Furthermore, the analysis results of the refueling lane where the refueler is present may be preferentially displayed.
Further, the analysis results of refueling lanes where illegal acts are occurring may be automatically selected and displayed.

Of the analysis results displayed in the analysis display area 303, a confirmation required button 304 lights up or blinks to prompt the supervisor to confirm the analysis results that may pose a security problem. The monitor operates (clicks or taps) the confirmation required button 304 after confirming the explanation of the situation of the analysis content. The operation result of the confirmation required button 304 is sequentially recorded in a monitoring log file in the monitoring personal computer 15 along with the details of the fraudulent act, live video, and date and time information.

The operation area 305 displays a refueling lane selection button and operation buttons (OK button, NG button) for controlling whether refueling is possible in the selected refueling lane. The supervisor allows the refueling person to refuel by operating the OK button. Furthermore, the supervisor prohibits (or temporarily stops) the refueling person from refueling by operating the NG button.

Step S43: The determining unit 13 records information regarding the vehicle number, the type of oil refueled, the amount of refueling, the date and time, the details of the fraudulent act, and whether or not refueling is possible in the refueling history 14.

By repeating the above series of monitoring support as needed until refueling is permitted, the supervisor can carefully and safely grant refueling permission with less burden.

[Operation of image analysis system 100 after refueling permission]
FIG. 20 is a diagram showing items of fraudulent acts that are detected even after refueling permission has been granted.
Hereinafter, operations after refueling permission will be explained for each item shown in the figure.

・"Unauthorized act of not inserting the refueling nozzle correctly"
Immediately before refueling or during refueling, there is a known practice of floating the refueling nozzle from the refueling port to delay liquid level detection by the refueling nozzle, and refueling until the very edge of the refueling port (hereinafter referred to as "additional refueling"). In this additional refueling action, the automatic refueling stop function may not function sufficiently, and there is a possibility that fuel may spill out from the refueling port.

Therefore, the image analysis system 100 continues to detect the insertion status of the refueling nozzle even after refueling is permitted. As a result, it becomes possible to detect an additional refueling action in which the refueling nozzle is lifted from the refueling port immediately before refueling or during refueling.

If an additional refueling action is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to refueling prohibited. In this state, the alarm 16a alerts the refueling person, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the refueling nozzle has been correctly inserted, the supervisor operates the monitoring computer 15 or the like to switch refueling from prohibited to refueling permitted.

Further, if an additional refueling action is detected during refueling, the image analysis system 100 temporarily changes refueling permission to refueling not possible, and performs control to temporarily stop refueling for safety. In this state, the alarm 16a alerts the refueling person, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the refueling nozzle is correctly inserted, the supervisor operates the monitoring personal computer 15 or the like to change refueling from prohibited to refueling permitted, and cancels the temporary suspension of refueling.

・"Unauthorized act of not stopping correctly at the refueling position"
If the driver is different from the refueler, the driver may mistakenly start the vehicle when refueling. In this case, not only will refueling not be carried out normally, but there is also the possibility that an unexpected situation may occur due to the engine starting during refueling, or that fuel may spill from the refueling port due to shaking of the vehicle body.

Therefore, the image analysis system 100 continues to detect the refueling position of the vehicle even after refueling is permitted. As a result, it becomes possible to detect vehicle movement immediately before refueling or during refueling.

When vehicle movement is detected immediately before refueling, the image analysis system 100 temporarily changes refueling permission to refueling not possible. In this state, the alarm 16a alerts the driver, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the vehicle has returned to the correct refueling position, the supervisor operates the monitoring computer 15 or the like to switch refueling from prohibited to refueling permitted.

Further, if vehicle movement is detected during refueling, the image analysis system 100 temporarily changes refueling permission to refueling not possible, and performs control to temporarily stop refueling for safety. In this state, the alarm 16a alerts the driver, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the vehicle has returned to the correct refueling position, the supervisor operates the monitoring computer 15 or the like to change refueling from prohibited to refueling permitted, and cancels the temporary suspension of refueling.

・"Fraudulent acts related to situations requiring attention"
The following situations that require caution should also be noted even after refueling permission has been granted.

(1) Use of open flames, (2) Cigarettes, (3) Lighters, (4) Matches, (5) Smoking implements, (6) Smoking, (7) Ignition, (8) Spark generation, (9) High temperature objects, ( 10) Flammable materials, (11) Combustible materials, (12) Explosive materials, (13) Dangerous materials, (14) Carrying cans, (15) Kerosene tanks, (16) Carelessness of talking or staring at screens, (17) Inattention due to mobile phones, (18) Inattentiveness due to mobile devices, (19) Presence of children outside the vehicle, (20) Looking away while refueling, (21) Conversation while refueling, (22) Falls of refueling person, etc. , (23) Disputes among refueling personnel, (24) Disturbances in the refueling space, (25) Losing or forgetting to close the refueling cap, (26) Abnormal behavior such as being drunk, falling down, crouching, or hiding something, ( 27) Careless behavior, (28) Dangerous behavior Therefore, the image analysis system 100 continues to detect caution-required situations even after refueling is permitted.

When a caution-required situation is detected immediately before refueling, the image analysis system 100 temporarily changes refueling permission to refueling prohibition. In this state, the alarm 16a alerts the refueling person, and the monitoring personal computer 15 or the like notifies the supervisor of the situation requiring attention. After confirming that the caution-required situation has been resolved, the supervisor operates the monitoring computer 15 or the like to change refueling prohibited to refueling permitted.

Further, if a caution-required situation is detected during refueling, the image analysis system 100 temporarily changes refueling permission to refueling not possible, and performs control to temporarily stop refueling for safety. In this state, the alarm 16a alerts the refueling person, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the caution-required situation has been resolved, the supervisor operates the monitoring computer 15 or the like to change refueling prohibited to refueling permitted, and cancels the temporary suspension of refueling.

・"Fraudulent act of allowing unnecessary number of people to stay"
Unnecessary gathering of people just before refueling or during refueling can lead to unexpected situations. Therefore, the image analysis system 100 detects the number of people staying in the refueling space even after refueling is permitted.

If a "fraudulent act in which an unnecessary number of people stay" is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to refueling prohibition. In this state, the alarm device 16a alerts the resident, and the monitoring personal computer 15 or the like notifies the supervisor of the situation requiring attention. After confirming that the situation is safe, the supervisor operates the monitoring computer 15 or the like to change refueling from prohibited to refueling permitted.

Additionally, if "unauthorized behavior in which unnecessary people stay" is detected during refueling, the image analysis system 100 temporarily changes the refueling permission to prohibit refueling, and also controls the temporary suspension of refueling for safety. conduct. In this state, the alarm 16a alerts the resident, and the monitoring personal computer 15 or the like notifies the supervisor of any fraudulent activity. After confirming that the situation is safe, the supervisor operates the monitoring computer 15 or the like to change refueling from prohibited to refueling permitted, and cancels the temporary suspension of refueling.
[Effects of Example 1]

(1) In the first embodiment, a learning model is used in which fraudulent acts are machine-learned for a group of videos of a refueling space. This type of machine learning results in a learning model that reacts to fraudulent behavior that observers perceive as dangerous or wrong. Therefore, it becomes possible to detect fraudulent activities in the refueling space in addition to (or in place of) the supervisor.

(2) In the first embodiment, a learning model 200 is used that is subjected to machine learning to determine whether or not a vehicle is stopped at a refueling position with respect to a group of images of a refueling space. Therefore, in addition to (or in place of) the supervisor, it becomes possible to detect fraudulent activity in which the vehicle does not stop at the refueling position.

(3) In the first embodiment, a learning model 210 is used that is subjected to machine learning to determine whether or not a refueling person has taken action to remove static electricity with respect to a group of images of a refueling space. Therefore, in addition to (or in place of) the supervisor, it becomes possible to detect fraudulent acts in which the refueling person does not take action to eliminate static electricity.

(4) In the first embodiment, a learning model 220 is used that performs machine learning to determine whether or not the refueling nozzle is correctly inserted at the refueling destination for a group of images of the refueling space. Therefore, in addition to (or in place of) the supervisor, it becomes possible to detect fraudulent acts regarding the insertion of the refueling nozzle.

(5) In the first embodiment, a learning model 230 is used in which the number of people in a refueling space is machine learned for a group of videos of the refueling space. Therefore, it becomes possible to detect fraudulent activity in which the number of people in the refueling space exceeds a predetermined number by supplementing the supervisor (or in place of the supervisor).

(6) In the first embodiment, a learning model 240 is used that is subjected to machine learning to determine whether or not the refueling destination is a vehicle for a group of images of a refueling space. Therefore, it becomes possible to detect fraudulent acts of refueling vehicles other than vehicles in addition to (or in place of) the supervisor.

(7) In Example 1, for the video group of the refueling space, the use of open flames, cigarettes, lighters, matches, smoking implements, smoking, ignition, spark generation, high temperature objects, flammable objects, combustible objects, explosives, dangerous objects, and carrying Cans, kerosene tanks, talking or staring at screens, cell phones, mobile devices, children outside the vehicle, looking away, talking, falling, fighting, disturbances, losing or forgetting to close the fuel cap, being drunk, falling down, crouching, hiding something, etc. A learning model 250 is used which is subjected to machine learning regarding at least one caution-required situation in the group consisting of abnormal behavior, careless behavior, and dangerous behavior. Therefore, it becomes possible to detect these caution-required situations as fraudulent acts in addition to (or in place of) the monitor.

(8) In the first embodiment, the vehicle number of the vehicle to be refueled is detected. Since the car number is different for all vehicles, individual vehicle identification becomes possible. By comparing the vehicle number with the past refueling history 14, it can be determined that refueling is not possible if refueling is the second or subsequent time and refueling of the same type of oil as in the past is not selected.

(9) Furthermore, in the first embodiment, the vehicle number of the vehicle to be refueled is inquired of a registration database in which oil types (or vehicle types whose oil types can be identified) are registered in association with vehicle numbers. If the vehicle is already registered, it can be determined that refueling is not possible if an oil type that matches the inquiry result is not selected.

(10) In Embodiment 1, if the car number to be refueled matches the car number of the rental car (including car sharing) (in Japan, the usage code is "wa" or "re"), there is a "possibility of incorrect oil type". Yes. Generally, a rental car is used temporarily by a refueling person, so it is easy for the refueling person to make a mistake in the type of oil. Therefore, by detecting rental cars and alerting them, it is possible to prevent "wrong oil type" with a high probability.

(11) In the first embodiment, it becomes possible to centrally operate the detection of fraud and the determination of whether or not to refuel on the screen of the monitoring computer 15 in the office (see FIG. 19).

(12) In the first embodiment, the same unified operation as that of the monitoring personal computer 15 is possible via the handheld terminal 17. Therefore, even if the supervisor leaves the office to go to the refueling lane, it is possible to use the handheld terminal 17 to centrally detect fraudulent activity and determine whether or not to refuel.

Next, Example 2 will be described.
Conventionally, companies that operate a predetermined number of vehicles, such as taxi companies and transportation companies, sometimes set up a dedicated refueling station in a corner of a company-owned parking lot. At such dedicated refueling stations, only vehicles such as company cars and company cars (hereinafter referred to as "qualified cars") are refueled at the expense of the company.

Usually, at a private refueling station, the driver of a qualified vehicle is lent an ID card indicating refueling qualifications to determine whether the vehicle is qualified or not. However, it has been difficult to prevent unauthorized vehicles from refueling for free by fraudulently using ID cards (hereinafter referred to as "fraudulent refueling without qualification").

[Explanation of configuration of Example 2]
FIG. 21 is a diagram showing the main parts of an image analysis system 500 suitable for monitoring refueling at a dedicated refueling station.

In the figure, an image analysis system 500 and a refueling device 510 are installed at the dedicated gas station GS2.
The image analysis system 500 includes an integrated camera unit 520, a detection section 530, a determination section 540, a registration section 550 in which the vehicle number of a qualified vehicle is registered in advance, and a monitoring personal computer 560.
Here, the detection unit 530 and the determination unit 540 are realized by the functions of the image processing server 570.

The integrated camera unit 520 includes a high-sensitivity camera 521 for photographing license plates, an infrared illumination device 522 for illuminating dark areas, a heater glass device 523 for removing fog, raindrops, and internal condensation, and a parking lot entrance gate. The vehicle is configured to include an installation pillar 524 for installing a camera in the parking lot or refueling space, and an alarm 525 for warning of fraudulent activity.

In other respects, the image analysis system 500 of the second embodiment has the same configuration as the image analysis system 100 of the first embodiment (see FIGS. 1 and 2).

[Explanation of operation of Example 2]
FIG. 22 is a diagram illustrating the operation of the image analysis system 500.
The steps will be explained below in accordance with the step numbers shown in the figure.

Step S111: The detection unit 530 acquires image information of the license plate of the vehicle to be refueled from the high-sensitivity camera 521.

Step S112: The detection unit 530 detects the vehicle number from the video information of the license plate in the same manner as in the first embodiment.

Step S113: The determination unit 540 refers the detected car number to the registration unit 550 that has registered the car numbers of qualified cars that are eligible for refueling.

Step S114: If the vehicle number is not registered in the registration unit 550, the determining unit 540 determines that unauthorized refueling is a fraudulent act, and moves the operation to step S115. On the other hand, if the vehicle number is registered in the registration unit 550, it is determined that the vehicle is qualified to refuel at the dedicated gas station GS2, and the operation proceeds to step S11 of the first embodiment (FIG. 12). From this point on, the same operations as in the first embodiment are performed.

Step S115: The determining unit 540 determines that unqualified refueling is a fraudulent act, outputs a refueling prohibition signal to the refueling device 510 to stop refueling, and prevents unqualified refueling.

Step S116: The determination unit 540 transmits information to the monitoring personal computer 560 and the alarm device 525 about the illegal act of refueling without qualification. The monitoring personal computer 560 outputs a warning of unqualified refueling to the supervisor to call attention to it. The alarm device 525 issues an alarm to the refueling person and explains the circumstances of unqualified refueling using an audio guide. The monitoring computer 560 records and saves the image of the license plate and the vehicle number as an attempt to refuel without qualification.

[Effects of Example 2]
In addition to the effects of Example 1, the second embodiment also provides the following effects.
(1) In the second embodiment, the vehicle number of the vehicle to be refueled is detected. Since the car number is different for all vehicles, individual vehicle identification becomes possible. By inquiring the vehicle number to the registration unit 550, it becomes possible to determine whether the vehicle is refueling without qualification.

<Supplementary information on the embodiment>
In the first and second embodiments described above, the vehicle number is used to identify the vehicle. However, the present invention is not limited thereto. Vehicles can also be identified using characteristics of the vehicle body instead of the vehicle number.
For example, in the case of a commercial vehicle such as a taxi, it is possible to identify the vehicle of the taxi company (whether or not it has refueling qualifications, etc.) by the paint pattern on the vehicle body or the rooftop light.
In addition, if the vehicle is a transportation company, it is possible to identify the transportation company's vehicle (such as whether or not it has refueling qualifications) by the logo mark, company name, and paint pattern on the vehicle body.
Furthermore, even if it is a regular car, there may be differences in the position and arrangement of the vehicle inspection stickers on the car body, the combination of the car body and tires and wheels , codes such as QR codes attached to the car body, and inherent damage to the car body. Individual vehicle identification is possible based on features or a combination of two or more of them.

Furthermore, in the first and second embodiments described above, the vehicle number is read from the video information. However, the present invention is not limited thereto. The vehicle number may be acquired not only by reading from video information but also by information communication through a wireless tag or wireless device mounted on the vehicle.

Furthermore, in the first and second embodiments described above, the supervisor makes the final decision on permission to refuel. However, the present invention is not limited thereto. If it does not violate laws and regulations, the determination unit may directly control whether or not the refueling device can refuel.

Furthermore, in the first and second embodiments described above, convolutional neural networks (see FIG. 3) were used as the learning models 200 to 260. However, the present invention is not limited thereto. For example, such learning models include decision tree learning, association rule learning, neural networks, genetic programming, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, principal component analysis, and extreme Learning models based on learning machines and/or other machine learning techniques may be employed.

Furthermore, in the first and second embodiments described above, the refueling of gasoline and light oil was explained. However, the refueling of the present invention is not limited to this. For example, in the case of a device that refuels kerosene, refueling a gasoline or diesel vehicle may be detected as a fraudulent act.

Furthermore, in the first and second embodiments described above, the process is carried out on a gas station GS or dedicated gas station GS2 basis. However, the present invention is not limited thereto. By installing part or all of the image analysis system on a cloud network and connecting it to individual refueling devices, it is possible to implement inventions in wide areas such as regions or gasoline supply companies. become.

Furthermore, in the first and second embodiments described above, the oil supply history is used to discover unsuitable oil types. However, the present invention is not limited thereto. The image analysis system may extract habitual fraudsters (habitual vehicles) from past fraudulent acts accumulated as refueling history and create a caution list. The image analysis system makes it possible to preventively detect addicts (addictive vehicles) before they commit fraudulent acts by referring to the store users (entering vehicles) on this caution list. As a result, the image analysis system can prompt the supervisor to warn the addict in advance. In addition, the image analysis system can prioritize detection of fraudulent acts against addicts.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described.
Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
Furthermore, it is possible to add, delete, or replace some of the configurations of each embodiment with other configurations.

GS...Gas station, 10...Refueling device, 11...Video acquisition section, 12...Detection section, 13...Judgment section, 14...Refueling history, 15...Monitoring computer, 16...Alarm device, 16a...Alarm device, 17...Handheld terminal , 18...Wireless LAN, 19...Interface unit, 20...Refueling control device, 21...Network switch, 22...Image processing server, 31...Preprocessing unit, 32...Input layer, 33...Convolution layer/activation function, 34... Pooling layer, 35... Neural network, 100... Image analysis system, 200... Learning model, 210... Learning model, 220... Learning model, 230... Learning model, 240... Learning model, 250... Learning model, 260... Learning model, 301 ...Live display area, 302...Information field, 303...Analysis display area, 304...Confirmation required button, 305...Operation area, GS2...Dedicated refueling station, 500...Image analysis system, 510...Refueling device, 520...Integrated camera unit , 521... High sensitivity camera, 522... Infrared illumination equipment, 523... Heater glass equipment, 524... Installation pillar, 525... Alarm, 530... Detection unit, 540... Judgment unit, 550... Registration unit, 560... Monitoring computer, 570 ...Image processing server

Claims (15)

an image acquisition unit that acquires an image of the refueling space;
a detection unit that detects the fraudulent activity in the video acquired by the video acquisition unit using a learning model that machine-learns fraudulent behavior in the video group of the refueling space;
a determination unit that generates information regarding refueling availability based on the detection result of the fraudulent act;
Equipped with
The detection section includes any of the following: the paint pattern of the car body, roof lights, logo marks, company names, differences in the position and arrangement of vehicle inspection stickers, combination of car body and tires, wheels attached to the car body, and damage to the car body. An image analysis system that detects vehicle characteristics. The image analysis system according to claim 1,
The detection unit is
Using a learning model that performs machine learning to determine whether or not the vehicle is stopped at the refueling position for the video group of the refueling space, fraud in which the vehicle does not stop at the refueling position is detected in the video acquired by the video acquisition unit. An image analysis system characterized by: The image analysis system according to any one of claims 1 to 2,
The detection unit is
Using a learning model that performs machine learning to determine whether or not a refueling person has taken action to remove static electricity with respect to the group of images of the refueling space, it is possible to detect fraudulent acts in which the refueling person does not take action to remove static electricity in the images acquired by the image acquisition unit. An image analysis system that detects The image analysis system according to any one of claims 1 to 3,
The detection unit is
Using a learning model that performs machine learning to determine whether or not the refueling nozzle was correctly inserted at the refueling destination for the video group of the refueling space, the image acquisition unit uses a learning model to determine whether or not the refueling nozzle has been correctly inserted into the refueling space. An image analysis system that detects. The image analysis system according to any one of claims 1 to 4,
The detection unit is
It is characterized by detecting fraudulent acts in which the number of people in the refueling space exceeds a predetermined number in the video acquired by the video acquisition unit using a learning model that machine-learns the number of people in the refueling space with respect to the video group of the refueling space. image analysis system. The image analysis system according to any one of claims 1 to 5,
The detection unit is
The system is characterized by detecting a fraudulent act of refueling a vehicle other than a vehicle in the video acquired by the video acquisition unit, using a learning model that performs machine learning to determine whether or not the refueling destination is a vehicle in the video group of the refueling space. Image analysis system. The image analysis system according to any one of claims 1 to 6,
The detection unit is
Regarding the video group of the refueling space,
Use of open flames, cigarettes, lighters, matches, smoking materials, smoking, ignition, spark generation, high temperature objects, flammable objects, combustible objects, explosives, dangerous objects, carrying cans, kerosene tanks, talking or staring at screens, mobile phones, mobile devices , children outside the car, looking away, conversations, falling, fighting, disturbances, losing or forgetting to close the fuel cap, abnormal behavior, careless behavior, and dangerous behavior. An image analysis system characterized in that, using a learning model, the caution-required situation is detected as a fraudulent act in the video acquired by the video acquisition unit. The image analysis system according to any one of claims 1 to 7,
The detection unit is
Detecting the vehicle number or the above characteristics of the vehicle to be refueled,
The determination unit includes:
The image is characterized in that the vehicle number or the characteristics detected by the detection unit are checked against the refueling history, and if refueling is the second or later refueling and refueling of the same type of oil as in the past is not selected, refueling is determined to be impossible. Analysis system. The image analysis system according to any one of claims 1 to 8,
The detection unit is
Detecting the vehicle number or the above characteristics of the vehicle to be refueled,
The determination unit includes:
Information on the oil type or vehicle type is acquired by querying the vehicle number or the characteristic read by the detection unit against a registered database in which oil type or vehicle type is registered in association with the vehicle number or characteristic, and the obtained oil is checked. An image analysis system that determines that refueling is not possible if the type of oil that matches the model or type of vehicle is not selected. The image analysis system according to any one of claims 1 to 9,
The detection unit is
Detects the vehicle number of the vehicle to be refueled,
The determination unit includes:
An image analysis system characterized in that when the car number detected by the detection unit matches the car number of a rental car, it is determined that there is a possibility that the oil type is incorrect. a registration unit for registering vehicle numbers or characteristics of vehicles eligible for refueling;
a detection unit that detects a vehicle number or characteristics from a vehicle to be refueled;
a determination unit that determines that unauthorized refueling is a fraudulent act if the vehicle number or characteristics of the vehicle detected by the detection unit is not registered in the registration unit;
Equipped with
The detection section includes any of the following: the paint pattern of the car body, roof lights, logo marks, company names, differences in the position and arrangement of vehicle inspection stickers, combination of car body and tires, wheels attached to the car body, and damage to the car body. An image analysis system that detects vehicle characteristics. an image acquisition step of acquiring an image of the refueling space;
a detection step of detecting the fraudulent act in the video acquired by the video acquisition step using a learning model that is machine-trained to detect fraudulent behavior in the video group of the refueling space;
a determination step of generating information regarding refueling availability based on the detection result of the fraudulent act;
Equipped with
The detection step includes any of the following: paint pattern on the car body, roof lights, logo mark, company name, position or arrangement of vehicle inspection stickers, combination of car body and tires, wheels attached to the car body, damage to the car body. An image analysis method characterized by detecting vehicle characteristics. a registration step for registering the vehicle number or characteristics of a vehicle eligible for refueling;
a detection step of detecting a vehicle number or characteristics from a vehicle to be refueled;
If the vehicle number or characteristics of the vehicle detected by the detection step is not registered in the registration step, a determination step of determining that unauthorized refueling is a fraudulent act;
Equipped with
The detection step includes any of the following: paint pattern on the car body, roof lights, logo mark, company name, position or arrangement of vehicle inspection stickers, combination of car body and tires, wheels attached to the car body, damage to the car body. An image analysis method characterized by detecting vehicle characteristics. The image analysis system according to any one of claims 1 to 11,
The image analysis system is characterized in that the detection unit detects the characteristics of the vehicle at a refueling station exclusively for qualified persons. In the image analysis method according to any one of claims 12 to 13,
The image analysis method is characterized in that, in the detection step , the characteristics of the vehicle are detected at a refueling station exclusively for qualified persons.

Images