JP7737946B2 - FRAUDULENT ACTIVITY ESTIMATION DEVICE, ITS CONTROL PROGRAM, AND FRAUDULENT ACTIVITY ESTIMATION METHOD - Google Patents

Description

Embodiments of the present invention relate to a fraud estimation device, a control program therefor, and a fraud estimation method.

In recent years, self-service point-of-sale (POS) terminals have been gaining attention in supermarkets and other retail stores as a way to reduce labor costs and prevent the spread of infectious diseases. Self-service POS terminals are fully self-service payment terminals that allow customers to perform all operations themselves, from registering purchased items to paying. In many cases, surveillance cameras are installed to monitor customers operating such self-service POS terminals.

In a store system equipped with such self-service POS terminals and surveillance cameras, customer behavior is recognized from the movement of the hand holding a purchased item, for example, based on image data captured by the surveillance camera. The customer's behavior is then used to determine whether or not fraudulent activity has occurred, and if fraudulent activity is determined, measures such as notifying a store employee or issuing a warning on the self-service POS terminal are taken. However, if a hand enters the surveillance camera's blind spot or if the hand is obscured by a shadow caused by lighting conditions, the customer's behavior will not be recognized correctly. This unreliable recognition result could result in a customer's normal behavior being mistaken for fraud. There is also a possibility that a customer's fraudulent activity could be mistaken for normal behavior. For this reason, there is a desire to avoid situations such as trouble with customers and a decline in trust in the store.

Japanese Patent Application Laid-Open No. 2018-41255

The problem that embodiments of the present invention aim to solve is to provide a fraud estimation device, its control program, and a fraud estimation method that can take appropriate action depending on the reliability of a customer's fraudulent activity.

In one embodiment, the fraudulent activity estimation device comprises a first acquisition means, a behavior recognition means, a second acquisition means, a fraudulent activity estimation means, and an output means. The first acquisition means acquires operation information from a purchaser's product registration operation on a payment terminal. The behavior recognition means recognizes the behavior of the purchaser performing the product registration operation. The second acquisition means acquires a reliability level , which is the degree of reliability of the recognition result that recognizes the purchaser's behavior. The fraudulent activity estimation means estimates the purchaser's fraudulent activity based on the operation information and the reliability level. If the output means estimates the purchaser's fraudulent activity, it outputs result information to deter the purchaser's fraudulent activity based on the operation information and the reliability level.

FIG. 1 is a diagram showing the system configuration of a store in which a self-checkout POS terminal is installed. FIG. 2 is a diagram for explaining the positional relationship between the self-checkout POS terminal and the camera. FIG. 3 is a schematic diagram showing an example of a monitoring image displayed on the display of the attendant terminal. FIG. 4 is a block diagram showing the main circuit configuration of the fraudulent activity inference device. FIG. 5 is a schematic diagram showing an example of the data structure of the behavior file. FIG. 6 is a schematic diagram illustrating an example of a time-series buffer. FIG. 7 is a schematic diagram showing an example of the output table. FIG. 8 is a flowchart for explaining the functions of the behavior recognition unit and the second acquisition unit. FIG. 9 is a flowchart for explaining the functions of the operation recognition unit and the first acquisition unit. FIG. 10 is a flowchart for explaining the function of the fraud estimation unit. FIG. 11 is a flowchart for explaining the function of the output unit. FIG. 12 is a schematic diagram for explaining another embodiment related to behavior recognition.

An embodiment will be described below with reference to the drawings.
FIG. 1 is a system configuration diagram of a store in which a self-checkout POS terminal 11 has been installed. The system includes a self-checkout POS system 100 and a fraudulent activity estimation system 200. The self-checkout POS system 100 includes multiple self-checkout POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The multiple self-checkout POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a local area network (LAN). The LAN may be a wired LAN or a wireless LAN.

The self-service POS terminal 11 is a fully self-service payment terminal that allows customers to perform all operations themselves, from registering purchased items to paying. A customer may also be referred to as a purchaser, consumer, or customer. The POS server 12 is a server computer that centrally controls the operation of each self-service POS terminal 11. The display control device 13 is a controller that generates a monitoring image SC (see Figure 3) for each self-service POS terminal 11 based on the data signals output from each self-service POS terminal 11 and displays it on the display device of the attendant terminal 14. The attendant terminal 14 is a terminal used by store clerks known as attendants to monitor the status of each self-service POS terminal 11. The attendant terminal 14 is equipped with a display such as an LCD display or an organic EL display. The attendant terminal 14 divides the display screen into multiple sections, and displays a different monitoring image SC for the attendant terminal 14 in each section. The attendant terminal 14 is an example of a store clerk terminal. This self-service POS system 100 can be a conventional, well-known system.

The fraudulent activity estimation system 200 includes multiple cameras 21 and a fraudulent activity estimation device 22. Each of the multiple cameras 21 corresponds one-to-one to a corresponding one of the multiple self-service POS terminals 11. Each camera 21 is used to capture an image of a customer operating the corresponding self-service POS terminal 11.

The fraudulent activity estimation device 22 has the functions of a behavior recognition unit 221, an operation recognition unit 222, a first acquisition unit 223, a second acquisition unit 224, a fraud estimation unit 225, and an output unit 226. The behavior recognition unit 221 has a function of recognizing the behavior of a customer performing a product registration operation on the self-service POS terminal 11 based on the image data output from each camera 21. The behavior recognition unit 221 can also be referred to as a behavior recognition means. The operation recognition unit 222 has a function of recognizing a product registration operation by a customer on the self-service POS terminal 11 based on the data of the surveillance image SC output from the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be referred to as an operation recognition means. The first acquisition unit 223 has a function of acquiring operation information from the recognized product registration operation. The first acquisition unit 223 can also be referred to as a first acquisition means. The second acquisition unit 224 has a function of acquiring the reliability of the recognition result based on the recognition result by the behavior recognition unit 221. The second acquisition unit 224 can also be referred to as second acquisition means. The fraud estimation unit 225 has a function that estimates fraudulent behavior by a customer based on the operation information from the first acquisition unit 223 and the reliability from the second acquisition unit 224. The fraud estimation unit 225 can also be referred to as fraud estimation means. The output unit 226 has a function that outputs estimated result information. The output unit 226 can also be referred to as output means.

Figure 2 is a diagram illustrating the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external configuration of the self-service POS terminal 11 will be described.

The self-service POS terminal 11 comprises a main body 40 placed on the floor and a bagging stand 50 placed next to the main body 40. A touch panel 41 is attached to the top of the main body 40. The touch panel 41 is composed of a display and a touch sensor. The touch panel 41 is an example of a display unit. The display is a device for displaying various screens to the operator operating the self-service POS terminal 11. The touch sensor is a device for detecting touch inputs made by the operator on the screen. In the case of the self-service POS terminal 11, the operator is usually a customer.

The main body 40 has a basket stand 60 in the center of its side opposite the side where the bagging stand 50 is installed. The basket stand 60 is for customers coming from the sales floor to place baskets containing purchased items. Customers work by standing on the front side of the main body 40 in Figure 2 so that they can see the screen of the touch panel 41. Therefore, from the customer's perspective, the basket stand 60 is on the right side of the main body 40, and the bagging stand 50 is on the left side. In this embodiment, the side where the customer stands is the front of the main body 40, the side where the bagging stand 50 is installed is the left side of the main body 40, and the side where the basket stand 60 is installed is the right side of the main body 40.

The main body 40 is equipped with a scanner, card reader, receipt printer, cash processing machine, etc. The front of the main body 40 is formed with a scanner reading window 42, card insertion slot 43, receipt issuing slot 44, coin slot 45, coin dispensing slot 46, bill insertion slot 47, and bill dispensing slot 48. A communication cable extends from the right side of the main body 40 to the outside, and a handheld scanner 61 is connected to the tip of this communication cable. Although not shown, the main body 40 is also equipped with a reader/writer for electronic money media.

A display pole 64 is attached to the top surface of the main body 40. The display pole 64 has a light-emitting unit 65 at its tip. The light-emitting unit 65 selectively emits light, for example, blue or red. The display pole 64 displays the status of the self-service POS terminal 11, such as standby, operating, calling, error, or fraudulent activity, depending on the color of the light emitted by the light-emitting unit 65. The display pole 64 may also display the status of the self-service POS terminal 11 by flashing the light-emitting unit 65.

The bagging table 50 has a bag holder 52 attached to its top. The bag holder 52 has a pair of holding arms 53 that hold plastic bags provided by the store or shopping bags brought by customers, known as "my bags."

Next, the positional relationship between the self-checkout POS terminal 11 and the camera 21 will be described.
As shown in FIG. 2, the camera 21 is installed in a position where it can photograph a customer standing in front of the self-service POS terminal 11 and facing the main body 40, bagging table 50, basket table 60, and other components from above.

A customer standing in front of the self-service POS terminal 11 first places the basket containing the purchased items on the basket stand 60 on the right side, and then places a shopping bag or personal shopping bag on the holding arm 53 on the left side. Next, the customer follows the guidance displayed on the touch panel 41 and operates the touch panel 41 to declare that they want to begin using the self-service POS terminal 11.

The customer then picks up each item from the basket placed on the basket stand 60. If the item has a barcode, the customer registers the item by holding the barcode over the reading window 42 and having it read by the scanner. If the item does not have a barcode, the customer registers the item by operating the touch panel 41 to select the item from a list of items without barcodes. After registering the items, the customer places them in a shopping bag, their own bag, or similar.

After registering all of their purchases, the customer operates the touch panel 41 to select their payment method. For example, if they select cash payment, they insert bills or coins into the bill slot 47 or coin slot 45, and remove the change from the bill dispensing outlet 48 or coin dispensing outlet 46. For example, if they select electronic money payment, they hold the electronic money medium over the reader/writer. For example, if they select credit card payment, they insert their credit card into the card insertion slot 43. Once payment is complete, the customer receives the receipt issued from the receipt issuing slot 44 and leaves the store with the shopping bag or their own bag removed from the holding arm 53.

In other words, the camera 21 is installed in a position that allows it to capture the hand movements of a customer acting as described above in front of the self-service POS terminal 11.

Figure 3 is a schematic diagram showing an example of a monitoring image SC displayed on the display of the attendant terminal 14. As mentioned above, the display of the attendant terminal 14 displays separate monitoring images SC for each of the multiple self-service POS terminals 11. Figure 3 shows an example of a monitoring image SC for one of these self-service POS terminals 11. The configuration of the monitoring images SC for the other self-service POS terminals 11 is similar, so a description of them will be omitted here.

As shown in Figure 3, the monitoring image SC includes a register number column 71, a terminal status column 72, an error information column 73, a declaration information column 74, a details column 75, and a total column 76.

The register number column 71 is a column for displaying the register number. The register number is a series of numbers assigned to each self-checkout POS terminal 11 so that each terminal can be individually identified. The register number is identification information used to identify each self-checkout POS terminal 11.

The terminal status column 72 is a column for displaying the operating status of the self-service POS terminal 11. For example, the terminal status column 72 displays one of the following operating statuses: "Waiting," "Starting use," "Registering," "Payment started," and "Payment in progress."

"Waiting" is the state from when the previous customer has finished paying until the next customer is informed that they can begin using the service. When the self-service POS terminal 11 is in the "waiting" state, an initial image is displayed on the touch panel 41. The initial image is an image that includes touch buttons that allow the customer to select whether to use the plastic bag provided by the store or their own bag, for example.

"Start of use" is the state in which a customer standing in front of the self-service POS terminal 11 declares that they want to start using the terminal to make a payment. The customer selects whether to use a plastic bag or their own bag on the initial image. This selection operation declares that they want to start using the terminal. In response to this selection operation, the operating state of the self-service POS terminal 11 changes to "Start of use."

"Registering" is the state in which the system is accepting the customer's registration of the purchased item. When the first purchased item is registered, the self-service POS terminal 11's operating status changes to "Registering." From then on, the self-service POS terminal 11's operating status remains "Registering" until the transition to payment is declared.

"Payment Start" is the state in which a customer who has finished registering the items they wish to purchase declares that they wish to proceed to payment. The "Checkout" soft key is displayed on the touch panel 41 of the self-service POS terminal 11 in the "Registering" state. When a customer has finished registering the items they wish to purchase, they touch the "Checkout" soft key. This operation declares that they wish to proceed to payment. In response to this operation, the operating state of the self-service POS terminal 11 changes to "Payment Start."

"Payment in progress" is the state in which payment processing, such as cash payment, electronic money payment, or credit card payment, is being carried out. For example, when a bill or coin is inserted into the bill slot 47 or coin slot 45, the operating status of the self-service POS terminal 11 changes to "payment in progress." Then, when the payment processing is completed, the operating status of the self-service POS terminal 11 returns to "standby."

The error information field 73 is a field for displaying error information that has occurred on the self-service POS terminal 11. Error information includes communication errors, receipt shortage errors, etc. The declaration information field 74 is a field for displaying the customer's declaration operation. For example, if a customer selects "My bag," "No bag needed" will be displayed, indicating that a plastic bag is not required.

The details column 75 is a column for displaying detailed information about the purchased items registered on the self-checkout POS terminal 11. Detailed information includes, for example, the product name, number of items, and price. The total column 76 is a column for displaying total information about the purchased items registered on the self-checkout POS terminal 11. Total information includes, for example, the total number of items, the total price, the amount deposited, and change.

Note that the configuration of the monitoring image SC is not limited to this. Columns displaying other items may be arranged. Furthermore, the items of text data displayed in Figure 3 are not limited to this. Text data of other items may also be displayed.

Figure 4 is a block diagram showing the main circuit configuration of the misconduct estimation device 22. The misconduct estimation device 22 includes a processor 81, a main memory 82, an auxiliary storage device 83, a clock 84, a speaker 85, a camera interface 86, a communication interface 87, and a system bus 88. The system bus 88 includes an address bus, a data bus, etc. The misconduct estimation device 22 constitutes a computer by connecting the processor 81 with the main memory 82, the auxiliary storage device 83, the clock 84, the speaker 85, the camera interface 86, and the communication interface 87 via the system bus 88.

The processor 81 corresponds to the central part of the computer. The processor 81 controls each part to realize the various functions of the fraudulent activity estimation device 22 in accordance with an operating system or application program. The processor 81 is, for example, a CPU (Central Processing Unit).

The main memory 82 corresponds to the main storage portion of the computer. The main memory 82 includes a non-volatile memory area and a volatile memory area. The main memory 82 stores an operating system or application programs in the non-volatile memory area. The main memory 82 stores data required for the processor 81 to execute processes to control each part in the volatile memory area. This type of data may also be stored in the non-volatile memory area. The main memory 82 uses the volatile memory area as a work area where data can be rewritten by the processor 81 as appropriate. The non-volatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

The auxiliary storage device 83 corresponds to the auxiliary memory portion of the computer. The auxiliary storage device 83 may be a well-known storage device such as an SSD (Solid State Drive), HDD (Hard Disc Drive), or EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), either singly or in combination. The auxiliary storage device 83 stores data used by the processor 81 when performing various processes, data generated by the processes performed by the processor 81, and the like. The auxiliary storage device 83 may also store application programs.

Application programs stored in main memory 82 or auxiliary storage device 83 include the control program described below. There are no particular limitations on the method for installing the control program into main memory 82 or auxiliary storage device 83. The control program can be recorded on a removable recording medium, or distributed via communication over a network and installed in main memory 82 or auxiliary storage device 83. The recording medium can be in any form, such as a CD-ROM or memory card, as long as it can store the program and is readable by the device.

The clock 84 functions as a time information source for the fraudulent activity estimation device 22. The processor 81 obtains the current date and time based on the time information measured by the clock 84.

The speaker 85 is an output device for outputting audio data. Audio data includes sounds, voices, etc.

The camera interface 86 is an interface for communicating with each camera 21. The photographed data output from each camera 21 is imported into the fraudulent activity estimation device 22 via the camera interface 86. The photographed data includes photographed video, photographed images, etc. of a customer operating the self-service POS terminal 11 corresponding to the camera 21.

The communication interface 87 is an interface for communicating data between the self-checkout POS terminal 11, POS server 12, display control device 13, etc. in accordance with a communication protocol. For example, image data output from the display control device 13 is imported into the fraudulent activity estimation device 22 via the communication interface 87. The image data is data of the surveillance image SC generated for each self-checkout POS terminal 11.

A misconduct estimation device 22 configured as described above uses part of the volatile memory area in the main memory 82 as an area for the behavior file 821 (see FIG. 5), the time series buffer 822 (see FIG. 6), and the output table 823 (see FIG. 7). The misconduct estimation device 22 then creates the behavior file 821, the time series buffer 822, and the output table 823 in this area.

Figure 5 is a schematic diagram showing an example of the data structure of the action file 821. As shown in Figure 5, the action file 821 is a data file that records, in association with each other, the time TM, action status AST, recognition rate RP, and frame image for each register number that identifies the self-service POS terminal 11. The time TM is the time when the action status AST is acquired. The action status AST represents the state in which the customer's action is recognized by the function of the action recognition unit 221 of the processor 81, which will be described later. In this embodiment, the customer's action is removal action and bagging action.

A take-out action is an action of taking a purchased item from a basket placed on the basket stand 60. For example, if the skeleton of one or both hands moves to the right side of the main body 40, and a movement such as lifting up a purchased item is detected, the processor 81 recognizes that a take-out action has occurred.

Bagging behavior is the behavior of placing registered purchased items into a shopping bag or personal bag on the bagging table 50. For example, if the skeleton of the hand that performed the registered behavior moves to the left side of the main body and a movement that places the purchased items into a shopping bag or personal bag is detected, the processor 81 recognizes that a bagging behavior has occurred.

The recognition rate RP is a numerical value calculated based on the recognition rate at which the processor 81 recognizes the customer's behavior included in the captured image as a pick-up behavior and the recognition rate at which the processor 81 recognizes the customer's behavior as a bagging behavior. The recognition rate RP is, for example, a percentage. The frame images are captured by the camera 21. The frame images are recorded in the behavior file 821 in the order in which they were captured. Note that the items displayed in Figure 5 are not limited to these. Other items may also be displayed. The content of the text data displayed in Figure 5 is an example.

Fig. 6 is a schematic diagram showing an example of the time-series buffer 822. As shown in Fig. 6, the time-series buffer 822 has areas for describing the start time STM, end time FTM, status ST or output code OC (see Fig. 7), and reliability CD in association with each other for each register number that identifies the self-checkout POS terminal 11.
The start time STM is the time TM at which the behavior status AST is earliest written in the behavior file 821. The start time STM is the time at which the operation status HST is acquired. The start time STM is the time at which the output code OC is acquired. The end time FTM is the time TM at which the behavior status AST is latest written in the behavior file 821.
The status ST includes an action status AST and an operation status HST. The operation status HST indicates a state in which a product registration operation by a customer to the self-checkout POS terminal 11 is recognized by the function of an operation recognition unit 222 of the processor 81 (described later).
The reliability CD is the reliability of the recognition result that the processor 81 recognizes as the customer's behavior. The reliability CD is a numerical value calculated based on the recognition rate RP. The reliability CD is, for example, a percentage. The reliability CD is an example of the reliability of the purchaser's behavior.

The time series buffer 822 stores the status ST or output code OC and reliability CD in order of earliest start time STM. In this embodiment, when an action status AST is stored as the status ST, the start time STM, end time FTM, and reliability CD are stored. When an operation status HST or output code OC is stored as the status ST, the start time STM is stored, and the end time FTM and reliability CD are stored with, for example, a NULL value. Note that the items displayed in Figure 6 are not limited to these. Other items may also be displayed.

Fig. 7 is a schematic diagram showing an example of the output table 823. As shown in Fig. 7, the output table 823 is a data table in which the output code OC, the threshold value of the reliability CD of the action status AST "11", the threshold value of the reliability CD of the action status AST "12", and the output data are described in association with each other.
The output code OC is unique output identification information identified by the reliability CD threshold of the action status AST "11", the reliability CD threshold of the action status AST "12", and the output data in the same row. The action status AST "11" indicates a pick-up action. The action status AST "12" indicates a bagging action.
The threshold for the reliability CD of the behavior status AST "11" and the threshold for the reliability CD of the behavior status AST "12" can be set arbitrarily. The threshold may be a fixed numerical value in the fraudulent activity estimation device 22, or may be changeable to a desired threshold value by the store. In this embodiment, the store sets the threshold for the reliability CD of the behavior status AST "11" and the threshold for the reliability CD of the behavior status AST "12" in advance. The threshold can also be said to be a numerical value that allows the store to output output data of the same row, a reference value for outputting output data, etc. The threshold is an example of a predetermined condition.
The output data is data that is output to the self-service POS terminal 11 or the attendant terminal 14 when the reliability CD threshold of the behavior status AST "11" and the reliability CD threshold of the behavior status AST "12" in the same row are satisfied. The output data is, for example, text data for the self-service POS terminal, text data for the attendant terminal, sound data, color data, etc.

The text data for the self-checkout POS terminal may be, for example, "There is a purchased item that was not registered." The text data for the self-checkout POS terminal may contain any content that notifies the customer of fraudulent activity. The text data for the self-checkout POS terminal is an example of result information. The text data for the self-checkout POS terminal is an example of information that notifies the customer of fraudulent activity.
The text data for the attendant terminal may be, for example, "Fraudulent activity has occurred at register No. X." The text data for the attendant terminal may contain content that notifies the attendant of a customer's fraudulent activity. The text data for the attendant terminal is an example of result information. The text data for the attendant terminal is an example of information that notifies the attendant of fraudulent activity.
The sound data may be, for example, a continuous sound or an intermittently repeated sound. The sound data may be, for example, a voice saying, "There is a purchase item that was not registered." The sound data may be any sound that notifies a customer or an attendant of the customer's fraudulent activity. The sound data is an example of result information. The sound data is an example of a sound that notifies of fraudulent activity.
The color data is, for example, data representing the emitted color. The color data may be any color that notifies the attendant of a customer's fraudulent behavior. The color data is an example of result information. In this embodiment, the store sets the output data in advance. The output data may include, for example, data regarding the output destination and data regarding the output method.

As shown in FIG. 7, for example, if the reliability CD of action status AST "11" is 95 or higher and the reliability CD of action status AST "12" is 85 or higher, text data and sound data for the self-service POS terminal corresponding to output codes OC "91" and "93" are output as output data. Depending on the setting of the reliability CD threshold for action status AST "11" and the reliability CD threshold for action status AST "12", one output data may be output, or a combination of multiple output data may be output. Note that the items displayed in FIG. 7 are not limited to these. Other items may also be displayed. The content of the text data displayed in FIG. 7 is an example.

The fraudulent activity inference device 22's functions as the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud inference unit 225, and output unit 226 are functions provided for each self-checkout POS terminal 11. The functions as the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud inference unit 225, and output unit 226 for other self-checkout POS terminals 11 are similar. The fraudulent activity inference device 22 realizes the functions as the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud inference unit 225, and output unit 226 by a processor 81 and a control program that controls the processor 81.

Figures 8 to 11 are flowcharts showing the main control procedures of the processor 81 in the fraudulent activity estimation device 22. Below, these flowcharts will be used to explain the main operations of the system in a store where a self-service POS terminal 11 has been installed. Note that the operations described below are just an example. There are no particular limitations on the procedures as long as similar results are obtained.

FIG. 8 is a flowchart for explaining the functions of the behavior recognition unit 221 and the second acquisition unit 224.
The processor 81 waits for a customer to be recognized in ACT 1. The camera 21 is installed in a position where it can capture an image from above of a customer standing in front of the self-service POS terminal 11. When the processor 81 detects from the image captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, it determines that the customer has been recognized.

When the processor 81 recognizes the customer, it determines YES in ACT 1 and proceeds to ACT 2. In ACT 2, the processor 81 obtains the register number of the self-service POS terminal 11. Each camera 21 corresponds one-to-one with each self-service POS terminal 11. Therefore, the processor 81 identifies the self-service POS terminal 11 from the identification information of the camera 21 capturing the image of the customer standing in front of the self-service POS terminal 11, and obtains the register number of that self-service POS terminal 11.

In ACT 3, the processor 81 begins acquiring frame images captured by the camera 21. Each time the processor 81 acquires a frame image, it saves the frame image in the behavior file 821.

In ACT 4, the processor 81 estimates the skeleton of a person, i.e., a customer, from the frame image captured by the camera 21. Skeleton estimation can be achieved even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the customer's take-out or bagging behavior from the hand movements obtained through the skeleton estimation and the temporal changes in the positional relationship between the hand and a region of interest (ROI) that changes with the movement, such as the main body 40 of the self-service POS terminal 11, the bagging table 50, or the basket table 60, and calculates a recognition rate RP for each. Specifically, the processor 81 calculates the recognition rate for take-out behavior and the recognition rate for bagging behavior from the hand movements. The processor 81 determines the highest recognition rate among these recognition rates as the recognition rate RP. For example, if the recognition rate for take-out behavior is 98 percent and the recognition rate for bagging behavior is 2 percent, the recognition rate RP is 98 percent. The processor 81 recognizes the take-out behavior for which a recognition rate RP of 98 percent is calculated as customer behavior. Note that the process for calculating the recognition rate RP based on hand movements is an existing process that is well known, so a detailed explanation will be omitted.

In ACT 5, the processor 81 checks whether a take-out action has been recognized based on the processing in ACT 4. If a take-out action has been recognized by the function of the action recognition unit 221, the processor 81 determines YES in ACT 5 and proceeds to ACT 6.

In ACT 6, the processor 81 sets the action status AST to "11". The processor 81 then proceeds to processing in ACT 12.

In ACT 12, the processor 81 obtains the current time TM measured by the clock 84. In ACT 13, the processor 81 associates the time TM, the action status AST of "11", and the recognition rate RP calculated in the processing of ACT 4 with the frame image saved in the processing of ACT 3 in the action file 821 to which the register number obtained in the processing of ACT 2 has been set. Then, the processor 81 returns to ACT 3.

If the removal action is not recognized, the processor 81 determines NO in ACT 5 and proceeds to ACT 7.

In ACT 7, the processor 81 checks whether the bagging behavior has been recognized based on the processing in ACT 4. If the bagging behavior has been recognized by the function of the behavior recognition unit 221, the processor 81 determines YES in ACT 7 and proceeds to ACT 8.

In ACT 8, the processor 81 checks whether "11" is written as the action status AST, along with the start time STM, end time FTM, and reliability CD, in the time series buffer 822 where the register number obtained in the processing of ACT 2 is set.

If "11" is not written as the action status AST in the time series buffer 822, the processor 81 determines NO in ACT 8 and proceeds to ACT 9.

In ACT 9, the processor 81 uses the function of the second acquisition unit 224 to refer to the action file 821 and calculates the reliability CD based on the recognition rate RP for which "11" is described as the action status AST. For example, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP for which "11" is described as the action status AST. The processor 81 sets this highest recognition rate RP as the reliability CD of the take-out action. If the reliability CD of the take-out action is high, this means that the reliability of the recognition result recognized by the processor 81 as a take-out action is high, i.e., the reliability that the customer performed the take-out action is high. If the reliability CD of the take-out action is low, this means that the reliability of the recognition result recognized by the processor 81 as a take-out action is low, i.e., the reliability that the customer performed the take-out action is low.

The processor 81 writes "11" as the action status AST in the time series buffer 822 as ACT10, along with the start time STM, end time FTM, and reliability CD. The start time STM is the time TM at which "11" is written earliest as the action status AST in the action file 821. The end time FTM is the time TM at which "11" is written latest as the action status AST in the action file 821. The processor 81 then proceeds to ACT11.

If "11" is written as the action status AST in the time series buffer 822, the processor 81 determines YES in ACT 8, skips the processing of ACT 9 and ACT 10, and proceeds to ACT 11.

The processor 81 sets the action status AST to "12" in ACT11.

In ACT 12, the processor 81 obtains the current time TM measured by the clock 84. In ACT 13, the processor 81 associates and records in the action file 821 the time TM, the action status AST of "12", and the recognition rate RP calculated in the processing of ACT 4, corresponding to the frame image saved in the processing of ACT 3. Then, the processor 81 returns to ACT 3.

If the bagging behavior is not recognized, the processor 81 determines NO in ACT 7 and proceeds to ACT 14.

The processor 81 checks whether "12" is written as the action status AST along with the start time STM, end time FTM, and reliability CD in the time series buffer 822 as ACT14.

If "12" is not written as the action status AST in the time series buffer 822, the processor 81 determines NO in ACT 14 and proceeds to ACT 15.

In ACT15, the processor 81 uses the function of the second acquisition unit 224 to refer to the behavior file 821 and calculates the reliability CD based on the recognition rate RP for which "12" is described as the behavior status AST. For example, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP for which "12" is described as the behavior status AST. The processor 81 sets this highest recognition rate RP as the reliability CD of the bagging behavior. If the reliability CD of the bagging behavior is high, this means that the reliability of the recognition result recognized by the processor 81 as the bagging behavior is high, i.e., the reliability that the customer performed the bagging behavior is high. If the reliability CD of the bagging behavior is low, this means that the reliability of the recognition result recognized by the processor 81 as the bagging behavior is low, i.e., the reliability that the customer performed the bagging behavior is low.

The processor 81 writes "12" as the behavior status AST, along with the start time STM, end time FTM, and reliability CD, in the time series buffer 822 as ACT16. The start time STM is the time TM at which "12" is written earliest as the behavior status AST in the behavior file 821. The end time FTM is the time TM at which "12" is written latest as the behavior status AST in the behavior file 821. With this, the processor 81 terminates its functions as the behavior recognition unit 221 and the second acquisition unit 224.

If "12" is written as the behavior status AST in the time series buffer 822, the processor 81 determines YES in ACT 14, skips the processing in ACT 15 and ACT 16, and terminates the functions of the behavior recognition unit 221 and the second acquisition unit 224.

Typically, a customer registers data on purchased items in the self-service POS terminal 11 by repeatedly performing the actions of picking up and bagging items in sequence. Therefore, the action status AST is written in chronological order in the time series buffer 822, in the order "11", "12".

After that, when the image captured by the camera 21 again detects that a person is standing in front of the self-service POS terminal 11, the processor 81 executes the processing of ACT 2 to ACT 16 in the same manner as described above.

FIG. 9 is a flowchart for explaining the functions of the operation recognition unit 222 and the first acquisition unit 223.
The processor 81 waits for the start of use to be declared for the self-checkout POS terminal 11 in ACT21. When the start of use is declared, "Start of use" is displayed in the terminal status column 72 of the monitoring image SC corresponding to that self-checkout POS terminal 11. The processor 81 checks whether it can recognize the words "Start of use" from the terminal status column 72 of the monitoring image SC acquired via the display control device 13. If the words "Start of use" can be recognized, the processor 81 recognizes that the start of use has been declared using the function of the operation recognition unit 222.

When the processor 81 recognizes that the start of use has been declared, it determines YES in ACT 21 and proceeds to ACT 22. In ACT 22, the processor 81 acquires the register number of the self-service POS terminal 11. The register number is displayed in the register number column 71 of the monitoring image SC. The processor 81 recognizes the characters of the register number from the register number column 71 of the monitoring image SC acquired via the display control device 13, and acquires those characters as the register number.

In ACT 23, the processor 81 sets the operation status HST to "21". The operation status HST "21" indicates a start-of-use operation. In ACT 24, the processor 81 acquires the current time TM measured by the clock 84. In ACT 25, the processor 81 associates the time TM as the start time STM with "21" as the operation status HST and writes them in the time series buffer 822, to which the register number acquired in the processing of ACT 22 has been set. The processor 81 also writes NULL values for the end time FTM and the reliability CD.

Therefore, when a customer standing in front of a self-service POS terminal 11 declares the start of use, "21" is first written as the operation status HST along with the time TM in the time series buffer 822 corresponding to that self-service POS terminal 11.

In ACT 26, the processor 81 begins recognizing operations on the self-service POS terminal 11. Specifically, the processor 81 recognizes, for example, product registration operations, payment initiation operations, etc., from the transition of information obtained by character recognition of the monitoring image SC acquired via the display control device 13.

For example, when detailed information such as the product name, quantity, and price of the purchased product is added to the details column 75, the processor 81 recognizes that a product registration operation has been performed. For example, when the display in the terminal status column 72 switches to "Payment started," the processor 81 recognizes that a payment start operation has been performed.

The processor 81 waits to recognize a product registration operation or a payment start operation in ACT 27 or ACT 28.

When the operation recognition unit 222 recognizes a product registration operation during the standby state of ACT 27 or ACT 28, the processor 81 determines YES in ACT 27 and proceeds to ACT 29.

In ACT 29, the processor 81 sets the operation status HST to "22" using the function of the first acquisition unit 223. The operation status HST "22" indicates a product registration operation. In ACT 30, the processor 81 uses the function of the first acquisition unit 223 to acquire the current time TM measured by the clock 84. In ACT 31, the processor 81 associates the time TM as the start time STM with "22" as the operation status HST and writes it in the time series buffer 822. The processor 81 writes NULL values for the end time FTM and the reliability CD. The operation status HST is an example of operation information. The current time TM is an example of operation information. Thereafter, the processor 81 returns to the standby state in ACT 27 or ACT 28.

When the operation recognition unit 222 recognizes a payment start operation during the standby state in ACT 27 or ACT 28, the processor 81 determines YES in ACT 28 and proceeds to ACT 32. In ACT 32, the processor 81 ends operation recognition for the self-checkout POS terminal 11.

In ACT 33, the processor 81 sets the operation status HST to "23". The operation status HST "23" indicates a payment start operation. In ACT 34, the processor 81 acquires the current time TM measured by the clock 84. In ACT 35, the processor 81 associates the time TM as the start time STM with "23" as the operation status HST and writes it in the time series buffer 822. The processor 81 also writes NULL values for the end time FTM and the reliability CD. With this, the processor 81 terminates its functions as the operation recognition unit 222 and the first acquisition unit 223.

Normally, a customer performs operations on the self-service POS terminal 11 in the following order: start use, product registration, and payment start. Therefore, the operation status HST is written in chronological order in the time series buffer 822: "21", "22", and "23".

After that, when the start operation of the self-service POS terminal 11 is detected again from the data of the monitoring image SC, the processor 81 executes the processing of ACT 22 to ACT 35 in the same manner as described above.

FIG. 10 is a flowchart for explaining the function of the fraud estimation unit 225.
In this embodiment, the processor 81 executes the process shown in FIG. 10 for each purchased item.
In ACT 41, the processor 81 checks whether or not the action status AST is "12", i.e., whether or not a bagging action is described in the time-series buffer 822. If "12" is described as the action status AST, the processor 81 determines "YES" in ACT 41 and proceeds to ACT 42.

In ACT42, the processor 81 checks whether "22" is written as the operation status HST, i.e., whether a product registration operation is written. Specifically, the processor 81 checks whether "22" is written as the operation status HST in association with the time TM as the start time STM, one time before the start time STM at which "12" is written as the action status AST. If "22" is written as the operation status HST, the processor 81 determines YES in ACT42 and terminates its function as the fraud estimation unit 225. Note that when "22" is written as the operation status HST, the time series buffer 822 will chronologically write the action status AST "11", operation status HST "22", and action status AST "12" in that order.

If "22" is not written as the operation status HST, the processor 81 determines NO in ACT 42 and proceeds to ACT 43. Note that if "22" is not written as the operation status HST, the time series buffer 822 will chronologically write the action status AST "11" and then the action status AST "12" in that order.

In ACT43, the processor 81 checks in the order of output codes OC whether the reliability CD of the retrieval action in the same row as the action status AST "11" written in the time series buffer 822, i.e., the reliability CD of the retrieval action, meets the reliability CD threshold of the action status AST "11" written in the output table 823. If the reliability CD of the retrieval action written in the time series buffer 822 does not meet the reliability CD threshold of the action status AST "11" written in the output table 823, the processor 81 determines NO in ACT43 and proceeds to ACT45. The processing of ACT45 will be described later.

If the reliability CD of the retrieval action described in the time series buffer 822 meets the reliability CD threshold of the action status AST "11" described in the output table 823, the processor 81 determines YES in ACT 43 and proceeds to ACT 44.

In ACT 44, the processor 81 checks whether the reliability CD of the packing action in the same row as the action status AST "12" written in the time series buffer 822, i.e., the reliability CD of the packing action, satisfies the reliability CD threshold of the action status AST "11" confirmed in the processing of ACT 43 and the reliability CD threshold of the action status AST "12" in the same row written in the output table 823. If the reliability CD of the packing action written in the time series buffer 822 does not satisfy the reliability CD threshold of the action status AST "12" written in the output table 823, the processor 81 determines NO in ACT 44 and proceeds to ACT 45.

In ACT 45, the processor 81 records the behavior file 821 and the time series buffer 822 as a log. For example, the processor 81 may store the behavior file 821 and the time series buffer 822 in part of a non-volatile memory area in the main memory 82. In this case, the main memory 82 is an example of a storage unit. The processor 81 then proceeds to ACT 49. The processing in ACT 49 will be described later.

If the reliability CD of the bagging action described in the time series buffer 822 meets the reliability CD threshold of the action status AST "12" described in the output table 823, the processor 81 determines YES in ACT 44 and proceeds to ACT 46.

In ACT 46, the processor 81 obtains the output code OC. In ACT 47, the processor 81 obtains the current time TM measured by the clock 84. In ACT 48, the processor 81 associates the time TM as the start time STM with the output code OC obtained in the processing of ACT 46 and writes them in the time series buffer 822. The processor 81 also writes NULL values for the end time FTM and the reliability CD.

In ACT 49, the processor 81 checks whether the reliability CD of the take-out action and the packing action has been compared with the threshold value for all output codes OC in the output table 823. If the reliability CD of the take-out action and the packing action has not been compared with the threshold value for all output codes OC, the processor 81 determines NO in ACT 49 and returns to ACT 43. Thereafter, the processor 81 executes the processes in ACT 43 to ACT 48 in the same manner as described above.

If the reliability of the picking and packing actions has been compared with the threshold for all output codes OC, the processor 81 determines YES in ACT 49 and terminates its function as the fraud estimation unit 225.

In this way, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the reliability CD thresholds of the action status AST "11" and the action status AST "12" written in the output table 823, respectively, the processor 81 infers that there is a high degree of reliability that the customer performed the bagging action of the product that they picked up but did not register. In other words, the processor 81 infers that the customer's action is fraudulent using the function of the fraud inference unit 225. In this case, the processor 81 writes in the time series buffer 822 an output code OC that satisfies the reliability CD thresholds of the action status AST "11" and the reliability CD thresholds of the action status AST "12".

If "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the removal action written in the time series buffer 822 does not meet the reliability CD threshold of the action status AST "11" written in the output table 823, and/or the reliability CD of the bagging action written in the time series buffer 822 does not meet the reliability CD threshold of the action status AST "12" written in the output table 823, the processor 81 infers that the reliability of the bagging action of the product that the customer removed but did not perform the product registration operation is low. In this case, the processor 81 records the action file 821 and the time series buffer 822 as a log.

FIG. 11 is a flowchart for explaining the function of the output unit 226.
In ACT 51, the processor 81 waits for the output code OC to be written in the time series buffer 822. When the output code OC is written in the time series buffer 822, the processor 81 determines NO in ACT 51 and proceeds to ACT 52. In ACT 52, the processor 81 checks whether the output code OC is "91".

If the output code OC is "91", the processor 81 determines YES in ACT 52 and proceeds to ACT 53. In ACT 53, the processor 81 uses the function of the output unit 226 to output the first output command from the communication interface 87 to the self-checkout POS terminal 11. Specifically, the processor 81 obtains the register number from the time-series buffer 822 in which "91" is written as the output code OC. The processor 81 references the output table 823 to obtain the output data on the same line as the output code OC "91", i.e., the text data for the self-checkout POS terminal. The first output command includes the register number and the text data for the self-checkout POS terminal.

The self-checkout POS terminal 11 displays the self-checkout POS terminal text data on the touch panel 41 of the self-checkout POS terminal 11 identified by the register number included in the first output command. The processor 81 then proceeds to ACT 54.

If the output code OC is not "91", the processor 81 determines NO in ACT 52 and proceeds to ACT 54. That is, after processing in ACT 53 or if the output code OC is not "91", the processor 81 checks whether the output code OC is "92" in ACT 54.

If the output code OC is "92", the processor 81 determines YES in ACT 54 and proceeds to ACT 55. In ACT 55, the processor 81 uses the function of the output unit 226 to output the second output command from the communication interface 87 to the attendant terminal 14 via the display control device 13. Specifically, the processor 81 obtains the register number from the time series buffer 822 in which "92" is written as the output code OC. The processor 81 references the output table 823 to obtain the output data on the same line as the output code OC "92", i.e., the text data for the attendant terminal. The second output command includes the register number and the text data for the attendant terminal.

The display control device 13 displays the text data for the attendant terminal on the monitoring image SC of the attendant terminal 14 identified by the register number included in the second output command. The processor 81 then proceeds to ACT 56.

If the output code OC is not "92", the processor 81 determines NO in ACT 54 and proceeds to ACT 56. That is, after processing in ACT 65 or if the output code OC is not "92", the processor 81 checks whether the output code OC is "93" in ACT 56.

If the output code OC is "93", the processor 81 determines YES in ACT 56 and proceeds to ACT 57. In ACT 57, the processor 81 uses the function of the output unit 226 to output a third output command from the communication interface 87 to the self-service POS terminal 11. Specifically, the processor 81 obtains the register number from the time series buffer 822 in which "93" is written as the output code OC. The processor 81 references the output table 823 and obtains the output data, i.e., sound data, in the same row as the output code OC "93". The third output command includes the register number and sound data.

The self-checkout POS terminal 11 outputs sound data to the speaker 85 of the self-checkout POS terminal 11 identified by the register number included in the third output command. The processor 81 then proceeds to ACT 68.

If the output code OC is not "93", the processor 81 determines NO in ACT 56 and proceeds to ACT 58. That is, after processing in ACT 57 or if the output code OC is not "93", the processor 81 checks whether the output code OC is "94" in ACT 58.

If the output code OC is "94", the processor 81 determines YES in ACT 58 and proceeds to ACT 59. In ACT 59, the processor 81 uses the function of the output unit 226 to output a fourth output command from the communication interface 87 to the self-service POS terminal 11. Specifically, the processor 81 obtains the register number from the time series buffer 822 in which "94" is written as the output code OC. The processor 81 references the output table 823 to obtain the output data, i.e., color data, on the same row as the output code OC "94". The fourth output command includes the register number and color data.

The self-checkout POS terminal 11 causes the light-emitting unit 65 of the self-checkout POS terminal 11 identified by the register number included in the fourth output command to emit light based on the color data. The light-emitting unit 65 may selectively emit a predetermined color or may blink. This completes the process of the processor 81 ending its function as the output unit 226.

If the output code OC is not "94", the processor 81 determines NO in ACT 58 and terminates its function as the output unit 226.

As described above in detail, when the processor 81 of the fraudulent activity estimation device 22 recognizes the removal and bagging actions of a customer performing a product registration operation on the self-service POS terminal 11 based on the frame image captured by the camera 21, it calculates a recognition rate RP. The processor 81 writes the time TM, action status AST, and recognition rate RP corresponding to the frame image in the action file 821. The processor 81 calculates the reliability CD of the removal and bagging actions based on the recognition rate RP written in the action file 821. The processor 81 writes the start time STM, end time FTM, action status AST, and reliability CD in the time series buffer 822. Furthermore, when the processor 81 recognizes a product registration operation based on the data of the surveillance image SC output from the display control device 13 to the attendant terminal 14, it writes the operation status HST and the current time TM as the start time STM as operation information in the time series buffer 822. The processor 81 outputs result information based on the operation information of the product registration operation and the reliability CD of the picking and bagging actions.

For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the respective thresholds for the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12" written in the output table 823, the processor 81 obtains the output code OC and writes it in the time series buffer 822 together with the current time TM as the start time STM. The processor 81 then outputs one or more output commands including output data corresponding to the output code OC in the output table 823 to the self-service POS terminal 11 or attendant terminal 14.

For example, when the processor 81 outputs a first output command including self-service POS terminal text data as output data to the self-service POS terminal 11, the self-service POS terminal text data notifying the user of fraudulent activity is displayed on the touch panel 41 of the self-service POS terminal 11. This alerts the customer to the fraudulent activity and allows the customer to learn the circumstances. As a result, fraudulent activity by customers at the self-service POS terminal 11 is deterred.

For example, when the processor 81 outputs a second output command including attendant terminal text data as output data to the attendant terminal 14, the attendant terminal text data notifying the attendant of fraudulent activity is displayed on the monitoring image SC of the attendant terminal 14. This allows the attendant to easily learn that fraudulent activity has occurred. The attendant can quickly warn the customer in question and hear the details from the customer. As a result, fraudulent activity by customers at the self-service POS terminal 11 is deterred.

For example, when the processor 81 outputs a third output command including sound data as output data to the self-checkout POS terminal 11, sound data informing the customer of fraudulent activity is output to the speaker 85 of the self-checkout POS terminal 11. This alerts the customer to the fact that fraudulent activity has occurred. An attendant who hears the sound or audio can quickly alert the customer in question and hear the details from the customer. As a result, fraudulent activity by customers against the self-checkout POS terminal 11 is deterred.

For example, when the processor 81 outputs a fourth output command including color data as output data to the self-checkout POS terminal 11, it causes the light-emitting unit 65 of the self-checkout POS terminal 11 to emit light based on the color data to alert the customer to fraudulent activity. This allows an attendant who sees the light-emitting unit 65 emitting or flashing light to quickly alert the customer and hear the details from them. As a result, fraudulent activity by customers against the self-checkout POS terminal 11 is deterred.

In this way, if it is estimated that the reliability of the bagging of items that a customer has removed but not registered is high, output data is output according to the behavior status AST "11", i.e., the threshold value for the reliability CD of the removal behavior, and the behavior status AST "12", i.e., the threshold value for the reliability CD of the bagging behavior. These thresholds and output data can be set in advance by the store. Therefore, the fraudulent activity reporting criteria, fraudulent activity reporting destinations, fraudulent activity reporting methods, etc. can be set according to the store's policies.

For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action written in the time series buffer 822 does not meet the reliability CD threshold of the action status AST "11" written in the output table 823, and/or the reliability CD of the bagging action written in the time series buffer 822 does not meet the reliability CD threshold of the action status AST "12" written in the output table 823, the processor 81 records the action file 821 and the time series buffer 822 as a log.

In this way, if it is estimated that the reliability of the bagging behavior of a customer who has taken out an item but not registered it is low, the behavior file 821 and time series buffer 822 are recorded as a log. Because no output data is output, a customer's normal behavior will not be mistakenly judged as fraudulent, and vice versa. This prevents problems from occurring with customers and a decline in trust in the store. Also, by analyzing the log, attendants can confirm whether a customer has engaged in fraudulent behavior. Furthermore, the behavior file 821 recorded as a log can be used in learning processes to improve the accuracy of skeleton estimation using frame images.

Therefore, according to this embodiment, appropriate action can be taken depending on the reliability of the customer's fraudulent activity.

The above describes embodiments of the fraud estimation device, its control program, and the fraud estimation method, but these embodiments are not limited to these.

In the above embodiment, an example was given of a case where one camera 21 is placed for one self-checkout POS terminal 11. A camera 21 does not necessarily have to be placed for each self-checkout POS terminal 11. For example, if one camera 21 can capture an image of a customer operating two adjacent self-checkout POS terminals 11, the number of cameras 21 may be reduced. In that case, however, in ACT 2 of Figure 8, the register number of the self-checkout POS terminal 11 closest to the position of the person shown in the captured image will be obtained. Also, for example, multiple cameras 21 may be placed for one self-checkout POS terminal 11. By doing so, blind spots of the self-checkout POS terminal 11 can be reduced and customer behavior can be recognized with greater accuracy.

In the above embodiment, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP in which "11" is written as the action status AST, and sets this as the reliability CD of the take-out action. For example, if there are multiple recognition rates RP in which "11" is written as the action status AST, the processor 81 may calculate their average value. The processor 81 may use this average value as the reliability CD of the take-out action. Similarly, for the bag-filling action, if there are multiple recognition rates RP in which "12" is written as the status ST, the processor 81 may calculate their average value. The processor 81 may use this average value as the reliability CD of the bag-filling action.

In the above embodiment, removal and bagging actions were given as examples of customer actions. For example, the processor 81 may also recognize registration actions, store exit actions, etc. Registration actions are actions in which data on purchased items removed from a basket is registered in the self-service POS terminal 11. For example, if the processor 81 detects a movement of the hand that performed the removal action, such as holding the purchased items over the reading window 42 in the center of the main body 40, the processor 81 recognizes that a registration action has occurred. Alternatively, if the processor 81 detects a movement of one hand that indicates operation of the touch panel 41 of the main body 40, the processor 81 recognizes that a registration action has occurred. Store exit actions are actions in which a customer who has completed payment moves away from the self-service POS terminal 11. For example, if the hand skeletal structure of a customer who has completed payment makes a movement that indicates removing a plastic bag or personal bag from the holding arm 53, and the customer can no longer be detected in the video captured by the camera 21, the processor 81 recognizes that a store exit action has occurred.

In the above embodiment, the output code OC "92" is text data for the attendant terminal.
For example, the output code OC "92" may be text data and sound data for the attendant terminal. In this case, the text data for the attendant terminal is displayed on the monitoring image SC of the attendant terminal 14, and sound data is output from the speaker of the attendant terminal 14. The sound data may be, for example, a continuous sound or an intermittently repeated sound. The sound data may be, for example, a voice saying, "Fraud has been committed at register No. X." The sound data may be any sound or voice that notifies the attendant of the customer's fraudulent activity. In this case, the text data and sound data for the attendant terminal are examples of information that notifies the attendant of fraudulent activity.
For example, in addition to output codes OC "91" to OC "94", output code OC "95" may be added to output table 823. The output data for output code OC "95" may be, for example, sound data for an attendant terminal. The sound data for an attendant terminal is an example of information that notifies of fraudulent activity. The sound data for an attendant terminal is an example of result information. In this case, the output data for output code OC "93" may be, for example, sound data for a self-service POS terminal. The sound data for a self-service POS terminal is an example of information that notifies of fraudulent activity. The sound data for a self-service POS terminal is an example of result information.

In the above embodiment, the processor 81 stores the behavior file 821 and time series buffer 822 in part of the non-volatile memory area of the main memory 82 to be recorded as a log. For example, the processor 81 may send the behavior file 821 and time series buffer 822 to the POS server 12 via the communication interface 87. The POS server 12 may store the behavior file 821 and time series buffer 822 in the main memory or an auxiliary storage device. The behavior file 821 and time series buffer 822 may be stored in the memory of an external device with which the fraudulent activity estimation device 22 can communicate, for example.

In the above embodiment, an example has been given of a case where the processor 81 records the action file 821 and the time series buffer 822 as a log when the operation status HST is not described as "22" in the time series buffer 822, and the reliability CD of the removal action described in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "11" described in the output table 823 and/or the reliability CD of the bagging action described in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "12" described in the output table 823.
For example, even if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the thresholds of the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12" written in the output table 823, respectively, the processor 81 may record the action file 821 and the time series buffer 822 as a log.

In the above embodiment, the output table 823 describes a threshold for the reliability CD of the action status AST "11" and a threshold for the reliability CD of the action status AST "12." For example, the output table 823 may be a data table that describes the output code OC, a first threshold for the reliability CD of the action status AST "11," a second threshold for the reliability CD of the action status AST "11," a first threshold for the reliability CD of the action status AST "12," a second threshold for the reliability CD of the action status AST "12," and output data in association with each other. The first and second thresholds may be set arbitrarily. The store may set the first and second thresholds for the reliability CD of the action status AST "11" and the first and second thresholds for the reliability CD of the action status AST "12" in advance.
For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the first threshold value of the reliability CD of the action status AST "11" written in the output table 823 and the first threshold value of the reliability CD of the action status AST "12", respectively, the processor 81 may obtain the output code OC and write it in the time series buffer 822 together with the current time TM as the start time STM.
For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action written in the time series buffer 822 does not satisfy the first threshold value of the reliability CD of the action status AST "11" written in the output table 823 but satisfies the second threshold value, and/or the reliability CD of the bagging action written in the time series buffer 822 does not satisfy the first threshold value of the reliability CDD of the action status AST "12" written in the output table 823 but satisfies the second threshold value, the processor 81 may record the action file 821 and the time series buffer 822 as a log.

In the above embodiment, the attendant terminal 14 may also incorporate the functionality of the display control device 13. In this case, the operation recognition unit 222 acquires monitoring image SC data from the attendant terminal 14 and recognizes customer operations on the self-service POS terminal 11. Alternatively, the operation recognition unit 222 may acquire data signals output from each self-service POS terminal 11 from the communications network 15 via a router, for example, and recognize customer operations on the self-service POS terminal 11 based on those data signals.

In the above embodiment, the misconduct inference device 22 has the functions of a behavior recognition unit 221, an operation recognition unit 222, a first acquisition unit 223, a second acquisition unit 224, a misconduct inference unit 225, and an output unit 226. The misconduct inference device 22 may also be realized by a system in which functions are distributed across multiple devices.

In the above embodiment, the customer's skeleton is estimated from the image captured by the camera 21, and the customer's picking and bagging actions are recognized based on the temporal changes in the correspondence between the hand positions and the regions of interest based on the skeleton estimation, such as the positions of the main body 40 of the self-service POS terminal 11, the bagging table 50, or the basket table 60. In this way, the behavior recognition method is not limited to the method based on skeleton estimation.
12, the processor 81 estimates the customer's left and right hands for each frame image captured by the camera 21, and infers a first bounding box 91 indicating the area of the left hand and a second bounding box 92 indicating the area of the right hand. The processor 81 also infers a third bounding box 93 indicating the area of the basket 90 placed on the basket stand 60.
The processor 81 selects the bounding box 91 or 92 indicating the area of the hand holding the product from the first bounding box 91 or the second bounding box 92. For example, in FIG. 12 , the second bounding box 92 is selected. The processor 81 tracks the movement of the first or second bounding box 91 or 92 selected from each frame image. The processor 81 also sets the third bounding box as a region of interest (ROI). The processor 81 then recognizes the customer's picking and bagging behaviors from the temporal change in the positional relationship between the first or second bounding box 91 or 92 and the ROI, which changes in accordance with the movement of the first or second bounding box 91 or 92.
The region of interest is not limited to the third bounding box 93 indicating the region of the basket 90. For example, a fourth bounding box indicating the region of a shopping bag or a personal bag placed on the bagging table 50 may be inferred, and the customer's picking and bagging behavior may be recognized from the temporal change in the positional relationship between the movement of the first or second bounding box 91 or 92 and the region of interest.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims.
The following is a summary of the scope of claims as originally filed in this application.
[C1]
a first acquisition means for acquiring operation information from a product registration operation performed by a purchaser on a payment terminal;
behavior recognition means for recognizing the behavior of the purchaser who performs the product registration operation;
A second acquisition means for acquiring the reliability of the behavior of the purchaser;
an output means for outputting result information based on the operation information and the reliability;
A fraudulent activity estimation device comprising:
[C2]
The fraudulent activity estimation device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means outputs information informing of fraudulent activity on the display unit of the payment terminal.
[C3]
The fraudulent activity inference device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means outputs information informing a store clerk terminal of the fraudulent activity.
[C4]
The fraudulent activity estimation device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means outputs a sound to the payment terminal to notify the user of fraudulent activity.
[C5]
The fraudulent activity estimation device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means causes a light-emitting unit of the payment terminal to emit light to indicate fraudulent activity.
[C6]
The fraudulent activity estimation device according to [C1], wherein if the reliability does not satisfy a predetermined condition, the output means does not output the result information and stores the operation information and the reliability in a memory unit.
[C7]
The computer of the fraud estimation device
A function to acquire operation information from the purchaser's product registration operation on the payment terminal,
a function of recognizing the behavior of the purchaser who performs the product registration operation;
A function for obtaining the reliability of the purchaser's behavior; and
a function of outputting result information based on the operation information and the reliability;
A control program to achieve this.
[C8]
The fraud estimation device
Acquire operation information from the purchaser's product registration operation on the payment terminal,
Recognizing the behavior of the purchaser who performs the product registration operation;
Obtaining a reliability of the purchaser's behavior;
outputting result information based on the operation information and the reliability.
[C9]
The behavior recognition means recognizes the behavior of the purchaser based on temporal changes in the correspondence between the position of the hand based on the purchaser's skeleton estimated from images taken by a camera and the position of the area of interest related to the payment terminal.Fraudulent activity estimation device described in [C1].
[C10]
The behavior recognition means recognizes the behavior of the purchaser based on temporal changes in the correspondence between the position of a bounding box indicating the area of the purchaser's hand estimated from an image taken by a camera and the position of an area of interest related to the payment terminal.Fraudulent activity estimation device described in [C1]

11...self-service POS terminal, 12...POS server, 13...display control device, 14...attendant terminal, 15...communication network, 21...camera, 22...fraudulent activity estimation device, 40...main body, 41...touch panel, 42...reading window, 43...card insertion slot, 44...receipt issuing slot, 45...coin insertion slot, 46...coin dispensing outlet, 47...banknote insertion slot, 48...banknote dispensing outlet, 50...bagging stand, 52...bag holder, 53...holding arm, 60...basket stand, 61...handheld scanner, 63...standing stand, 64...display pole, 65...light emitting unit, 81...processor, 82...main memory, 83...auxiliary storage device, 84...clock, 85...speaker, 86...camera interface, 87...communication interface, 88...system bus, 100...self-service POS system, 200...fraudulent activity estimation system, 821...behavior file, 822...time series buffer, 823...output table, 221...behavior recognition unit, 222...operation recognition unit, 223...first acquisition unit, 224...second acquisition unit, 225...fraudulent activity estimation unit, 226...output unit.

Claims (10)

a first acquisition means for acquiring operation information from a product registration operation performed by a purchaser on a payment terminal;
behavior recognition means for recognizing the behavior of the purchaser who performs the product registration operation;
a second acquiring means for acquiring a reliability level representing the reliability of the recognition result of recognizing the behavior of the purchaser;
a fraud inference means for inferring fraudulent behavior by the purchaser based on the operation information and the reliability;
an output means for outputting result information for deterring fraudulent behavior by the purchaser based on the operation information and the reliability when fraudulent behavior by the purchaser is suspected;
Equipped with
The behavior recognition means is a fraudulent activity estimation device that recognizes the behavior of the purchaser based on temporal changes in the correspondence between the position of the purchaser's hand estimated from images taken by a camera and the position of the area of interest related to the payment terminal . The fraudulent activity inference device of claim 1, wherein, if the reliability satisfies a predetermined condition, the output means outputs information indicating fraudulent activity on the display unit of the payment terminal. The fraudulent activity inference device of claim 1, wherein, if the reliability satisfies a predetermined condition, the output means outputs information informing a store clerk terminal of the fraudulent activity. The fraudulent activity estimation device of claim 1, wherein, if the reliability satisfies a predetermined condition, the output means outputs a sound to the payment terminal to notify the user of fraudulent activity. The fraudulent activity inference device of claim 1, wherein, when the reliability satisfies a predetermined condition, the output means causes a light-emitting unit of the payment terminal to emit light to indicate fraudulent activity. The fraudulent activity inference device of claim 1, wherein if the reliability does not satisfy a predetermined condition, the output means does not output the result information and stores the operation information and the reliability in a storage unit. The computer of the fraud estimation device
A function to acquire operation information from the purchaser's product registration operation on the payment terminal,
a function of recognizing the behavior of the purchaser who performs the product registration operation;
A function of acquiring a reliability indicating the reliability of the recognition result of the purchaser's behavior ;
A function of inferring fraudulent behavior by the purchaser based on the operation information and the reliability; and
a function of outputting result information for deterring fraudulent behavior by the purchaser based on the operation information and the reliability when fraudulent behavior by the purchaser is suspected;
To achieve this ,
A control program in which the function of recognizing the behavior of the purchaser performing the product registration operation is a function of recognizing the behavior of the purchaser based on temporal changes in the correspondence between the position of the purchaser's hand estimated from an image taken by a camera and the position of the area of interest related to the payment terminal . The fraud estimation device
Acquire operation information from the purchaser's product registration operation on the payment terminal,
Recognizing the behavior of the customer performing the product registration operation based on a temporal change in the correspondence between the position of the customer's hand estimated from the image captured by the camera and the position of the area of interest related to the payment terminal;
A reliability level representing the reliability of the recognition result of the purchaser's behavior is obtained;
Inferring fraudulent behavior by the purchaser based on the operation information and the reliability;
When fraudulent behavior by the purchaser is predicted, result information for deterring fraudulent behavior by the purchaser is output based on the operation information and the reliability . The fraudulent activity estimation device described in claim 1, wherein the behavior recognition means estimates the customer's skeleton from images taken by a camera, and recognizes the customer's behavior based on temporal changes in the correspondence between the hand position based on the estimated skeleton and the position of the area of interest related to the payment terminal. The fraudulent activity estimation device described in claim 1, wherein the behavior recognition means estimates the customer's hand from an image captured by a camera and recognizes the customer's behavior based on temporal changes in the correspondence between the position of a bounding box indicating the estimated hand area and the position of an area of interest related to the payment terminal.