JP6915314B2 - Information processing equipment, control methods, and programs - Google Patents

Description

The present invention relates to the settlement of goods.

Customers who shop at stores need to settle the products they want to purchase. Therefore, the customer brings the product he / she wants to purchase to the cashier counter. At the cash register terminal, a process of registering a product brought by a customer as a payment target is performed. Then, the customer purchases the product by paying the price of the product registered as the payment target.

In this way, a system for registering products as payment targets has been developed. Patent Document 1 discloses a technique for identifying a product photographed by a video camera and targeting the specified product for settlement. Here, in order to identify the product, information in which the image data of the product and the product code are associated with each other is prepared in advance.

Japanese Unexamined Patent Publication No. 06-028575

The process of identifying a product from an image requires a longer time than the process of scanning a code such as a barcode to identify a product. Therefore, the present inventor has investigated a method of shortening the time required to identify a product from an image and register it as a payment target.

The present invention has been made in view of the above problems. One of the objects of the present invention is to provide a technique for shortening the time required to identify a product from an image and register it as a payment target.

The information processing apparatus of the present invention includes 1) an extraction means for extracting an image area including a product from an captured image of a product to be settled, and 2) an acquisition means for acquiring a purchase schedule list provided by a customer. 3) A specific means for specifying the product included in the extracted image area from the products included in the acquired purchase schedule list, and 4) a registration means for registering the specified product as a settlement target. Have.

The control method of the present invention is executed by a computer. The control method includes 1) an extraction step of extracting an image area including the product from the captured image of the product to be settled, 2) an acquisition step of acquiring a purchase schedule list provided by the customer, and 3). It has a specific step of specifying the product included in the extracted image area from the products included in the acquired purchase schedule list, and 4) a registration step of registering the specified product as a settlement target.

The program of the present invention causes a computer to execute each step of the control method of the present invention.

According to the present invention, there is provided a technique for shortening the time required to identify a product from an image and register it as a payment target.

It is a figure which illustrates the product registration apparatus of Embodiment 1. It is a figure which illustrates the functional structure of the product registration apparatus. It is a figure which illustrates the computer for realizing the product registration apparatus. It is a flowchart which illustrates the flow of the process executed by the product registration apparatus of Embodiment 1. FIG. It is a figure which illustrates the structure of product information in a table format. It is a flowchart which illustrates the flow of the process of specifying the product represented by the product area. It is a figure which illustrates the message which is output when the product represented by the product area cannot be specified. It is a figure exemplifying the message output when the product not shown in the purchase schedule list is about to be registered as the settlement target. It is a figure which illustrates the structure of the settlement list in a table format. It is a figure exemplifying a message notifying that there is a product which is not registered as a settlement target in a purchase schedule list. It is a block diagram which illustrates the product registration apparatus of Embodiment 2. It is a figure which illustrates the message displayed by the designated reception part. It is a figure exemplifying the flow which a message confirming whether or not there is a product which has forgotten to purchase is output. It is a figure which illustrates the message which shows the candidate of the product which should be registered as the settlement target.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by the same reference numerals, and description thereof will be omitted as appropriate. Further, in each block diagram, unless otherwise specified, each block represents a functional unit configuration rather than a hardware unit configuration.

[Embodiment 1]
FIG. 1 is a diagram illustrating an information processing device of the first embodiment (hereinafter, referred to as a product registration device 2000). Note that FIG. 1 is a diagram for facilitating the understanding of the operation of the product registration device 2000, and does not limit the operation of the product registration device 2000.

When a customer makes a purchase at a store, the product that the customer intends to purchase is registered as a checkout target at a cashier counter or the like. The customer purchases the product by making a payment (paying the price) for the product registered as the payment target.

The product registration device 2000 of the present embodiment is a device used for registering a product to be purchased by a customer as a settlement target. Therefore, the product registration device 2000 acquires the captured image 12 in which the product is captured. The captured image 12 is generated by the camera 10. The camera 10 is provided at a position where the product registered as the payment target can be imaged. For example, the camera 10 is installed at a cash register counter or the like together with the product registration device 2000. For example, when a clerk at the cashier counter causes the camera 10 to take an image of a product brought to the cashier counter by a customer to purchase, an captured image 12 in which the product is taken is generated. However, the operation of causing the camera 10 to image the product may be performed by the customer instead of the store clerk.

The product registration device 2000 extracts an image area (hereinafter, product area 20) including a product from the captured image 12. The product registration device 2000 identifies the product represented by the product area 20 by collating the information about the product included in the product area 20 with the information of each product (hereinafter, product information) prepared in advance. For example, product information is acquired from a database that manages product information. Then, the product registration device 2000 registers the specified product as a settlement target.

The information about the product included in the product area 20 is information representing the appearance of the product included in the product area 20. In addition, the product information includes collation information used for collation with information about the product included in the product area 20. The collation information is information representing the appearance of the product (for example, a template image). The collation between the information about the product included in the product area 20 and the product information (hereinafter, also referred to as the collation between the product area 20 and the product information) is the appearance of the product whose appearance of the product included in the product area 20 is indicated by the product information. It is done by comparing with. The specific method of this collation will be described later.

Here, when the product area 20 and the product information are collated, the product registration device 2000 limits the product information used for the collation to a part of the product information prepared in advance. Therefore, the product registration device 2000 acquires the purchase schedule list 30 provided by the customer. The purchase schedule list 30 is created as information indicating the products that the customer plans to purchase. As will be described later, the products actually purchased by the customer may include products not shown in the purchase schedule list 30. Similarly, the products shown in the planned purchase list 30 may include products that are not actually purchased by the customer.

The product registration device 2000 limits the product information used for collation with the product area 20 to the product information of each product included in the purchase schedule list 30. That is, the product registration device 2000 specifies the product information of the product included in the product area 20 from the product information of each product included in the purchase schedule list 30. Then, the product registration device 2000 registers the product shown in the specified product information as a product to be settled.

According to the product registration device 2000 of the present embodiment, the product information collated with the product area 20 included in the captured image 12 is limited to the product information of the product shown in the purchase schedule list 30. Therefore, the time required for collation between the product area 20 and the product information is shortened as compared with the case where the collation is performed without such limitation. Therefore, the time required to register the product as the settlement target is shortened. As a result, the time required for the customer to settle the product is shortened, and the convenience of the store is improved for the customer.

It should be noted that the method of identifying a product using the image of the product captured in this way is preferably used for things such as vegetables and fruits that are difficult to attach a barcode or the like. However, this method may be used for products that can be easily attached with a barcode or the like, such as products packed in a box.

Hereinafter, the present embodiment will be described in more detail.

<Example of functional configuration>
FIG. 2 is a diagram illustrating the functional configuration of the product registration device 2000. The product registration device 2000 has an extraction unit 2020, an acquisition unit 2030, a specific unit 2040, and a registration unit 2060. The extraction unit 2020 extracts the product area 20 from the captured image 12. The acquisition unit 2030 acquires the purchase schedule list 30. The identification unit 2040 specifies the products included in the product area 20 from the products included in the purchase schedule list 30. The registration unit 2060 registers the specified product as a settlement target.

<Example of hardware configuration of product registration device 2000>
Each functional component of the product registration device 2000 may be realized by hardware that realizes each functional component (eg, a hard-wired electronic circuit, etc.), or a combination of hardware and software (eg, example). It may be realized by a combination of an electronic circuit and a program that controls it). Hereinafter, a case where each functional component of the product registration device 2000 is realized by a combination of hardware and software will be further described.

FIG. 3 is a diagram illustrating a computer 1000 for realizing the product registration device 2000. The computer 1000 is a cash register terminal such as a POS terminal. However, the computer 1000 is not limited to the cash register terminal. For example, the computer 1000 is a personal computer (PC), a server machine, a tablet terminal, a smartphone, or the like. The computer 1000 may be a dedicated computer designed to realize the product registration device 2000, or may be a general-purpose computer.

The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission line for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is an arithmetic processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory 1060 is a main storage device realized by RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device realized by a hard disk, an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like.

The input / output interface 1100 is an interface for connecting the computer 1000 and the input / output device. In FIG. 3, the camera 10 is connected to the input / output interface 1100. However, the camera 10 does not have to be connected to the product registration device 2000.

The network interface 1120 is an interface for connecting to a communication network. The network interface 1120 may be an interface that connects to the communication network by a wireless line, or may be an interface that connects to the communication network by a wired line.

The storage device 1080 stores a program module that realizes each function of the product registration device 2000. The processor 1040 reads each of these program modules into the memory 1060 and executes them to realize each function corresponding to the program module.

The hardware configuration of the computer 1000 is not limited to the configuration shown in FIG. 3, and various other configurations may be adopted. For example, the method of connecting the processors 1040 and the like to each other is not limited to the bus connection.

<Processing flow>
FIG. 4 is a flowchart illustrating the flow of processing executed by the product registration device 2000 of the first embodiment. The extraction unit 2020 acquires the captured image 12 (S102). The extraction unit 2020 extracts the product area 20 from the captured image 12 (S104). The acquisition unit 2030 acquires the purchase schedule list 30 (S106). The identification unit 2040 specifies the product represented by the product area 20 from the products included in the purchase schedule list 30 (S108). The registration unit 2060 registers the specified product as a settlement target (S110).

<Realization example of product registration device 2000>
The computer 1000 that realizes the product registration device 2000 is, for example, a cash register terminal installed in a cash register counter. In addition, for example, the computer 1000 is realized as a server device (such as a store server for managing a store) having a function of communicating with a cash register terminal installed in a cash register counter. In this case, the computer 1000 acquires the captured image 12 generated by the camera 10 provided near the cash register terminal, identifies the product, and registers the product as the settlement target. Then, the computer 1000 notifies the cashier terminal of information (such as a product name and a price) regarding the specified product.

<Product imaging method>
The number of products imaged by the camera 10 at one time may be one or a plurality. In other words, the number of product areas 20 included in one captured image 12 may be one or a plurality. In the latter case, the identification unit 2040 specifies the product represented by the product area 20 for each of the plurality of product areas 20 included in the captured image 12.

<Acquisition of captured image 12: S102>
The extraction unit 2020 acquires the captured image 12 (S102). The camera 10 is, for example, a still camera that generates still image data. In this case, for example, the camera 10 takes an image in response to an operation (such as pressing a predetermined button) of a user (for example, a clerk or a customer) to generate an captured image 12. However, the camera 10 may be configured to perform imaging (for example, to perform imaging periodically) without relying on the user's operation.

The camera 10 may be a video camera that generates moving image data. In this case, the captured image 12 is a moving image frame constituting the moving image data generated by the camera 10. In this case as well, the camera 10 may be configured to perform imaging according to the user's operation, or may be configured to perform imaging regardless of the user's operation.

The method by which the extraction unit 2020 acquires the captured image 12 is arbitrary. For example, the extraction unit 2020 acquires the captured image 12 by accessing the storage device in which the captured image 12 is stored. The storage device in which the captured image 12 is stored may be provided inside the camera 10 that generated the captured image 12, or may be provided outside the camera 10. In addition, for example, the extraction unit 2020 may acquire the captured image 12 by receiving the captured image 12 transmitted from the camera 10 or the like.

<Extraction of product area 20: S104>
The extraction unit 2020 extracts the product area 20 from the captured image 12. Here, an existing technique can be used as a technique for extracting an image area representing a product from an image captured of a product to be registered as a settlement target. For example, it is assumed that the background of the imaging range of the camera 10 is known. In this case, the extraction unit 2020 defines the image area extracted by calculating the difference between the captured image 12 and the known background as the product area 20. Here, when a plurality of image regions separated from each other are extracted by this process, the extraction unit 2020 extracts each of the plurality of image regions as a product region 20. That is, a plurality of product areas 20 are extracted from one captured image 12.

<About product information>
The product information indicates information about the product such as a product identifier (hereinafter, product identifier), a product name, and a price. Further, the product information includes the above-mentioned collation information. The collation information is information representing the appearance of the product. For example, the collation information indicates a feature amount which is one or more values representing the features of the appearance of the product. In addition, for example, the collation information indicates an image (hereinafter, template image) showing the appearance of the product. The template image shows, for example, a photograph of a product.

FIG. 5 is a diagram illustrating the structure of product information in a table format. The table of FIG. 5 is referred to as a table 200. Table 200 has five columns: product identifier 202, product name 204, price 206, and collation information 208. In FIG. 5, the collation information 208 shows a template image of the product.

<Acquisition of purchase schedule list 30: S106>
The acquisition unit 2030 acquires the purchase schedule list 30 (S106). There are various methods for the acquisition unit 2030 to acquire the purchase schedule list 30. For example, the acquisition unit 2030 acquires the purchase schedule list 30 from the customer's mobile terminal (for example, a smartphone). Various methods can be used to acquire information from the customer's mobile terminal. For example, the acquisition unit 2030 acquires the purchase schedule list 30 transmitted by wireless communication from the customer's mobile terminal. In addition, for example, the information obtained by encoding the purchase schedule list 30 (for example, QR code (registered trademark)) may be displayed on the display screen of the customer's mobile terminal. For example, this display is performed by the customer operating a mobile terminal. The acquisition unit 2030 acquires the purchase schedule list 30 by recognizing and decoding the encoded information. An existing technology can be used as a technology for reading encoded information such as a QR code (registered trademark) and decoding the information.

In addition, for example, the purchase schedule list 30 may be stored in advance in a storage device accessible from the product registration device 2000. In this case, the purchase schedule list 30 is stored in association with the customer's identifier. For example, it is assumed that a membership system is introduced in a store where the product registration device 2000 is used. In this case, for example, the customer uploads the purchase schedule list 30 by using the Web page provided to the member. The uploaded purchase schedule list 30 is stored in a storage device accessible from the product registration device 2000 in association with the customer identifier.

In this case, for example, the product registration device 2000 acquires the customer identifier by a method such as reading the customer's membership card before registering the product purchased by the customer as a settlement target. Then, the product registration device 2000 acquires the purchase schedule list 30 stored in association with the acquired customer identifier from the storage device.

According to the method of acquiring the purchase schedule list 30 from the customer's mobile terminal, there is an advantage that the product registration device 2000 can be introduced even in a store where a membership system or the like is not introduced. On the other hand, according to the method of storing the purchase schedule list 30 in the storage device in advance, there is an advantage that the process of registering the product as the settlement target can be executed more smoothly.

<Collaboration of product area and product information: S108>
The identification unit 2040 identifies the product included in the product area 20 by collating the product area 20 with the product information (S108). Here, the product is specified by specifying the product information indicating the information about the product included in the product area 20.

There are various methods for collating the product area 20 with the product information. For example, this matching is performed by feature matching. In this case, the collation information of the product information indicates the feature amount of the product. The specific unit 2040 calculates the feature amount of the product from the product area 20, and specifies the product information of the product included in the product area 20 by matching the feature amount with the feature amount of the product indicated by each product information. .. An existing technique can be used as a technique for identifying an object included in an image by feature amount matching.

In addition, for example, the product area 20 and the product information are collated by template matching. In this case, the collation information of the product information indicates a template image of the product. The identification unit 2040 specifies the product information of the product included in the product area 20 by matching the image of the product included in the product area 20 with the template image of each product. An existing technique can be used as a technique for identifying an object included in an image by template matching.

Here, the specific unit 2040 specifies the product included in the product area 20 by using the purchase schedule list 30. Hereinafter, the flow of this process will be described with reference to a flowchart. FIG. 6 is a flowchart illustrating a flow of processing for identifying a product included in the product area 20. This flowchart specifically illustrates the process performed in S108 of FIG.

S202 to S210 are loop processes A executed for each product included in the purchase schedule list 30. In S202, if any of the products included in the purchase schedule list 30 is already subject to the loop process A, the loop process A ends. As a result, the process of FIG. 6 ends.

If some of the products included in the purchase schedule list 30 are not yet targeted by loop processing A, the specific unit 2040 selects one of the products not yet targeted by loop processing A. do. The product selected here is referred to as product i. Then, the process of FIG. 6 proceeds to S204.

The specific unit 2040 acquires the product information of the product i (S204). The specific unit 2040 determines whether or not the product included in the product area 20 is the product i by collating the product area 20 with the collation information of the product i (S206). When the product included in the product area 20 is the product i (S206: YES), the registration unit 2060 specifies that the product included in the product area 20 is the product i (S208). Then, the process of FIG. 6 ends.

When the product included in the product area 20 is not the product i (S206: NO), the process of FIG. 6 proceeds to S210. Since S210 is the end of the loop process A, the process of FIG. 6 proceeds to S202.

It is conceivable that the product included in the product area 20 is not any of the products included in the purchase schedule list 30. In this case, for example, the product registration device 2000 outputs a notification indicating that the product to be registered as the settlement target could not be specified. For example, this notification is an audio output such as a buzzer sound. In addition, for example, this notification is an output of a message indicating that the product could not be identified. FIG. 7 is a diagram illustrating a message output when the product represented by the product area 20 cannot be specified. The message 50 is a message indicating that the product included in the product area 20 could not be specified from the products included in the purchase schedule list 30. For example, the message 50 is displayed on the touch panel connected to the product registration device 2000.

In addition, for example, when the product included in the product area 20 is not any of the products included in the purchase schedule list 30, the specific unit 2040 indicates the product information and the product area of each product not shown in the purchase schedule list 30. By further collating with 20, the product included in the product area 20 is specified. When the product included in the product area 20 is specified by the collation with the product not included in the purchase schedule list 30, that is, the product included in the product area 20 is included in the purchase schedule list 30. If the product is not available, the product registration device 2000 preferably gives a predetermined notification.

For example, this notification is a voice notification such as a buzzer sound. In addition, for example, this notification is a message indicating that a product not included in the purchase schedule list 30 is about to be registered as a settlement target. FIG. 8 is a diagram illustrating a message 60 that is output when a product that is not included in the purchase schedule list 30 is about to be registered as a settlement target. Like the message 50, this message 60 is displayed on a touch panel connected to, for example, the product registration device 2000.

<Registration of products to be settled>
The registration unit 2060 registers the product specified by the specific unit 2040 as a product to be settled. For example, the registration unit 2060 sets the product as the settlement target by adding the product identifier of the specified product to the specific unit 2040 to the list showing the product identifier of the product to be settled (hereinafter referred to as the settlement list). sign up.

FIG. 9 is a diagram illustrating the structure of the settlement list in a table format. The table of FIG. 9 is referred to as a table 300. Table 300 has two columns, the product identifier 302 and the number of purchases 304.

Here, it is assumed that the product included in the product area 20 is specified by the specific unit 2040 from the products not included in the purchase schedule list 30. In this case, the product may be a product intentionally added by the customer, or a product that the customer does not intend to purchase (for example, a product that the customer accidentally brought in). Therefore, in such a case, it is preferable to confirm with the customer whether or not he / she intends to purchase this product.

Therefore, when the product included in the product area 20 is specified from the products not included in the purchase schedule list 30, the registration unit 2060 inputs to specify whether or not to register the product as the settlement target. Accept. When the input operation to register the product as the settlement target is performed, the registration unit 2060 registers the product specified by the specific unit 2040 as the settlement target. On the other hand, when the input operation to the effect that the product is not registered as the settlement target is performed, the registration unit 2060 does not register the product specified by the specific unit 2040 as the settlement target.

For example, the message 60 of FIG. 8 described above includes an OK button 62 and a cancel button 64. When the OK button 62 is pressed, the registration unit 2060 registers the specified product as a settlement target. On the other hand, when the cancel button 64 is pressed, the registration unit 2060 does not register the specified product as a settlement target. By doing so, it is possible to prevent products that the customer does not intend to purchase (for example, products that have been brought in by mistake) from being registered as payment targets.

For example, the message 60 is displayed on a touch panel that can be viewed by the customer. The customer performs an input operation by pressing the OK button 62 or the cancel button 64 on this touch panel. In addition, for example, the message 60 is displayed on a touch panel that can be viewed only by the clerk. In this case, the clerk asks the customer whether to purchase the product and operates the touch panel according to the answer.

<Confirmation of products to be registered>
After the product registration device 2000 finishes identifying the product that the customer intends to purchase, the product registration device 2000 may execute a predetermined confirmation process. For example, if the purchase schedule list 30 includes products that are not registered as payment targets, the product registration device 2000 notifies that fact. As an example, it is assumed that the purchase schedule list 30 shows the product A, the product B, and the product C. Further, it is assumed that the products registered as the settlement target are the product A and the product B, and the product C is not registered as the settlement target. In this case, the customer may have forgotten to purchase the product C. Therefore, the product registration device 2000 outputs, for example, a message confirming whether or not the product C does not have to be purchased.

FIG. 10 is a diagram illustrating a message 70 notifying that there is a product that is not registered as a settlement target in the purchase schedule list 30. In the example of FIG. 10, it is assumed that the banana is not registered as the settlement target among the products included in the purchase schedule list 30. Therefore, message 70 notifies that the banana may have been forgotten to be bought. When the customer actually forgets to buy the banana, for example, the customer or the clerk presses the additional registration button 72. As a result, the state of the product registration device 2000 shifts to the state of registering the product as the settlement target. Therefore, for example, a customer can return to the sales floor to pick up a product that he / she has forgotten to buy, so that he / she can purchase the product without forgetting to buy it.

On the other hand, when the customer has not forgotten to buy (for example, when the customer stops buying bananas), for example, the customer or the clerk presses the OK button 74. This completes the process of registering the product to be settled.

The predetermined confirmation process performed by the product registration device 2000 is not limited to the above-mentioned confirmation process. For example, the confirmation process is a process for confirming whether or not the registration result is correct when the total number of the specified products is larger than the total number of the products shown in the purchase schedule list 30. In this case, it is assumed that the purchase schedule list 30 shows the number of purchase schedules for each product.

It should be noted that the existing technology is used as a method for the product registration device 2000 to grasp that the product that the customer is trying to purchase has been specified (all the products that the customer is trying to purchase have been registered as payment targets). Can be used. For example, the product registration device 2000 receives an input from a customer or a clerk to specify that the product to be purchased by the customer has been specified. For example, the product registration device 2000 detects that the product registration process has been completed by detecting that a predetermined button such as a subtotal button has been pressed.

[Embodiment 2]
FIG. 11 is a block diagram illustrating the product registration device 2000 of the second embodiment. Except for the points described below, the product registration device 2000 of the second embodiment has the same function as the product registration device 2000 of the first embodiment.

The product registration device 2000 of the second embodiment has a designated reception unit 2080. The designated reception unit 2080 accepts an input for designating whether or not a product not included in the purchase schedule list 30 is included in the product to be settled (the product to be registered as the payment target). For example, this designation is entered before starting the process of registering the product as the settlement target.

FIG. 12 is a diagram illustrating a message 80 displayed by the designated reception unit 2080. The YES button 82 is a button for designating that the products not included in the purchase schedule list 30 are included in the products to be settled. The NO button 84 is a button for designating that the products not included in the purchase schedule list 30 are not included in the products to be settled. The message 80 is displayed on a touch panel connected to, for example, the product registration device 2000, similarly to the message 50 and the like. This touch panel may be viewable by the customer or may be viewable only by the clerk.

When it is specified that a product not included in the purchase schedule list 30 is included in the product to be settled, the specific unit 2040 includes the product information of the product not shown in the purchase schedule list 30 in the product information to be collated with the product area 20. Also use. For example, the specific unit 2040 first collates the product information of the product shown in the purchase schedule list 30 with the product area 20, and shows it in the purchase schedule list 30 only when the product included in the product area 20 is not specified. The product information of the product that has not been processed is collated with the product area 20. In addition, for example, the specific unit 2040 may collate all the product information with the product area 20 without limiting the product information used for collation from the beginning.

On the other hand, when it is specified that the products not included in the purchase schedule list 30 are not included in the products to be settled (all the products to be settled are included in the purchase schedule list 30), the specific unit 2040 is set to the product area 20. Only the product information of the product shown in the purchase schedule list 30 is used as the product information to be collated with.

As described above, in the product registration device 2000 of the present embodiment, the range of product information used for collation with the product area 20 is determined according to the above-mentioned designation. By doing so, the product information and the product area 20 are collated within an appropriate range intended by the customer. Therefore, while quickly identifying the products to be registered as payment targets, for example, even when the customer purchases additional products that were not originally planned, all products can be correctly registered as payment targets. ..

<Confirmation message>
When it is specified that the products not included in the purchase schedule list 30 are included in the payment target products, it is assumed that all the products registered as the payment target are included in the purchase schedule list 30. In this case, the customer may have forgotten to purchase some item. Therefore, it is preferable that the product registration device 2000 outputs a message confirming whether or not there is a product that has been forgotten to be purchased.

FIG. 13 is a diagram illustrating a flow in which a message 90 confirming whether or not there is a product that has been forgotten to be purchased is output. In this example, the message 80 described above is displayed and the NO button 84 is pressed before the product is registered. Therefore, it is considered that the customer is trying to purchase a product that is not included in the purchase schedule list. However, as a result of the subsequent process of registering the product as the payment target, all the products registered as the payment target are included in the purchase schedule list 30. Therefore, the product registration device 2000 displays a message 90 for confirming whether or not there is a product that has been forgotten to be purchased.

If there is no product that you have forgotten to purchase, the OK button 92 is pressed. By doing so, the process of registering the product as a settlement target is completed. On the other hand, if there is a product that has been forgotten to be purchased, the additional registration button 94 is pressed. As a result, the state of the product registration device 2000 shifts to the state of registering the product as the settlement target. Therefore, for example, a customer can return to the sales floor to pick up a product that he / she has forgotten to buy, so that he / she can purchase the product without forgetting to buy it.

The message 90 is displayed on a touch panel that can be viewed by the customer, for example. In addition, for example, the message 90 is displayed on a touch panel that can be viewed only by the clerk. In the latter case, the clerk who sees the confirmation message asks the customer if there is a product that he / she has forgotten to purchase, and performs an input operation in response to the response.

<Display of product candidates>
When it is specified that a product not included in the purchase schedule list 30 is included in the settlement target, as a result of collating the product area 20 with the product information, the product represented by the product area 20 can be specified as one. Suppose it wasn't. For example, it is conceivable that the product area 20 is similar to any of the collation information of a plurality of types of products (lettuce, cabbage, etc.) that are similar in appearance. In such a case, the product registration device 2000 may display the product candidates included in the product area 20 and select the product to be registered as the settlement target from the candidates.

FIG. 14 is a diagram illustrating a message 100 indicating a candidate product to be registered as a settlement target. The display 102 indicates that the product matching process is being executed. For example, the display 102 is blinking. In the selection area 104, product candidates to be registered as payment targets are displayed separately for each type. The customer or the clerk specifies the product to be registered as the settlement target by selecting one of the displayed candidates (for example, pressing the display).

<Hardware configuration>
The hardware configuration of the computer that realizes the product registration device 2000 of the second embodiment is represented by, for example, FIG. 3 as in the first embodiment. However, the storage device 1080 of the computer 1000 that realizes the product registration device 2000 of the present embodiment further stores a program module that realizes the function of the product registration device 2000 of the present embodiment.

Although the embodiments of the present invention have been described above with reference to the drawings, these are examples of the present invention, and a combination of the above embodiments or various configurations other than the above can be adopted.

Some or all of the above embodiments may also be described, but not limited to:
1. 1. An extraction means for extracting an image area including the product from the captured image of the product to be settled, and
How to get the purchase schedule list provided by the customer,
A specific means for identifying the products included in the extracted image area from the products included in the acquired purchase schedule list, and
An information processing device having a registration means for registering the specified product as a settlement target.
2. The specific means sets the extracted image area and the product information in which the identifiers of the plurality of products are associated with the collation information representing the appearance of the product within the range of the products shown in the purchase schedule list. By collating, the product included in the extracted image area is specified. The information processing device described in.
3. 3. When the identification means does not specify the product included in the extracted image area by the collation within the range of the product shown in the purchase schedule list, the product information is also included in the range of the product not shown in the purchase schedule list. Is collated with the extracted image area. The information processing device described in.
4. When the product included in the extracted image area is specified by collating the product information with the extracted image area in the range of the product not shown in the purchase schedule list, a predetermined notification is given. The information processing device described in.
5. It has a first designated reception means that accepts input to specify whether or not products that are not included in the purchase schedule list are included in the products to be settled.
When it is specified that a product not included in the purchase schedule list is included in the payment target product, the specific means also includes the product information and the extracted image area even in the range of the product not shown in the purchase schedule list. Check with 2. The information processing device described in.
6. The specific means is when it is specified that products not included in the purchase schedule list are included in the payment target products, and all the products registered as payment targets are included in the purchase schedule list. , Give the prescribed notification, 5. The information processing device described in.
7. After the identification means finishes the identification of each product to be purchased by the customer, if the number of the identified products is larger than the number of products shown in the purchase schedule list, a predetermined notification is given. To 4. The information processing device according to any one.
8. After the specific means has finished identifying each product to be purchased by the customer, if the purchase schedule list includes products not specified by the specific means, a predetermined notification is given. To 4. The information processing device according to any one.
9. The acquisition means acquires the purchase schedule list from the customer's mobile terminal. ~ 8. The information processing device according to any one.
10. The acquisition means acquires the customer's identifier, and acquires the purchase schedule list stored in the storage device in association with the identifier. ~ 8. The information processing device according to any one.

11. A control method performed by a computer
An extraction step that extracts an image area including the product from the captured image of the product to be settled,
The acquisition step to acquire the purchase schedule list provided by the customer, and
A specific step of identifying the products included in the extracted image area from the products included in the acquired purchase schedule list, and
A control method having a registration step of registering the specified product as a settlement target.
12. In the specific step, the extracted image area and the product information in which the identifiers of the plurality of products are associated with the collation information representing the appearance of the product are within the range of the products shown in the purchase schedule list. By collating, the product included in the extracted image area is specified. The control method described in.
13. In the specific step, if the product included in the extracted image area is not specified by the collation within the range of the products shown in the purchase schedule list, the product information is also included in the range of the products not shown in the purchase schedule list. Is collated with the extracted image area. The control method described in.
14. When the product included in the extracted image area is specified by collating the product information with the extracted image area in the range of the product not shown in the purchase schedule list, a predetermined notification is given. The control method described in.
15. It has a first designated reception step that accepts an input to specify whether or not a product that is not included in the purchase schedule list is included in the product to be settled.
In the specific step, when it is specified that a product not included in the purchase schedule list is included in the product to be settled, the product information and the extracted image area are also included in the range of the product not shown in the purchase schedule list. Check with, 12. The control method described in.
16. When it is specified in the specific step that products not included in the purchase schedule list are included in the payment target products, and all the products registered as payment targets are included in the purchase schedule list. , Give the prescribed notification, 15. The control method described in.
17. After the identification of each product to be purchased by the customer is completed in the specific step, if the number of the identified products is larger than the number of products shown in the purchase schedule list, a predetermined notification is given. To 14. The control method according to any one.
18. After the identification of each product to be purchased by the customer is completed in the specific step, if the purchase schedule list includes products not specified by the specific step, a predetermined notification is given. To 14. The control method according to any one.
19. In the acquisition step, the purchase schedule list is acquired from the customer's mobile terminal. ~ 18. The control method according to any one.
20. In the acquisition step, the customer's identifier is acquired, and the purchase schedule list stored in the storage device in association with the identifier is acquired. ~ 18. The control method according to any one.

21. 11. ~ 20. A program that causes a computer to execute each step of the control method described in any one of them.

10 Camera 12 Captured image 20 Product area 30 Purchase schedule list 50 Message 60 Message 62 OK button 64 Cancel button 70 Message 72 Additional registration button 74 OK button 80 Message 82 YES button 84 NO button 90 Message 92 OK button 94 Additional registration button 100 message 102 Display 104 Selection area 200 Table 202 Product identifier 204 Product name 206 Price 208 Matching information 300 Table 302 Product identifier 304 Number of purchases 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 I / O interface 1120 Network interface 2000 Product registration device 2000 Information Processing device 2020 Extraction unit 2030 Acquisition unit 2040 Specific unit 2060 Registration unit 2080 Designated reception unit

Claims (12)

An extraction means for extracting an image area including the product from the captured image of the product to be settled, and
How to get the purchase schedule list provided by the customer,
A specific means for identifying the products included in the extracted image area from the products included in the acquired purchase schedule list, and
An information processing device having a registration means for registering the specified product as a settlement target. The specific means indicates in the purchase schedule list information about the product included in the extracted image area and product information in which the identifiers of the plurality of products are associated with the collation information representing the appearance of the product. The information processing apparatus according to claim 1, wherein the product included in the extracted image area is specified by collating within the range of the products. When the identification means does not specify the product included in the extracted image area by the collation within the range of the product shown in the purchase schedule list, the product information is also included in the range of the product not shown in the purchase schedule list. The information processing apparatus according to claim 2, wherein the information processing device is collated with the information about the product included in the extracted image area. When the product included in the extracted image area is specified by collating the product information with the information about the product included in the extracted image area in the range of the product not shown in the purchase schedule list, a predetermined product is specified. The information processing apparatus according to claim 3, wherein the information processing device according to claim 3 is notified. It has a first designated reception means that accepts input to specify whether or not products that are not included in the purchase schedule list are included in the products to be settled.
When it is specified that a product not included in the purchase schedule list is included in the payment target product, the specific means includes the product information and the extracted image area even in the range of the product not shown in the purchase schedule list. The information processing apparatus according to claim 2, wherein the information processing device is collated with the information about the product contained in the above. The specific means is when it is specified that products not included in the purchase schedule list are included in the payment target products, and all the products registered as payment targets are included in the purchase schedule list. , The information processing apparatus according to claim 5, which gives a predetermined notification. If the number of the specified products is larger than the number of the products shown in the purchase schedule list after the identification means finishes the identification of each product to be purchased by the customer, a predetermined notification is given. 4. The information processing device according to any one of the items. Claims 1 to 1, wherein after the specific means finishes specifying each product to be purchased by the customer, if the purchase schedule list includes products not specified by the specific means, a predetermined notification is given. 4. The information processing device according to any one of the items. The information processing device according to any one of claims 1 to 8, wherein the acquisition means acquires the purchase schedule list from the customer's mobile terminal. The information processing device according to any one of claims 1 to 8, wherein the acquisition means acquires an identifier of the customer and acquires the purchase schedule list stored in the storage device in association with the identifier. A control method performed by a computer
An extraction step that extracts an image area including the product from the captured image of the product to be settled,
The acquisition step to acquire the purchase schedule list provided by the customer, and
A specific step of identifying the products included in the extracted image area from the products included in the acquired purchase schedule list, and
A control method having a registration step of registering the specified product as a settlement target. A program that causes a computer to execute each step of the control method according to claim 11.

Images