JP7243766B2 - Information processing device and control method - Google Patents

Description

The present invention relates to checkout of goods.

A customer shopping at a store needs to pay for the item he or she wants to purchase. Therefore, the customer brings the desired product to the checkout counter. At the cash register terminal, a process of registering the product brought by the customer as an object of payment is performed. Then, the customer purchases the product by paying the price of the product registered as the settlement object.

Systems have been developed for registering commodities as objects of payment in this manner. Japanese Patent Laid-Open No. 2002-200000 discloses a technique for specifying a product photographed by a video camera and making the specified product an object of payment. Here, in order to specify the product, information in which the image data of the product and the product code are associated is prepared in advance.

JP-A-06-028575

The process of specifying the product from the image takes a longer time than the process of reading a code such as a bar code to specify the product. Therefore, the inventors have studied a method of shortening the time required to identify a product from an image and register it as a payment target.

The present invention has been made in view of the above problems. One of the objects of the present invention is to provide a technique for shortening the time required to identify a product from an image and register it as an object of payment.

The information processing apparatus of the present invention comprises: 1) acquisition means for acquiring an image including a product to be settled; 2) specifying means for specifying the product in the image using a customer's purchase plan list; registration means for registering the purchased product as an object of payment.

The control method of the present invention is executed by a computer. The control method includes: 1) an acquisition step of acquiring an image including the product to be settled, 2) a specifying step of specifying the product in the image using a customer's purchase plan list, and 3) identifying the specified product. and a registration step of registering as a settlement object.

The program of the present invention causes a computer to execute each step of the control method of the present invention.

ADVANTAGE OF THE INVENTION According to this invention, the technique of shortening the time required for identifying goods from an image and registering them as payment object is provided.

1 is a diagram illustrating a product registration device according to Embodiment 1; FIG. It is a figure which illustrates the functional structure of a goods registration apparatus. It is a figure which illustrates the computer for implement|achieving a goods registration apparatus. 4 is a flowchart illustrating the flow of processing executed by the product registration device of Embodiment 1; It is a figure which illustrates the structure of product information in a table format. 7 is a flowchart illustrating a flow of processing for identifying products represented by product areas; FIG. 11 is a diagram illustrating a message output when a product represented by a product area cannot be specified; FIG. 11 is a diagram illustrating a message output when a product not shown in the planned purchase list is about to be registered as a payment target; It is a figure which illustrates the structure of a settlement list|wrist in a table format. FIG. 11 is a diagram illustrating a message notifying that there is a product that is not registered as a payment target in the planned purchase list; FIG. 11 is a block diagram illustrating a product registration device according to Embodiment 2; FIG. 10 is a diagram illustrating an example of a message displayed by a designation reception unit; FIG. FIG. 10 is a diagram illustrating a flow of outputting a message for confirming whether there is any product that the user has forgotten to purchase; FIG. 10 is a diagram illustrating a message indicating candidates for products to be registered as settlement objects;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in all the drawings, the same constituent elements are denoted by the same reference numerals, and the description thereof will be omitted as appropriate. Also, in each block diagram, unless otherwise specified, each block does not represent a hardware unit configuration but a functional unit configuration.

[Embodiment 1]
FIG. 1 is a diagram illustrating an information processing apparatus (hereinafter referred to as a product registration apparatus 2000) according to the first embodiment. FIG. 1 is a diagram for facilitating understanding of the operation of the product registration device 2000, and does not limit the operation of the product registration device 2000. FIG.

When a customer makes a purchase at a store, the product the customer intends to purchase is registered as an object of payment at a checkout counter or the like. A customer purchases a product registered as an object of payment by paying for the product (paying the price).

The commodity registration device 2000 of this embodiment is a device used for registering a commodity to be purchased by a customer as an object of payment. Therefore, the product registration device 2000 acquires the captured image 12 in which the product is captured. A captured image 12 is generated by the camera 10 . The camera 10 is provided at a position where the product to be registered as a payment target can be imaged. For example, the camera 10 is installed at a checkout counter or the like together with the product registration device 2000 . For example, when a store clerk at a checkout counter causes the camera 10 to capture an image of an item that a customer has brought to the checkout counter to purchase, a captured image 12 of the item is generated. However, the operation of causing the camera 10 to image the product may be performed by the customer instead of the store clerk.

The product registration device 2000 extracts an image area including the product (hereinafter referred to as product area 20) from the captured image 12. FIG. The product registration device 2000 identifies the product represented by the product area 20 by collating information on the product included in the product area 20 with information on each product prepared in advance (hereinafter referred to as product information). For example, product information is acquired from a database that manages product information. Then, the product registration device 2000 registers the specified product as an object of payment.

The information about the product included in the product area 20 is information representing the appearance of the product included in the product area 20 . The product information also includes matching information used for matching with information on products included in the product area 20 . The collation information is information representing the appearance of the product (for example, a template image, etc.). The collation of product information and product information contained in the product area 20 (hereinafter also referred to as collation between the product area 20 and product information) is performed by comparing the appearance of the product contained in the product area 20 with the appearance of the product indicated by the product information. This is done by comparing with A specific method for this collation will be described later.

Here, the product registration device 2000, when matching the product area 20 with the product information, limits the product information used for the matching to a part of the product information prepared in advance. For this purpose, product registration device 2000 obtains purchase plan list 30 provided by the customer. The purchase plan list 30 is created as information indicating products that the customer plans to purchase. As will be described later, the products actually purchased by the customer may include products not shown in the planned purchase list 30 . Similarly, items shown in the planned purchase list 30 may include items that are not actually purchased by the customer.

The product registration device 2000 limits the product information used for matching with the product area 20 to the product information of each product included in the planned purchase list 30 . That is, the product registration device 2000 identifies the product information of the product included in the product area 20 from among the product information of each product included in the purchase plan list 30 . Then, the product registration device 2000 registers the product indicated by the specified product information as a product for settlement.

According to the product registration device 2000 of the present embodiment, the product information to be collated with the product area 20 included in the captured image 12 is limited to the product information of the products shown in the planned purchase list 30 . Therefore, the time required for matching the product area 20 with the product information is shortened as compared with the case where the matching is performed without such limitation. Therefore, the time required for registering the product for payment is shortened. As a result, the time required for the customer to pay for the merchandise is shortened, and the convenience of the store for the customer is improved.

It should be noted that the method of identifying the product using the captured image of the product in this way is preferably used for products such as vegetables and fruits that are difficult to attach a barcode or the like. However, this method may also be used for products to which a barcode or the like can be easily attached, such as products packed in boxes.

The present embodiment will be described in further detail below.

<Example of functional configuration>
FIG. 2 is a diagram illustrating the functional configuration of the product registration device 2000. As shown in FIG. Product registration device 2000 has extraction unit 2020 , acquisition unit 2030 , identification unit 2040 , and registration unit 2060 . The extraction unit 2020 extracts the product area 20 from the captured image 12 . Acquisition unit 2030 acquires purchase plan list 30 . The identifying unit 2040 identifies the products included in the product area 20 from among the products included in the planned purchase list 30 . The registration unit 2060 registers the identified product as a payment target.

<Example of Hardware Configuration of Product Registration Device 2000>
Each functional component of the product registration device 2000 may be realized by hardware (eg, hardwired electronic circuit) that implements each functional component, or may be a combination of hardware and software (eg, combination of an electronic circuit and a program for controlling it, etc.). A case in which each functional component of product registration device 2000 is implemented by a combination of hardware and software will be further described below.

FIG. 3 is a diagram illustrating a computer 1000 for realizing the product registration device 2000. As shown in FIG. A computer 1000 is, for example, a cash register terminal such as a POS terminal. However, computer 1000 is not limited to a cash register terminal. For example, the computer 1000 is a personal computer (PC), server machine, tablet terminal, smart phone, or the like. The computer 1000 may be a dedicated computer designed to realize the product registration device 2000, or may be a general-purpose computer.

Computer 1000 has bus 1020 , processor 1040 , memory 1060 , storage device 1080 , input/output interface 1100 and network interface 1120 . The bus 1020 is a data transmission path through which the processor 1040, memory 1060, storage device 1080, input/output interface 1100, and network interface 1120 mutually transmit and receive data. The processor 1040 is an arithmetic processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory 1060 is a main memory implemented by RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device realized by a hard disk, SSD (Solid State Drive), memory card, ROM (Read Only Memory), or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 and input/output devices. In FIG. 3, the input/output interface 1100 is connected to the camera 10 . However, camera 10 does not have to be connected to product registration device 2000 .

A network interface 1120 is an interface for connecting to a communication network. The network interface 1120 may be an interface that connects to a communication network via a wireless line, or an interface that connects to a communication network via a wired line.

The storage device 1080 stores program modules that implement each function of the product registration device 2000 . Processor 1040 reads each program module into memory 1060 and executes it, thereby realizing each function corresponding to the program module.

The hardware configuration of the computer 1000 is not limited to the configuration shown in FIG. 3, and can take various other configurations. For example, the method of connecting processors 1040 and the like to each other is not limited to bus connections.

<Process flow>
FIG. 4 is a flowchart illustrating the flow of processing executed by the product registration device 2000 of the first embodiment. The extraction unit 2020 acquires the captured image 12 (S102). The extraction unit 2020 extracts the product area 20 from the captured image 12 (S104). Acquisition unit 2030 acquires purchase plan list 30 (S106). The identifying unit 2040 identifies the product represented by the product area 20 from among the products included in the planned purchase list 30 (S108). The registration unit 2060 registers the identified product as an object of payment (S110).

<Implementation example of product registration device 2000>
A computer 1000 that implements the product registration device 2000 is, for example, a cash register terminal installed at a cash register counter. In addition, for example, the computer 1000 is implemented as a server device (such as a store server for managing a store) having a function of communicating with a cash register terminal installed at a cash register counter. In this case, the computer 1000 acquires the captured image 12 generated by the camera 10 provided near the cash register terminal, identifies the product, and registers the product as a checkout target. Then, the computer 1000 notifies the cash register terminal of information (product name, price, etc.) on the specified product.

<Product imaging method>
The number of products captured by camera 10 at one time may be one or may be plural. In other words, the number of product regions 20 included in one captured image 12 may be one or plural. In the latter case, the specifying unit 2040 specifies the product represented by each of the multiple product regions 20 included in the captured image 12 .

<Acquisition of captured image 12: S102>
The extraction unit 2020 acquires the captured image 12 (S102). The camera 10 is, for example, a still camera that generates still image data. In this case, for example, the camera 10 takes an image according to an operation (such as pressing a predetermined button) by a user (for example, a store clerk or a customer) and generates a captured image 12 . However, the camera 10 may be configured so as to take an image (for example, to take an image periodically) regardless of the user's operation.

Camera 10 may be a video camera that generates moving image data. In this case, the captured image 12 is a moving image frame forming moving image data generated by the camera 10 . In this case as well, the camera 10 may be configured to capture an image in response to a user's operation, or may be configured to capture an image independently of the user's operation.

The method by which the extraction unit 2020 acquires the captured image 12 is arbitrary. For example, the extraction unit 2020 acquires the captured image 12 by accessing a storage device in which the captured image 12 is stored. The storage device in which the captured image 12 is stored may be provided inside the camera 10 that generated the captured image 12 or may be provided outside the camera 10 . Alternatively, for example, the extraction unit 2020 may acquire the captured image 12 by receiving the captured image 12 transmitted from the camera 10 or the like.

<Extraction of product area 20: S104>
The extraction unit 2020 extracts the product area 20 from the captured image 12 . Here, an existing technique can be used as a technique for extracting an image area representing a product from an image in which the product to be registered as a settlement object is captured. For example, assume that the background of the imaging range of the camera 10 is known. In this case, the extraction unit 2020 sets the image area extracted by calculating the difference between the captured image 12 and the known background as the product area 20 . Here, when a plurality of image regions that are separated from each other are extracted by this process, the extraction unit 2020 extracts each of these plurality of image regions as the product region 20 . That is, multiple product areas 20 are extracted from one captured image 12 .

<About product information>
The product information indicates information about products such as product identifiers (hereinafter referred to as product identifiers), product names, and prices. Furthermore, the product information includes the collation information described above. The matching information is information representing the appearance of the product. For example, the collation information indicates a feature amount that is one or more values representing the features of the appearance of the product. In addition, for example, the collation information indicates an image representing the appearance of the product (hereinafter referred to as a template image). A template image indicates, for example, a photograph of a product.

FIG. 5 is a diagram illustrating the configuration of product information in a table format. The table in FIG. 5 is referred to as table 200. FIG. Table 200 has five columns: product identifier 202 , product name 204 , price 206 and matching information 208 . In FIG. 5, the collation information 208 indicates a template image of the product.

<Acquisition of purchase plan list 30: S106>
Acquisition unit 2030 acquires purchase plan list 30 (S106). There are various methods for the acquisition unit 2030 to acquire the purchase plan list 30 . For example, the acquisition unit 2030 acquires the planned purchase list 30 from the customer's mobile terminal (for example, smart phone). Various methods can be used to acquire information from the customer's mobile terminal. For example, the acquisition unit 2030 acquires the purchase plan list 30 transmitted by wireless communication from the customer's mobile terminal. Alternatively, for example, information obtained by encoding the planned purchase list 30 (for example, a QR code (registered trademark)) may be displayed on the display screen of the customer's portable terminal. For example, this display is performed by the customer operating the mobile terminal. Acquisition unit 2030 acquires purchase plan list 30 by recognizing and decoding the encoded information. Existing technology can be used as a technology for reading encoded information such as a QR code (registered trademark) and decoding the information.

Alternatively, for example, the planned purchase list 30 may be stored in advance in a storage device accessible from the product registration device 2000 . In this case, the purchase plan list 30 is stored in association with the customer's identifier. For example, assume that a store using the product registration device 2000 has introduced a membership system. In this case, for example, the customer uploads the planned purchase list 30 using a web page provided to the member. The uploaded purchase plan list 30 is associated with the customer identifier and stored in a storage device accessible from the product registration device 2000 .

In this case, for example, the product registration device 2000 acquires a customer identifier by reading the customer's membership card before registering the product purchased by the customer as an object of payment. Then, the product registration device 2000 acquires the purchase plan list 30 stored in association with the acquired customer identifier from the storage device.

According to the method of acquiring the purchase plan list 30 from the customer's portable terminal, there is an advantage that the product registration device 2000 can be introduced even in stores where a membership system or the like has not been introduced. On the other hand, according to the method of storing the planned purchase list 30 in advance in the storage device, there is an advantage that the process of registering the product as the object of payment can be executed more smoothly.

<Collation of product area and product information: S108>
The identification unit 2040 identifies the product included in the product area 20 by matching the product area 20 with the product information (S108). Here, the specification of the product is performed by specifying the product information indicating information on the product included in the product area 20 .

There are various methods for collating the product area 20 with the product information. For example, this collation is performed by feature quantity matching. In this case, the collation information of product information indicates the feature quantity of the product. The specifying unit 2040 calculates the feature amount of the product from the product area 20, and matches the feature amount with the feature amount of the product indicated by each item of product information to specify the product information of the product included in the product region 20. . An existing technique can be used as a technique for specifying an object included in an image by feature amount matching.

In addition, for example, matching between the product area 20 and product information is performed by template matching. In this case, the matching information of the product information indicates the template image of the product. The identifying unit 2040 identifies the product information of the products included in the product area 20 by matching the product image included in the product area 20 with the template image of each product. An existing technique can be used as a technique for specifying an object included in an image by template matching.

Here, the specifying unit 2040 uses the purchase plan list 30 to specify the products included in the product area 20 . The flow of this processing will be described below using a flowchart. FIG. 6 is a flowchart illustrating the flow of processing for identifying products included in the product area 20. As shown in FIG. This flowchart specifically illustrates the processing performed in S108 of FIG.

S202 to S210 are loop processing A executed for each product included in the planned purchase list 30 . In S202, if all of the products included in the planned purchase list 30 have already been subjected to the loop processing A, the loop processing A ends. As a result, the processing of FIG. 6 ends.

If there is a product that has not yet been subjected to loop processing A among the products included in the planned purchase list 30, the identifying unit 2040 selects one of the products that have not yet been subjected to loop processing A. do. The product selected here is described as product i. Then, the process of FIG. 6 proceeds to S204.

The identification unit 2040 acquires the product information of the product i (S204). The identifying unit 2040 compares the product area 20 with the matching information of the product i to determine whether the product included in the product region 20 is the product i (S206). If the product included in the product area 20 is the product i (S206: YES), the registration unit 2060 identifies that the product included in the product area 20 is the product i (S208). Then, the processing of FIG. 6 ends.

If the product included in the product area 20 is not product i (S206: NO), the process of FIG. 6 proceeds to S210. Since S210 is the end of loop processing A, the processing in FIG. 6 proceeds to S202.

Note that there may be a case where the product included in the product area 20 is not any of the products included in the planned purchase list 30 . In this case, for example, the product registration device 2000 outputs a notification to the effect that the product to be registered for settlement could not be specified. For example, this notification is an audio output such as a buzzer sound. In addition, for example, this notification is output of a message indicating that the product could not be specified. FIG. 7 is a diagram exemplifying a message output when the product represented by the product area 20 cannot be identified. The message 50 is a message indicating that the product included in the product area 20 could not be identified from the products included in the planned purchase list 30 . For example, message 50 is displayed on a touch panel connected to product registration device 2000 .

In addition, for example, if the product included in the product area 20 is not any of the products included in the planned purchase list 30, the specifying unit 2040 identifies the product information of each product not shown in the planned purchase list 30 and the product area. 20, the product included in the product area 20 is specified. It should be noted that if a product included in the product area 20 is identified by matching with products not included in the planned purchase list 30 in this way, that is, if the product included in the product area 20 is included in the planned purchase list 30, If the product is not available, the product registration device 2000 preferably gives a predetermined notification.

For example, this notification is notification by sound such as buzzer sound. In addition, for example, this notification is a message indicating that a product not included in the planned purchase list 30 is about to be registered as an object of payment. FIG. 8 is a diagram exemplifying a message 60 that is output when a product that is not included in the planned purchase list 30 is about to be registered for payment. This message 60 is displayed, for example, on a touch panel connected to the product registration device 2000, like the message 50. FIG.

<Registration of products for payment>
The registration unit 2060 registers the product identified by the identification unit 2040 as a product for payment. For example, the registration unit 2060 adds the product identifier of the product specified by the specifying unit 2040 to a list showing the product identifiers of the products to be settled (hereinafter referred to as a checkout list), so that the product is designated as the product to be settled. register.

FIG. 9 is a diagram illustrating the configuration of the settlement list in a table format. The table in FIG. 9 is referred to as table 300. FIG. Table 300 has two columns, product identifier 302 and purchase count 304 .

Here, it is assumed that the product included in the product area 20 is specified by the specifying unit 2040 from among the products not included in the planned purchase list 30 . In this case, the product may be a product intentionally added by the customer, or may be a product that the customer does not intend to purchase (for example, a product that the customer brought by mistake). Therefore, in such cases, it is preferable to confirm with the customer whether or not the customer is willing to purchase this product.

Therefore, when a product included in the product area 20 is specified from among products not included in the planned purchase list 30, the registration unit 2060 receives an input for specifying whether or not to register the product as an object of payment. accept. When an input operation is performed to register the product as an object of payment, the registration unit 2060 registers the product identified by the identification unit 2040 as an object of payment. On the other hand, when an input operation is performed to indicate that the product is not to be registered as an object of payment, the registration unit 2060 does not register the product specified by the specifying unit 2040 as an object of payment.

For example, message 60 in FIG. 8 described above includes OK button 62 and Cancel button 64 . When the OK button 62 is pressed, the registration unit 2060 registers the specified product as an object of payment. On the other hand, when the cancel button 64 is pressed, the registration unit 2060 does not register the identified product as a payment target. By doing so, it is possible to prevent a product that the customer does not intend to purchase (for example, a product brought by mistake) from being registered as an object of payment.

For example, message 60 is displayed on a touch panel viewable by the customer. The customer performs an input operation by pressing the OK button 62 or the cancel button 64 on this touch panel. In addition, for example, the message 60 is displayed on a touch panel that can be viewed only by the store clerk. In this case, the store clerk asks the customer whether or not to purchase the product, and operates the touch panel according to the answer.

<Confirmation of products to be registered>
After the product registration device 2000 finishes specifying the product that the customer is going to purchase, the product registration device 2000 may execute a predetermined confirmation process. For example, if the planned purchase list 30 includes a product that is not registered as a payment target, the product registration device 2000 notifies the user of the fact. As an example, assume that the purchase plan list 30 indicates product A, product B, and product C. FIG. It is also assumed that the products registered for settlement are the products A and B, and the product C is not registered for settlement. In this case, the customer may forget to purchase the product C. Therefore, the product registration device 2000 outputs, for example, a message to confirm whether or not the product C does not need to be purchased.

FIG. 10 is a diagram exemplifying a message 70 notifying that there is an item in the planned purchase list 30 that is not registered as an item to be settled. In the example of FIG. 10, it is assumed that among the commodities included in the planned purchase list 30, bananas are not registered as items to be settled. Therefore, the message 70 notifies that the user may have forgotten to buy bananas. When the customer actually forgets to buy bananas, for example, the customer or store clerk presses the additional registration button 72 . As a result, the state of the product registration device 2000 transitions to a state in which the product is registered as an object of payment. Therefore, for example, the customer can purchase the product without forgetting to buy it by returning to the sales floor to pick up the product that the customer forgot to buy.

On the other hand, if the customer has not forgotten to buy (for example, the customer has stopped buying bananas), the customer or the store clerk presses the OK button 74, for example. This completes the process of registering the product for payment.

The predetermined confirmation process performed by the product registration device 2000 is not limited to the confirmation process described above. For example, the confirmation process is a process of confirming whether or not the registration result is correct when the total number of specified products is greater than the total number of products shown in the planned purchase list 30 . In this case, it is assumed that the planned purchase list 30 indicates the number of planned purchases for each product.

The method for the product registration device 2000 to recognize that the product to be purchased by the customer has been specified (that all the products to be purchased by the customer have been registered as objects of payment) does not include any existing technology. can be used. For example, the product registration device 2000 receives an input from a customer or a store clerk designating that the product that the customer intends to purchase has been specified. For example, the product registration apparatus 2000 detects that a predetermined button such as a subtotal button has been pressed, thereby detecting completion of product registration processing.

[Embodiment 2]
FIG. 11 is a block diagram illustrating the product registration device 2000 of the second embodiment. The product registration device 2000 of the second embodiment has the same functions as the product registration device 2000 of the first embodiment, except for the points described below.

The product registration device 2000 of Embodiment 2 has a designation reception unit 2080 . The designation receiving unit 2080 receives an input designating whether or not a product not included in the planned purchase list 30 is included in the products to be settled (products to be registered as objects of payment). For example, this designation is entered before starting the process of registering the product as the object of payment.

FIG. 12 is a diagram exemplifying a message 80 displayed by the designation reception unit 2080. As shown in FIG. The YES button 82 is a button for designating that a product not included in the planned purchase list 30 is included in the products to be settled. The NO button 84 is a button for designating that a product not included in the planned purchase list 30 is not included in the products to be settled. The message 80 is displayed, for example, on a touch panel connected to the product registration device 2000, like the message 50 and the like. This touch panel may be browsable by the customer, or may be browsable only by the store clerk.

When it is specified that a product not included in the planned purchase list 30 is included in the products to be settled, the identifying unit 2040 adds product information of products not shown in the planned purchase list 30 to the product information to be collated with the product area 20. also use. For example, the specifying unit 2040 first compares the product information of the product shown in the planned purchase list 30 with the product area 20, and only when the product included in the product area 20 is not specified, the product shown in the planned purchase list 30 is identified. The product information of the product that has not been checked is collated with the product area 20. - 特許庁Alternatively, for example, the specifying unit 2040 may match all product information with the product area 20 without limiting the product information to be used for matching from the beginning.

On the other hand, when it is specified that a product not included in the planned purchase list 30 is not included in the products to be settled (all the products to be settled are included in the planned purchase list 30), the specifying unit 2040 Only the product information of the products shown in the purchase plan list 30 is used as the product information to be collated with.

As described above, in the product registration device 2000 of the present embodiment, the range of product information used for matching with the product area 20 is determined according to the above designation. By doing this, the product information and the product area 20 are collated within an appropriate range intended by the customer. Therefore, it is possible to quickly specify the products to be registered for settlement, and to correctly register all the products for settlement, even if the customer purchases additional products that were not initially planned. .

<Confirmation message>
It is assumed that all the products registered as settlement objects are included in the purchase plan list 30 when it is specified that the products not included in the planned purchase list 30 are included in the products to be settled. In this case, the customer may forget to purchase some product. Therefore, it is preferable for the product registration device 2000 to output a message for confirming whether there is any product that the user has forgotten to purchase.

FIG. 13 is a diagram exemplifying the flow of outputting a message 90 for confirming whether there is any product that the customer has forgotten to purchase. In this example, the aforementioned message 80 is displayed and the NO button 84 is pressed before the product is registered. Therefore, it is considered that the customer intends to purchase a product that is not included in the planned purchase list. However, as a result of the subsequent processing of registering the products for payment, all the products registered for payment are included in the planned purchase list 30 . Therefore, the product registration device 2000 displays a message 90 for confirming whether there is any product that the user has forgotten to purchase.

If there is no product that the user has forgotten to purchase, the OK button 92 is pressed. By doing so, the process of registering the product as the object of payment is completed. On the other hand, if there is a product that the user has forgotten to purchase, the additional registration button 94 is pressed. As a result, the state of the product registration device 2000 transitions to a state in which the product is registered as an object of payment. Therefore, for example, the customer can purchase the product without forgetting to buy it by returning to the sales floor to pick up the product that the customer forgot to buy.

The message 90 is displayed, for example, on a touch panel that can be viewed by the customer. In addition, for example, the message 90 is displayed on a touch panel that can be viewed only by the store clerk. In the latter case, the store clerk who sees the confirmation message asks the customer if there are any items that the customer has forgotten to purchase, and performs an input operation according to the reply.

<Display of product candidates>
When items not included in the planned purchase list 30 are specified to be included in the settlement target, as a result of matching the item area 20 with the item information, one item represented by the item area 20 cannot be specified. Suppose there was not. For example, it is conceivable that the product region 20 is similar to all matching information of a plurality of types of products (lettuce, cabbage, etc.) that are similar in appearance. In such a case, the product registration device 2000 may display product candidates included in the product area 20, and select a product to be registered as a checkout target from among the candidates.

FIG. 14 is a diagram exemplifying a message 100 indicating product candidates to be registered as settlement objects. A display 102 indicates that the product matching process is in progress. For example, display 102 is blinking. In the selection area 104, candidates for commodities to be registered as payment targets are displayed separately for each type. The customer or store clerk selects one of the displayed candidates (for example, presses the display) to specify the product to be registered as the payment target.

<Hardware configuration>
The hardware configuration of a computer that implements the product registration device 2000 of the second embodiment is shown in FIG. 3, for example, as in the first embodiment. However, the storage device 1080 of the computer 1000 that implements the product registration apparatus 2000 of this embodiment further stores a program module that implements the functions of the product registration apparatus 2000 of this embodiment.

Although the embodiments of the present invention have been described above with reference to the drawings, these are examples of the present invention, and combinations of the above embodiments or various configurations other than those described above can also be adopted.

Some or all of the above-described embodiments can also be described in the following supplementary remarks, but are not limited to the following.
1. an extracting means for extracting an image area including the product from the captured image of the product to be settled;
an acquisition means for acquiring a purchase plan list provided by a customer;
identifying means for identifying products included in the extracted image area from products included in the obtained purchase plan list;
and registering means for registering the identified product as an object of payment.
2. The specifying means stores the extracted image area and product information in which the identifiers of the plurality of products are associated with collation information representing the appearance of the product, within the range of the products shown in the planned purchase list. 1. identifying products included in the extracted image area by collation; The information processing device according to .
3. If the product included in the extracted image area is not specified in the collation within the range of the products shown in the planned purchase list, the specifying means also stores the product information in the range of products not shown in the planned purchase list. and the extracted image area;2. The information processing device according to .
4. 3. When the product included in the extracted image area is specified by matching the product information with the extracted image area in the range of products not shown in the purchase plan list, a predetermined notification is made. The information processing device according to .
5. a first designation receiving means for receiving an input designating whether or not a product not included in the planned purchase list is included in the product to be settled;
When it is specified that a product not included in the planned purchase list is included in the product to be settled, the identifying means is configured to include the product information and the extracted image area even in the range of products not shown in the planned purchase list. 2. The information processing device according to .
6. When it is specified that products not included in the planned purchase list are included in the products to be settled, the identifying means includes all products registered as products to be settled in the planned purchase list. 5. giving a predetermined notification; The information processing device according to .
7. After the specifying means finishes specifying each product to be purchased by the customer, if the number of the specified products is greater than the number of products shown in the planned purchase list, a predetermined notification is made;1. to 4. The information processing device according to any one of the above.
8. After the specification means finishes specifying each product to be purchased by the customer, if the planned purchase list includes products that have not been specified by the specification means, a predetermined notification is made; to 4. The information processing device according to any one of the above.
9. 1. The acquisition means acquires the purchase plan list from the customer's portable terminal; to 8. The information processing device according to any one of the above.
10. 1. The acquisition means acquires an identifier of the customer and acquires the purchase plan list stored in a storage device in association with the identifier; to 8. The information processing device according to any one of the above.

11. A control method implemented by a computer, comprising:
an extracting step of extracting an image area including the product from the imaged image of the product to be settled;
a obtaining step of obtaining a shopping list provided by the customer;
a specifying step of specifying the products included in the extracted image area from among the products included in the obtained purchase plan list;
and a registration step of registering the specified product as an object of payment.
12. In the specifying step, the extracted image area and the product information in which the identifiers of the plurality of products are associated with matching information representing the appearance of the product, within the range of the products shown in the planned purchase list. 11. Identifying products included in the extracted image area by collation; The control method described in .
13. In the specifying step, if the product included in the extracted image area is not specified in the collation within the range of products shown in the planned purchase list, the product information is also included in the range of products not shown in the planned purchase list. and the extracted image area;12. The control method described in .
14. 13. If a product included in the extracted image area is identified by matching the product information with the extracted image area in the range of products not shown in the purchase plan list, a predetermined notification is made. The control method described in .
15. a first designation receiving step for receiving an input designating whether or not a product not included in the planned purchase list is included in the product to be settled;
In the specifying step, when it is specified that a product not included in the planned purchase list is included in the product to be settled, the product information and the extracted image area are also included in the range of products not shown in the planned purchase list. 12. The control method described in .
16. When it is specified in the identifying step that products not included in the planned purchase list are included in the products to be settled, and all the products registered for payment are included in the planned purchase list , giving a predetermined notification;15. The control method described in .
17. 11. After specifying each product to be purchased by the customer in the specifying step, if the number of the specified products is greater than the number of products shown in the planned purchase list, a predetermined notification is made; to 14. A control method according to any one of the preceding claims.
18. 11. After specifying each product to be purchased by the customer in the specifying step, if the planned purchase list includes products not specified in the specifying step, a predetermined notification is provided; to 14. A control method according to any one of the preceding claims.
19. 10. In the obtaining step, the purchase plan list is obtained from the customer's portable terminal; to 18. A control method according to any one of the preceding claims.
20. 11. In the acquisition step, the identifier of the customer is acquired, and the purchase plan list stored in the storage device is acquired in association with the identifier; to 18. A control method according to any one of the preceding claims.

21. 11. 20. A program that causes a computer to execute each step of the control method described in any one.

10 Camera 12 Captured image 20 Product area 30 Purchase plan list 50 Message 60 Message 62 OK button 64 Cancel button 70 Message 72 Additional registration button 74 OK button 80 Message 82 YES button 84 NO button 90 Message 92 OK button 94 Additional registration button 100 Message 102 display 104 selection area 200 table 202 product identifier 204 product name 206 price 208 matching information 300 table 302 product identifier 304 number of purchases 1000 calculator 1020 bus 1040 processor 1060 memory 1080 storage device 1100 input/output interface 1120 network interface 2000 product registration device 2000 information Processing device 2020 Extraction unit 2030 Acquisition unit 2040 Identification unit 2060 Registration unit 2080 Designation acceptance unit

Claims (11)

Acquisition means for acquiring an image including the product for payment;
identifying means for identifying the product of the image from the products included in the customer's purchase plan list;
a registration means for registering the specified product as an object of payment;
An information processing device comprising: The specifying means acquires product information including collation information, which is information representing the appearance of the product, of the product included in the purchase plan list of the customer, and uses the image and the product information to identify the product of the image. 2. The information processing apparatus according to claim 1, which specifies the . Acquisition means for acquiring an image including the product for payment;
identifying means for identifying the product of the image from the products included in the customer's purchase plan list;
a registration means for registering the specified product as an object of payment;
with
The information processing apparatus, wherein, when the product cannot be specified, the specifying means specifies the product by using product information including information on products other than products indicated in the planned purchase list. Acquisition means for acquiring an image including the product for payment;
identifying means for identifying the product of the image from the products included in the customer's purchase plan list;
a registration means for registering the specified product as an object of payment;
with
The specifying means includes information about the product included in the image and product information in which the identifier of each of a plurality of products is associated with collation information representing the appearance of the product, which is shown in the planned purchase list. An information processing device that identifies the product included in the image by matching within the range of the product. 4. The information processing apparatus according to claim 3 , wherein when the product is identified using product information including information on products other than the product indicated by the planned purchase list, a predetermined notification is made. a first designation reception means for receiving an input designating whether or not a product other than the product indicated in the planned purchase list is included in the product to be settled;
When it is specified that a product other than the product indicated by the planned purchase list is included in the products to be settled, the identifying means also includes the product information and the 5. The information processing apparatus according to claim 4 , wherein the image is collated with information on products included in the image. When it is specified that the products other than the products indicated in the planned purchase list are included in the products to be settled, the specifying means indicates that all the products registered as the products to be settled are shown in the planned purchase list. 7. The information processing apparatus according to claim 6 , wherein a predetermined notification is made when the time is up. After the identifying means finishes identifying each product to be purchased by the customer, if the number of the identified products is greater than the number of products indicated by the planned purchase list, a predetermined notification is made. 5. The information processing device according to any one of the items. after the specifying means finishes specifying each product to be purchased by the customer, if the product indicated by the planned purchase list includes a product not specified by the specifying means, performing a predetermined notification ; Item 6. The information processing apparatus according to any one of Items 1 to 5 . 10. The information processing apparatus according to any one of claims 1 to 9 , wherein said identifying means identifies the product of said image using said purchase plan list obtained from said customer's portable terminal. A control method implemented by a computer, comprising:
an acquisition step of acquiring an image including the product to be settled;
an identification step of identifying the product of the image from the products included in the customer's purchase plan list;
a registration step of registering the identified product as an object of payment;
A control method comprising:

Images