JP6987953B2 - Product registration device and product registration program - Google Patents

Description

Embodiments of the present invention relate to a product registration device and a product registration program.

In retail stores and the like, POS terminals that manage sales information are generally used. Further, a product registration device for registering a product that requires accounting in a POS terminal has also been put into practical use. The product registration device recognizes a product, for example, by reading a product identification code for identifying the product attached to the product from the product. In addition, a product registration device that automatically confirms a product reflected in an image based on a product image and a product recognition dictionary having a feature amount corresponding to the image of each product in advance has been generally put into practical use. There is.

Japanese Patent No. 5826152

Since the above automatic confirmation is a method of automatically identifying the product having the highest degree of similarity between the feature amount calculated from the image and the feature amount of the product recognition dictionary, erroneous recognition may occur. Some products may or may not have a product identification code. In order to prevent erroneous recognition, it is desirable that the product to which the product identification code is attached is recognized by reading the product identification code.

Therefore, an object of the present invention is to provide a product registration device and a product registration program capable of recognizing and registering products with high accuracy.

The product registration device according to one embodiment has an image acquisition unit that acquires an image of a product, a memory that stores a product recognition dictionary having a feature amount for each product, and a product in the image for identifying a product. Based on the image, a system identification unit that identifies whether the product is a first-series product to which the product identification code is attached or a second-series product to which the product identification code is not assigned, and the product from the image. A first product recognition unit that acquires an identification code and recognizes a product based on the acquired product identification code, a feature amount calculated from the image, and the calculated feature amount and the feature of each product in the product recognition dictionary. A second product recognition unit that outputs product candidates in a selectable state based on the quantity and recognizes the product according to the selection, and when the product is identified as the first system product, the second product. When the product recognition unit of the product recognition unit suspends the recognition of the product and the product identification code is acquired during the suspension, the first product recognition unit and the second product recognition unit so as to recognize the product by the first product recognition unit. When the product recognition unit is controlled and the product is identified as the second system product, the control unit that recognizes the product by the second product recognition unit, the product registration unit that registers the product recognition result, and the product registration unit. Equipped with.

FIG. 1 is a diagram for explaining an example of the appearance of the product registration device and the POS terminal according to the first embodiment. FIG. 2 is a diagram for explaining a configuration example of the control system of the product registration device and the POS terminal according to the first embodiment. FIG. 3 is a diagram for explaining an example of the operation of the product registration device and the POS terminal according to the first embodiment. FIG. 4 is a diagram for explaining an example of a product registration process of the product registration device according to the first embodiment. FIG. 5 is a diagram for explaining an example of a product registration process of the product registration device according to the second embodiment.

Hereinafter, the product registration device and the product registration program according to the embodiment will be described with reference to the drawings.
(First Embodiment)
First, the product registration device 1 according to the first embodiment will be described. 1 and 2 are explanatory views showing a configuration example of a product registration device 1 and a POS (point of sales) terminal 2 according to the first embodiment. FIG. 1 shows an example of the appearance of the product registration device 1 and the POS terminal 2. FIG. 2 shows a configuration example of the control system of the product registration device 1 and the POS terminal 2. The product registration device 1 and the POS terminal 2 are used, for example, for accounting in a store such as a retail store. The product registration device 1 and the POS terminal 2 are connected by a cable (not shown) or wirelessly.

The product registration device 1 is a device that acquires an image from a product, recognizes the product based on the acquired image, and registers the recognized product in the POS terminal 2 as a product that requires accounting. The product registration device 1 is arranged on, for example, the accounting counter 3. The product registration device 1 recognizes a product by acquiring a product identification code for identifying the product. The product identification code is converted into a pattern such as a bar code or a two-dimensional code and attached to the product. For example, the product registration device 1 recognizes a product by reading the product identification code assigned to the product. Further, for example, the product registration device 1 acquires an image of the appearance of the product, and refers to and acquires a dictionary (product recognition dictionary) in which the feature amount for each product generated from the image and the product identification code are associated with each other. The product is recognized by estimating the product identification code corresponding to the product shown in the image.

The product registration device 1 refers to a table (product table) in which a product identification code, a product name, and an amount of money are associated with each other, and supplies the product identification code, the product name, and the amount of the recognized product to the POS terminal 2. do. As a result, the product registration device 1 performs the product registration process of registering the product in the list of products (accounting product list) that requires accounting of the POS terminal 2.

As shown in FIGS. 1 and 2, the product registration device 1 includes a CPU 11, a ROM 12, a RAM 13, a non-volatile memory 14, a connection I / F15, a device I / F16, an operation unit 17, a first display unit 18, and a first. The display unit 19, the image pickup unit 20, and the housing 21 of No. 2 are provided.

The CPU 11 is an arithmetic element (for example, a processor) that executes arithmetic processing. The CPU 11 performs various processes based on data such as a program stored in the ROM 12. The CPU 11 functions as a control unit capable of executing various operations by executing a program stored in the ROM 12.

The ROM 12 is a read-only non-volatile memory. The ROM 12 stores a program, data used in the program, and the like.

The RAM 13 is a volatile memory that functions as a working memory. The RAM 13 temporarily stores data and the like being processed by the CPU 11. Further, the RAM 13 temporarily stores a program executed by the CPU 11.

The non-volatile memory 14 is a storage medium capable of storing various information. The non-volatile memory 14 stores a program, data used in the program, and the like. The non-volatile memory 14 is, for example, a solid state drive (SSD), a hard disk drive (HDD), or other storage device. Instead of the non-volatile memory 14, a memory I / F such as a card slot into which a storage medium such as a memory card can be inserted may be provided. For example, the non-volatile memory 14 stores a product recognition dictionary, a product table, and the like. Further, for example, the non-volatile memory 14 stores a program (product registration program) for recognizing a product using a product recognition dictionary and a product table and registering the recognized product in the accounting product list of the POS terminal 2. Further, the non-volatile memory 14 stores a system identification dictionary used in the product registration program. The system identification dictionary will be described later.

The connection I / F15 is an interface for communicating with other devices. The connection I / F15 is used for communication with the POS terminal 2. The connection I / F15 is a connection terminal having a configuration conforming to some communication standard such as the USB standard or the RS-232C standard. The connection I / F15 may be a cable to which the POS terminal 2 can be connected. Further, the connection I / F15 may be one that performs wireless communication with the POS terminal 2 according to a standard such as Bluetooth (registered trademark) or Wi-fi (registered trademark).

The device I / F 16 is an interface for communicating between various functional blocks and the CPU 11. For example, the device I / F 16 is used for communication with the operation unit 17, the first display unit 18, the second display unit 19, the image pickup unit 20, and the like. The device I / F16 is a connection terminal having a configuration conforming to some communication standard such as the USB standard or the RS-232C standard.

The operation unit 17 generates an operation signal based on the operation of the operation member. The operating member is, for example, a touch sensor, a numeric keypad, a keyboard, or the like. The touch sensor is, for example, a resistance film type touch sensor, a capacitance type touch sensor, or the like. The touch sensor acquires information indicating a designated position within a certain area. The touch sensor is configured as a touch panel integrally with the first display unit 18, so that a signal indicating the touched position on the screen displayed on the first display unit 18 is input to the CPU 11.

The first display unit 18 is a display device that displays a screen according to a video signal input from a display control unit such as a CPU 11 or a graphic controller (not shown). On the first display unit 18, a screen for asking the operator of the product registration device 1 to confirm is displayed.

The second display unit 19 is a display device that displays a screen according to a video signal input from a display control unit such as a CPU 11 or a graphic controller (not shown). The second display unit 19 displays a screen for the user (customer) to confirm.

The image pickup unit 20 is an image acquisition unit that acquires an image of a product. The image pickup unit 20 includes an image pickup element, a lens for forming a subject image on the image pickup element, and a drive circuit for driving the image pickup element. The image pickup unit 20 acquires an image corresponding to the subject image formed by the lens by reading out the electric charge accumulated in the image pickup element by the drive circuit. The image pickup unit 20 can acquire an image of a product when the product is present within the angle of view of the lens. The image pickup unit 20 acquires a plurality of images (frames) that are continuous in time by reading out images from the image pickup element by a drive circuit at a predetermined cycle. The image pickup unit 20 sequentially supplies the acquired images to the CPU 11.

The housing 21 includes the CPU 11, ROM 12, RAM 13, non-volatile memory 14, connection I / F 15, device I / F 16, operation unit 17, first display unit 18, second display unit 19, and image pickup of the product registration device 1. A housing that accommodates or supports the unit 20. The housing 21 is arranged, for example, on the accounting counter 3. The housing 21 supports the operation unit 17, the first display unit 18, and the image pickup unit 20 toward the operator side. Further, the housing 21 supports the second display unit 19 toward the customer side.

The POS terminal 2 executes the settlement process based on the total amount of the products registered in the accounting product list by the product registration device 1 and the deposit amount received from the user (customer). The POS terminal 2 is arranged on the drawer 5 arranged on the cash register counter 4, for example. The drawer 5 can be replaced with an automatic change machine. The drawer 5 is a drawer-type safe that opens and closes according to the control of the POS terminal 2. The POS terminal 2 includes a CPU 31, a ROM 32, a RAM 33, a non-volatile memory 34, a connection I / F35, a device I / F36, an operation unit 37, a third display unit 38, a fourth display unit 39, a printer 40, and a communication device. 41 is provided.

The CPU 31 is an arithmetic element (for example, a processor) that executes arithmetic processing. The CPU 31 performs various processes based on data such as a program stored in the ROM 32. The CPU 31 functions as a control unit capable of executing various operations by executing a program stored in the ROM 32.

The ROM 32 is a read-only non-volatile memory. The ROM 32 stores a program, data used in the program, and the like.

The RAM 33 is a volatile memory that functions as a working memory. The RAM 33 temporarily stores data and the like being processed by the CPU 31. Further, the RAM 33 temporarily stores a program executed by the CPU 31. The RAM 33 stores, for example, the above-mentioned accounting product list.

The non-volatile memory 34 is a storage medium capable of storing various information. The non-volatile memory 34 stores a program, data used in the program, and the like. The non-volatile memory 34 is, for example, a solid state drive (SSD), a hard disk drive (HDD), or other storage device. Instead of the non-volatile memory 34, a memory I / F such as a card slot into which a storage medium such as a memory card can be inserted may be provided.

The connection I / F35 is an interface for communicating with other devices. The connection I / F 35 is used for communication with the product registration device 1. The connection I / F35 is a connection terminal having a configuration conforming to some communication standard such as the USB standard or the RS-232C standard. The connection I / F35 may be a cable to which the product registration device 1 can be connected. Further, the connection I / F 35 may perform wireless communication with the product registration device 1 according to a standard such as Bluetooth (registered trademark) or Wi-fi (registered trademark).

The device I / F 36 is an interface for communicating between various functional blocks and the CPU 31. For example, the device I / F 36 is used for communication with the operation unit 37, the third display unit 38, the fourth display unit 39, the printer 40, and the like. The device I / F36 is a connection terminal having a configuration conforming to some communication standard such as the USB standard or the RS-232C standard.

The operation unit 37 generates an operation signal based on the operation of the operation member. The operating member is, for example, a touch sensor, a numeric keypad, a keyboard, or the like. The touch sensor is, for example, a resistance film type touch sensor, a capacitance type touch sensor, or the like. The touch sensor acquires information indicating a designated position within a certain area. The touch sensor is configured as a touch panel integrally with the third display unit 38, so that a signal indicating the touched position on the screen displayed on the third display unit 38 is input to the CPU 31.

The third display unit 38 is a display device that displays a screen according to a video signal input from a display control unit such as a CPU 31 or a graphic controller (not shown). On the third display unit 38, a screen for asking the operator of the POS terminal 2 to confirm is displayed.

The fourth display unit 39 is a display device that displays a screen according to a video signal input from a display control unit such as a CPU 31 or a graphic controller (not shown). The fourth display unit 39 displays a screen for the user (customer) to confirm.

The printer 40 is a device that prints and outputs the result of payment processing by the POS terminal 2 on a medium such as paper. The printer 40 prints the result of the payment process on the medium based on the control of the CPU 31.

The communication device 41 is a circuit for communicating with other electronic devices. For example, the communication device 41 communicates with the store computer 6 that aggregates the results of payment processing by the POS terminal 2. The communication device 41 includes, for example, a configuration for communicating according to a communication standard such as LAN, USB, Wi-fi, or Bluetooth.

(About payment processing on POS terminal 2)
The CPU 31 of the POS terminal 2 registers the product identification code, the product name, and the amount of the product requiring accounting from the product registration device 1 in the accounting product list of the RAM 33 each time. Further, the CPU 31 calculates the total amount of the products in the accounting product list. The CPU 31 causes the third display unit 38 and the fourth display unit 39 to display the product name of the product registered in the accounting product list, the amount of each product, and the total amount. Further, the CPU 31 recognizes the deposit amount, which is the amount received by the operator of the POS terminal 2 from the user, in response to the operation signal supplied from the operation unit 37. The CPU 31 performs payment processing based on the deposit amount and the total amount. When the payment process is executed, the CPU 31 causes the printer 40 to print the result of the payment process. Further, the CPU 31 controls the drawer 5 so as to open the drawer 5 when the payment process is executed. Furthermore, the CPU 31 transmits the result of the payment process to the store computer 6 via the communication device 41.

(About the product registration process in the product registration device 1)
The CPU 11 of the product registration device 1 recognizes and registers the product by executing the product registration program. The CPU 11 registers the product by transmitting the product identification code, the product name, and the amount of the recognized product to the POS terminal 2. The CPU 11 recognizes a product by any one of a first product recognition method, a second product recognition method, or a third product recognition method.

When the product is recognized by the first product recognition method, the CPU 11 acquires the product identification code by decoding the image of the product identification code captured by the image pickup unit 20, and obtains the product based on the acquired product identification code. recognize. That is, the CPU 11 acquires a product identification code from the image and functions as a product recognition unit that recognizes the product based on the acquired product identification code.

When the product is recognized by the second product recognition method, the CPU 11 calculates the feature amount from the image of the appearance of the product captured by the image pickup unit 20, and extracts the product candidate from the product recognition dictionary based on the calculated feature amount. do. For example, the CPU 11 calculates the similarity between the feature amount for each product in the product recognition dictionary and the feature amount extracted from the image, and extracts the product candidates in descending order of the calculated similarity. For example, the CPU 11 extracts a predetermined number of product candidates in descending order of the calculated similarity. Further, for example, the CPU 11 extracts a product whose calculated similarity exceeds a predetermined value as a product candidate. The CPU 11 displays the extracted product candidates on the first display unit 18 in a selectable state, and recognizes the products according to the operation of selecting the candidates in the operation on the displayed screen.

When the product is recognized by the third product recognition method, the CPU 11 calculates the feature amount from the image of the appearance of the product captured by the image pickup unit 20, and based on the calculated feature amount, one product is selected from the product recognition dictionary. Identify and recognize. For example, the CPU 11 calculates the similarity between the feature amount for each product in the product recognition dictionary and the feature amount extracted from the image, and recognizes the product for which the highest similarity is calculated. That is, the CPU 11 calculates the feature amount from the image even when the operation of reading the product identification code and selecting the product candidate is not performed, and the calculated feature amount and the feature amount for each product in the product recognition dictionary. It functions as an automatic confirmation unit that automatically recognizes products according to the degree of similarity with.

The CPU 11 of the product registration device 1 uses the above-mentioned first product recognition method, second product recognition method, and third product recognition method as methods used for product recognition according to the characteristics of the product to be registered. Switch. When the CPU 11 can acquire the product identification code, the CPU 11 recognizes the product by the first product recognition method. Further, when the product identification code cannot be acquired, the CPU 11 recognizes the product by either the second product recognition method or the third product recognition method.

The products processed by the product registration device 1 include, for example, a product to which a product identification code is assigned (first system product) and a product to which a product identification code (second system product) is assigned. The first line product is, for example, an artifact having a package. The second line product is, for example, fruits and vegetables without a package.

Since the second system product is not given a product identification code, it cannot be recognized by the first product recognition method. Therefore, the CPU 11 recognizes the second system product by the second product recognition method or the third product recognition method.

On the other hand, since the product identification code is attached to the first system product, it can be recognized by any of the first product recognition method, the second product recognition method, and the third product recognition method. However, since the third product recognition method is a method of automatically identifying the product having the highest degree of similarity between the feature amount calculated from the image and the feature amount of the product recognition dictionary, erroneous recognition may occur. be. It is desirable that the CPU 11 recognizes the first system product by the first product recognition method or the second product recognition method in order to prevent erroneous recognition.

Therefore, the CPU 11 of the product registration device 1 identifies whether the product is the first system product or the second system product, and if the product is the first system product, the third product recognition method is not performed and the second product is not performed. The product is recognized by the product recognition method of 1 or the product recognition method of the second. For this purpose, the CPU 11 calculates the feature amount from the image of the appearance of the product, and based on the calculated feature amount and the above-mentioned system identification dictionary, whether the product in the image is the first system product or the second system product. Perform system identification to identify whether it is. That is, the CPU 11 is a system identification unit that identifies based on the image whether the product in the image is the first system product to which the product identification code is attached or the second system product to which the product identification code is not assigned. Functions as.

The feature quantity used for system identification is, for example, a multidimensional vector. Specifically, the CPU 11 calculates the HOG feature amount from the image of the appearance of the product.

Since the first line product is an artificial product, it has a feature such as many straight lines. Since the second line product is fruits and vegetables, it has a feature that, for example, there are few straight lines. Therefore, the group of the feature amount calculated from the plurality of images of the first system product and the group of the feature amount calculated from the plurality of images of the second system product are biased in the multidimensional space.

The system identification dictionary includes a group of feature quantities calculated from a plurality of images of the above-mentioned first system product and a group of feature quantities calculated from a plurality of images of the second system product on the first system product side and the first. It has a hyperplane (discrimination surface) on a multidimensional space that is separated from the two-system product side. This identification surface is based on, for example, a group of feature quantities calculated from a plurality of images of the first system product and a group of feature quantities calculated from a plurality of images of the second system product, such as a support vector machine. It is preset by the learning model.

When the feature amount of the multidimensional vector generated from the image of the appearance of the product is plotted on the above-mentioned multidimensional space, the CPU 11 first produces the product according to which side of the discrimination surface the plotted points exist. Estimate whether it is a line product or a second line product. For example, the CPU 11 estimates that the product is the first system product when the plotted points are on the first system product side of the identification surface. Further, for example, when the plotted points are on the second system product side of the identification surface, the CPU 11 estimates that the product is the second system product.

As described above, when the CPU 11 acquires one image of a product from the image pickup unit 20, the CPU 11 determines whether or not the product identification code can be acquired from the image. When the product identification code cannot be obtained, the CPU 11 estimates whether the product is a first-system product or a second-system product from the appearance characteristics of the product. When the product is presumed to be the first system product, the CPU 11 does not perform the third product recognition method, but recognizes the product by the second product recognition method.

FIG. 3 shows an example of the operation of the product registration device 1 and the POS terminal 2. The product registration device 1 and the POS terminal 2 register and account for the products held in the image pickup range of the image pickup unit 20 by the operator. First, the CPU 11 of the product registration device 1 performs a product registration process based on the image acquired by the image pickup unit 20 (ACT 11). The CPU 31 of the POS terminal 2 calculates the total amount each time a product is newly registered in the accounting product list by the ACT 11.

The CPU 11 determines whether or not the product registration process has been completed (ACT 12). For example, when the operator performs an operation to execute the payment process on the POS terminal 2, the CPU 11 determines that the product registration process has been completed. The operation of executing the payment process is, for example, the operation of inputting the deposit amount and the operation of the button for executing the payment, or the operation of selecting some other payment method other than cash. The CPU 11 repeatedly executes the ACT 11 process until the product registration process is completed.

When the CPU 31 of the POS terminal determines that the product registration process has been completed, it executes the settlement process (ACT13) and ends the process. For example, the CPU 31 calculates the change amount by subtracting the total amount of the products registered in the accounting product list from the deposit amount, displays it on the third display unit 38, the fourth display unit 39, and the like, and also displays the drawer 5 open.

FIG. 4 shows an example of the product registration process of ACT11 of FIG. The CPU 11 of the product registration device 1 has a counter n that counts the number of images (frames) in which an object is detected, which is an image captured by the image pickup unit 20 in, for example, a RAM 13, and the product is a second system product. A counter m that counts the number of identified images (frames) and a counter m are stored. The CPU 11 may be configured to store the counter n and the counter m in the non-volatile memory 14.

When executing the product registration process, the CPU 11 first resets the values of the counter n and the counter m to "0" (ACT21).

The CPU 11 acquires an image for one frame captured by the image pickup unit 20 (ACT 22).

The CPU 11 performs a process of detecting an object in the image by image analysis on the acquired image. The CPU 11 determines whether or not an object has been detected (ACT 23). If no object is detected (ACT23, NO), the CPU 11 returns to ACT21.

When the object is detected (ACT23, YES), the CPU 11 detects the product identification code from the acquired image. The CPU 11 detects the product identification code from the image and determines whether or not the decoding is successful (ACT 24).

When the CPU 11 detects the product identification code from the image and determines that the decoding is successful (ACT24, YES), the CPU 11 extracts the product name and the amount of money associated with the decoded product identification code from the product recognition dictionary, and identifies the product. Register the code, product name, and amount in the POS terminal 2 Product registration is performed (ACT25), and the process is completed. That is, when the CPU 11 detects the product identification code from the image and determines that the decoding is successful, the CPU 11 recognizes the product by the first product recognition method and registers it in the POS terminal 2.

Further, when it is determined in the ACT 24 that the product identification code cannot be detected from the image or the decoding has failed (ACT 24, NO), the CPU 11 counts up the value of the counter n (for example, +1) (ACT 26).

The CPU 11 determines whether or not the value of the counter n is less than the preset value "N" (ACT27). When the CPU 11 determines that the value of the counter n is not less than the preset value "N", that is, the value of the counter n has reached "N" (ACT27, NO), the CPU 11 shifts to the ACT 31 described later. do.

When the CPU 11 determines that the value of the counter n is less than the preset value "N" (ACT27, YES), the CPU 11 performs system identification to identify the system of the product (ACT28).

The CPU 11 refers to the result of the system identification and determines whether or not the product is a second system product. (ACT29). When the CPU 11 determines that the product is not a second system product (ACT29, NO), the CPU 11 shifts to ACT31, which will be described later.

When the CPU 11 determines that the product is a second system product (ACT29, YES), the CPU 11 counts up the value of the counter m (for example, +1) (ACT30).

Next, the CPU 11 extracts product candidates (ACT31). For example, the CPU 11 calculates the feature amount from the image of the product captured by the image pickup unit 20, calculates the similarity between the calculated feature amount and the feature amount for each product in the product recognition dictionary, and calculates the similarity degree. Product candidates are extracted according to the above.

The CPU 11 determines whether or not the product candidate has been extracted in the ACT 31 (ACT 32). When the CPU 11 determines that the product candidate has not been extracted (ACT 32, NO), the CPU 11 proceeds to the process of the ACT 22 and acquires an image of the next frame.

When the CPU 11 determines that the product candidate has been extracted (ACT 32, YES), the CPU 11 displays the extracted product candidate on the first display unit 18 in a selectable state (ACT 33). For example, the CPU 11 displays the product name of the extracted product candidate on the first display unit 18 in a selectable state. Further, for example, an image for each product may be stored in the product recognition dictionary. In this case, the CPU 11 may display the extracted image of the product candidate on the first display unit 18 in a selectable state.

The CPU 11 determines whether or not any of the product candidates is selected on the product candidate screen displayed by the ACT 33 (ACT 34).

When the CPU 11 determines that the product candidate has been selected on the product candidate screen (ACT34, YES), the CPU 11 proceeds to ACT25 and registers the product identification code, product name, and amount of the selected product candidate in the POS terminal 2. ..

When the CPU 11 determines that the product candidate has not been selected on the product candidate screen (ACT34, NO), the CPU 11 determines whether or not the value of the counter m is "M" or less, which is a preset value (ACT35). .. When the value of the counter m exceeds "M", the CPU 11 determines that the product is likely to be a second system product (that is, fruits and vegetables). Further, when the value of the counter m is "M" or less, the CPU 11 determines that the possibility that the product is a second system product is not sufficient.

When the CPU 11 determines that the value of the counter m is "M" or less, which is a preset value (ACT35, YES), the process proceeds to the processing of the ACT 22 and the image of the next frame is acquired. That is, the CPU 11 continues to acquire the image, and continues the process until it becomes possible to detect the barcode, select the product candidate, or recognize the product by automatic confirmation.

When the CPU 11 determines that the value of the counter m is not "M" or less in the ACT 35, that is, when it is determined that the value of the counter m exceeds "M" (ACT35, NO), the CPU 11 executes automatic confirmation. It is determined whether or not to do so (ACT36). The CPU 11 switches whether to enable or disable the automatic confirmation according to the operation of the operation unit 37 or the information received from the POS terminal 2. For example, the CPU 11 stores a setting indicating whether or not automatic determination is valid in the RAM 13 or the non-volatile memory 14. The CPU 11 refers to this setting and determines whether or not to execute the automatic confirmation. When the CPU 11 determines that the setting is not to execute the automatic confirmation (ACT36, NO), the CPU 11 shifts to the ACT 22.

When the CPU 11 determines that the automatic confirmation is executed (ACT36, YES), the CPU 11 shifts to the ACT25. In this case, the CPU 11 identifies and recognizes one product based on the similarity calculated for each product so far. For example, the CPU 11 calculates the similarity between the feature amount calculated from the image and the feature amount for each product in the product recognition dictionary for each frame, and integrates the similarity of a plurality of frames acquired after the object is detected. , Identify and recognize the product corresponding to the highest similarity among the similarities for each integrated product. The CPU 11 registers the product identification code, product name, and amount of the recognized product in the POS terminal 2.

In the above configuration, when the frame rate of image acquisition of the image pickup unit 20 is 30 fps, the value "N" is 10, and the value M is "7", for at least 7 frames after the object is detected, Automatic confirmation is suspended.

As described above, when the product identification code cannot be read, the CPU 11 of the product registration device 1 indicates that the product is a first-system product such as an artificial product to which the product identification code is attached, or the product identification code is assigned. System identification is performed to identify whether it is a second system product such as a fruit or vegetable that is not present, for each frame. When the CPU 11 performs system identification over a plurality of frames and determines that the number m of the frames identified as the second system product exceeds the preset value "M", the product is the second system product. It is judged that the product is recognized by automatic confirmation. That is, the CPU 11 suspends the recognition of the product by automatic determination until it is determined that the product is the second system product. In other words, when the CPU 11 determines that the product is likely to be the first system product, it continues to suspend the recognition of the product by automatic confirmation, and the product recognition by reading the product identification code or the product by selecting the product candidate. Recognize the product by recognition. As a result, the product registration device 1 can prevent the product registration device 1 from recognizing the first system product to which the product identification code is attached by automatic confirmation. As a result, it is possible to provide a product registration device and a product registration program capable of recognizing and registering products with high accuracy.

(Second embodiment)
Next, the product registration device 1 according to the second embodiment will be described. Since the configuration is the same as that of the product registration device 1 according to the first embodiment, the description of the configuration will be omitted.

FIG. 5 shows an example of the product registration process according to the second embodiment of ACT11 of FIG. Since ACT 41 to ACT 54 are the same as ACT 21 to ACT 34 in FIG. 4, they will be briefly described.

When executing the product registration process, the CPU 11 first resets the values of the counter n and the counter m to "0" (ACT41).

The CPU 11 acquires an image for one frame captured by the image pickup unit 20 (ACT 42).

The CPU 11 performs a process of detecting an object in the image by image analysis on the acquired image. The CPU 11 determines whether or not an object has been detected (ACT43). If no object is detected (ACT43, NO), the CPU 11 returns to ACT41.

When the object is detected (ACT43, YES), the CPU 11 detects the product identification code from the acquired image. The CPU 11 detects the product identification code from the image and determines whether or not the decoding is successful (ACT44).

When the CPU 11 detects the product identification code from the image and determines that the decoding is successful (ACT44, YES), the CPU 11 extracts the product name and the amount of money associated with the decoded product identification code from the product recognition dictionary, and identifies the product. Register the code, product name, and amount in the POS terminal 2 Product registration is performed (ACT45), and the process is completed.

Further, when it is determined in the ACT 44 that the product identification code cannot be detected from the image or the decoding has failed (ACT 44, NO), the CPU 11 counts up the value of the counter n (for example, +1) (ACT 46).

The CPU 11 determines whether or not the value of the counter n is less than the preset value "N" (ACT47). When the CPU 11 determines that the value of the counter n is not less than the preset value "N", that is, the value of the counter n has reached "N" (ACT47, NO), the CPU 11 shifts to the ACT 51 described later. do.

When the CPU 11 determines that the value of the counter n is less than the preset value "N" (ACT47, YES), the CPU 11 performs system identification to identify the system of the product (ACT48).

The CPU 11 refers to the result of the system identification and determines whether or not the product is a second system product. (ACT49). When the CPU 11 determines that the product is not the second system product (ACT49, NO), the CPU 11 shifts to the ACT 51 described later.

When the CPU 11 determines that the product is a second system product (ACT49, YES), the CPU 11 counts up the value of the counter m (for example, +1) (ACT50).

Next, the CPU 11 extracts product candidates (ACT 51).

The CPU 11 determines whether or not the product candidate has been extracted in the ACT 51 (ACT 52). When the CPU 11 determines that the product candidate has not been extracted (ACT 52, NO), the CPU 11 proceeds to the process of the ACT 42 and acquires an image of the next frame.

When the CPU 11 determines that the product candidate has been extracted (ACT 52, YES), the CPU 11 displays the extracted product candidate on the first display unit 18 in a selectable state (ACT 53).

The CPU 11 determines whether or not any of the product candidates is selected on the product candidate screen displayed by the ACT 53 (ACT 54).

When the CPU 11 determines that the product candidate has been selected on the product candidate screen (ACT54, YES), the CPU 11 proceeds to ACT45 and registers the product identification code, product name, and amount of the selected product candidate in the POS terminal 2. ..

When the CPU 11 determines that the product candidate has not been selected on the product candidate screen (ACT54, NO), the CPU 11 determines whether or not the value of the counter m is "M" or less, which is a preset value (ACT55). .. When the value of the counter m exceeds "M", the CPU 11 determines that the product is likely to be a second system product (that is, fruits and vegetables). Further, when the value of the counter m is "M" or less, the CPU 11 determines that the possibility that the product is a second system product is not sufficient.

When the CPU 11 determines that the value of the counter m is "M" or less, which is a preset value (ACT55, YES), the CPU 11 determines whether or not a predetermined frame has elapsed (ACT56). That is, the CPU 11 determines whether or not the number of frames n, which is the elapsed frame since the object was detected from the image in the ACT 45, has reached the preset number of frames "X". When the CPU 11 determines that the predetermined frame has not elapsed (ACT56, NO), the CPU 11 proceeds to the process of the ACT 42 and acquires an image of the next frame. That is, the CPU 11 acquires the image of the next frame and continues the process.

When the CPU 11 determines that the value of the counter m is not "M" or less in the ACT 55, that is, when it is determined that the value of the counter m exceeds "M" (ACT35, NO), or when it is determined by the ACT 56. When it is determined that the frame has elapsed (ACT56, YES), it is determined whether or not to execute the automatic confirmation (ACT57). When the CPU 11 determines that the setting does not execute the automatic confirmation (ACT 57, NO), the CPU 11 shifts to the ACT 42.

When the CPU 11 determines that the automatic confirmation is executed (ACT57, YES), the CPU 11 shifts to the ACT 45. In this case, the CPU 11 identifies and recognizes one product based on the similarity calculated for each product so far. For example, the CPU 11 calculates the similarity between the feature amount calculated from the image and the feature amount for each product in the product recognition dictionary for each frame, and integrates the similarity of a plurality of frames acquired after the object is detected. , Identify and recognize the product corresponding to the highest similarity among the similarities for each integrated product. The CPU 11 registers the product identification code, product name, and amount of the recognized product in the POS terminal 2.

In the above configuration, for example, when the number of frames "X" is 30, the product identification code is not read, the product candidate is not selected, and the product is not identified as the second system product, the CPU 11 determines. The automatic confirmation process is suspended until the number of frames in which the object is detected reaches 30 frames.

As described above, the CPU 11 of the product registration device 1 detects the object even when the product identification code is not read, the product candidate is not selected, and the product is not identified as the second system product. When the number of frames (holding time) reaches the preset number of frames (predetermined time), the product is recognized by automatic confirmation. That is, the product registration device 1 performs product recognition by automatic determination when it is determined that the product is a second system product or when a certain period of time has elapsed. As a result, the product registration device 1 can adjust the time elapsed without the product recognition being executed. As a result, the product registration device 1 can efficiently register the product.

In the above embodiment, the product registration device 1 identifies whether the product in the image is the first system product or the second system product based on the identification surface defined on the multidimensional space by the support vector machine. Although it was explained that it is a configuration, it is not limited to this configuration. The product registration device 1 may be configured to identify whether the product is a first-system product or a second-system product by another method. For example, the product registration device 1 may be configured to identify whether the product is a first-system product or a second-system product by using a Deep Neural Network or the like.

Further, in the above embodiment, when the number m of the frame identified as the second system product exceeds the preset value "M", the product is the second system product. Although it was explained that the configuration is determined to be, the configuration is not limited to this configuration. In the product registration device 1, the product is first set based on the added value of the distance between the identification surface defined on the multidimensional space by the support vector machine and the point where the feature amount generated from the image is plotted on the multidimensional space. It may be configured to identify whether it is a system product or a second system product.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.
The following is a description of the scope of claims at the time of filing the application.
[C1]
The image acquisition unit that acquires the image of the product, and
A memory that stores a product recognition dictionary that has a feature amount for each product,
An automatic confirmation unit that calculates a feature amount from the image and recognizes the product according to the degree of similarity between the calculated feature amount and the feature amount for each product in the product recognition dictionary.
Based on the image, it is determined whether the product in the image is a first-series product to which a product identification code for identifying the product is attached or a second-series product to which the product identification code is not assigned. System identification unit and
When the product is identified as the second system product, the product is recognized by the automatic confirmation unit, and when the product is identified as the first system product, the recognition of the product by the automatic confirmation unit is suspended. A control unit that controls the automatic confirmation unit and
The product registration department that registers the product recognition results, and
A product registration device equipped with.
[C2]
The product registration device according to C1, wherein the control unit recognizes the first system product by the automatic confirmation unit when the holding time reaches a predetermined time.
[C3]
A product recognition unit that acquires the product identification code from the image and recognizes the product based on the acquired product identification code is further provided.
When the product is identified as the first system product and the product identification code is acquired, the control unit controls the product recognition unit to recognize the product by the product recognition unit C1 or 2. Product registration device described in.
[C4]
The first line product is an artifact having more straight lines than the second line product.
The product registration device according to any one of C1 to 3, wherein the second system product is a fruit or vegetable having fewer straight lines than the first system product.
[C5]
It is a product registration program executed by a product registration device including an image acquisition unit for acquiring an image of a product and a memory for storing a product recognition dictionary having a feature amount for each product.
The product registration device
An automatic confirmation unit that calculates a feature amount from the image and recognizes the product according to the degree of similarity between the calculated feature amount and the feature amount for each product in the product recognition dictionary.
Based on the image, it is determined whether the product in the image is a first-series product to which a product identification code for identifying the product is attached or a second-series product to which the product identification code is not assigned. System identification unit and
When the product is identified as the second system product, the product is recognized by the automatic confirmation unit, and when the product is identified as the first system product, the recognition of the product by the automatic confirmation unit is suspended. A control unit that controls the automatic confirmation unit and
The product registration department that registers the product recognition results, and
Product registration program to operate.

1 ... Product registration device, 2 ... POS terminal, 3 ... Accounting counter, 4 ... Cash register counter, 5 ... Drawer, 6 ... Store computer, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Non-volatile memory, 15 ... Connection I / F, 16 ... Device I / F, 17 ... Operation unit, 18 ... First display unit, 19 ... Second display unit, 20 ... Imaging unit, 21 ... Housing, 31 ... CPU, 32 ... ROM, 33 ... RAM, 34 ... non-volatile memory, 35 ... connection I / F, 36 ... device I / F, 37 ... operation unit, 38 ... third display unit, 39 ... fourth display unit, 40 ... printer, 41. … Communication device.

Claims (7)

The image acquisition unit that acquires the image of the product, and
A memory that stores a product recognition dictionary that has a feature amount for each product,
Based on the image, it is determined whether the product in the image is a first-series product to which a product identification code for identifying the product is attached or a second-series product to which the product identification code is not assigned. System identification unit and
A first product recognition unit that acquires the product identification code from the image and recognizes the product based on the acquired product identification code.
A second method in which a feature amount is calculated from the image, product candidates are output in a selectable state based on the calculated feature amount and the feature amount for each product in the product recognition dictionary, and the product is recognized according to the selection. Product recognition department and
When the product is identified as the first system product, the recognition of the product by the second product recognition unit is suspended, and when the product identification code is acquired during the suspension, the first product recognition unit The first product recognition unit and the second product recognition unit are controlled so as to recognize the product according to the above, and when the product is identified as the second system product, the second product recognition unit determines the product. The control unit to recognize and
The product registration department that registers the product recognition results, and
A product registration device equipped with. The product registration according to claim 1, wherein the control unit recognizes the product by the second product recognition unit when the holding time reaches a predetermined time and the product is identified as the first system product. Device. A third product recognition unit that calculates a feature amount from the image and recognizes a product having the highest degree of similarity between the calculated feature amount and the feature amount for each product in the product recognition dictionary as a product to be registered by the product registration unit. Further equipped,
The product according to claim 1 or 2, wherein the control unit recognizes the product by the third product recognition unit when the holding time reaches a predetermined time and the product is identified as the second system product. Registration device. The second product recognition unit calculates the similarity between the feature amount for each product in the product recognition dictionary and the calculated feature amount from the image, and selects a predetermined number of the product candidates in descending order of the calculated similarity. The product registration device according to any one of claims 1 to 3 to be output. The second product recognition unit calculates the similarity between the feature amount for each product in the product recognition dictionary and the calculated feature amount from the image, and the product whose calculated similarity exceeds a predetermined value is described. The product registration device according to any one of claims 1 to 4, which is output as a product candidate. The first line product is an artifact having more straight lines than the second line product.
The product registration device according to any one of claims 1 to 5, wherein the second system product is a fruit or vegetable having fewer straight lines than the first system product. It is a product registration program executed by a product registration device including an image acquisition unit for acquiring an image of a product and a memory for storing a product recognition dictionary having a feature amount for each product.
The product registration device
Based on the image, it is determined whether the product in the image is a first-series product to which a product identification code for identifying the product is attached or a second-series product to which the product identification code is not assigned. System identification unit and
A first product recognition unit that acquires the product identification code from the image and recognizes the product based on the acquired product identification code.
A second method in which a feature amount is calculated from the image, product candidates are output in a selectable state based on the calculated feature amount and the feature amount for each product in the product recognition dictionary, and the product is recognized according to the selection. Product recognition department and
When the product is identified as the first system product, the recognition of the product by the second product recognition unit is suspended, and when the product identification code is acquired during the suspension, the first product recognition unit The first product recognition unit and the second product recognition unit are controlled so as to recognize the product according to the above, and when the product is identified as the second system product, the second product recognition unit determines the product. The control unit to recognize and
The product registration department that registers the product recognition results, and
Product registration program to operate.

Images