JP7519633B2 - Narrowing processing system - Google Patents

Description

The present invention relates to a narrowing-down processing system for narrowing down the comparison targets when performing a comparison process to identify products shown in image information.

Knowing where products are displayed in a store is very important information from the perspective of a product's marketing strategy. For this reason, product manufacturers regularly conduct surveys to see where products are displayed in stores and in what quantities.

There are various methods for this survey, but one method is for an operator to actually visit the store and check the display location of each product one by one. This method is highly reliable, but since the operator must visually check, the workload on the operator is extremely heavy.

Therefore, there is a method to determine whether the product being investigated is on display by determining whether it is captured in image information taken of the product display shelves.

When determining whether a product is shown in image information in this way, it is common to carry out image matching processing between image information in which the product to be processed has been photographed and registered in advance and image information of the product obtained by photographing a display shelf. For example, the following Patent Document 1 discloses a system that identifies the products handled by a vending machine by using image recognition technology on image information photographed of the vending machine based on the registered images of each product.

JP 2014-191423 A

The specific process described in Patent Document 1 involves photographing a vending machine from multiple directions, aligning the relative positions of each image, and then superimposing the images to generate a composite image. The composite image is then compared with registered images that represent products that may be displayed in the vending machine, thereby identifying the products that the vending machine carries.

When comparing a synthetic image with a registered image that represents a product (matching process), the process is generally carried out by using the feature values of each image. However, when there are many products to be compared, the number of registered images also increases, posing a problem that the calculation time required for the comparison process (matching process) becomes enormous.

In other words, conventionally, the identification information of products estimated to be in a captured image is determined by comparing (matching) image information of an area where a product is estimated to be present with reference image information for each product. However, since the comparison process (matching) involves comparing the similarity between the image features of the image information of the area and the image features of the reference image information of a very large number of products (e.g., several thousand to tens of thousands) to be compared, it places an extremely large processing load on the computer and requires a long processing time. Furthermore, there is a limit to the accuracy of this comparison process (matching process), and the larger the population of reference image information for products, the lower the accuracy becomes.

Therefore, rather than identifying one product through a comparison process (matching process), it is possible to select and present multiple candidate products that are likely to be correct, and ultimately have a human being identify the product that is estimated to be in the area. In this case, since a human makes the final decision, the accuracy of identifying products is improved, but unless candidate products are presented appropriately, especially at the top of the list, the efficiency of the work will be significantly reduced.

In view of the above problems, the inventor invented a narrowing-down processing system that can reduce the processing load and processing time of a computer by narrowing down the reference image information of products to be compared in the comparison process (matching process) and then performing the comparison process (matching process). In addition, the inventor invented a narrowing-down processing system that can present the correct product at the top when presenting candidates.

The first invention is a narrowing down processing system for identifying products shown in image information, the narrowing down processing system having an image information processing unit that identifies a face area in the image information to be compared with sample information of the product, a narrowing down processing unit that narrows down products to be compared by using the judgment results of predetermined narrowing down factors for the face area and pre-stored narrowing down factor information of the product, and a comparison processing unit that compares the sample information of the narrowed down products with image information and/or its features of the face area, the narrowing down processing unit judging whether the image information and/or its features of the face area satisfy the conditions set for one or more predetermined narrowing down factors, and narrows down products having the same narrowing down factor information as the result of the judgment as the products to be compared.

By performing processing as in the present invention, it is possible to reduce the image information of the face area and the sample information of the product to be compared with the features, thereby reducing the processing load and processing time of the computer.

The first invention can be realized by loading and executing the program of the present invention into a computer. That is, the program causes the computer to function as an image information processing unit that identifies a face area to be compared with sample information of a product in image information, a filtering processing unit that narrows down products to be compared by using a judgment result of a predetermined filtering factor for the face area and pre- stored filtering factor information of the product, and a comparison processing unit that compares the sample information of the narrowed-down products with image information and/or feature amounts of the face area, and the filtering processing unit judges whether the image information and/or feature amounts of the face area satisfy a condition set for the filtering factor for one or more predetermined filtering factors, and narrows down products having the same filtering factor information as the result of the judgment as products to be compared .

By using the narrowing down processing system of the present invention, when identifying products shown in image information obtained by photographing display shelves, etc., it is possible to narrow down the comparison targets and perform comparison processing (matching processing), thereby reducing the processing load and processing time of the computer. In addition, when presenting candidates, it is possible to present the correct product at the top.

1 is a block diagram illustrating an example of the overall processing function of a narrowing-down processing system according to the present invention. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the narrowing-down processing system of the present invention. 10 is a flowchart showing an example of a processing process in the narrowing down processing system of the present invention. FIG. 2 is a diagram illustrating an example of a product information storage unit. FIG. 13 is a diagram showing a concept of processing subsequent to the narrowing down process in the narrowing down processing unit. FIG. 13 is a diagram illustrating an example of a product information storage unit including a collection of misjudgment examples in the second embodiment. 13 is a flowchart illustrating an example of a processing process in the narrowing down processing system according to the second embodiment. FIG. 13 is a diagram illustrating a concept of processing after the narrowing down process in the narrowing down processing unit in the second embodiment.

Figure 1 shows a block diagram of an example of the overall processing function of the narrowing down processing system 1 of the present invention. The narrowing down processing system 1 uses a management terminal 2 and an input terminal 3.

The management terminal 2 is a computer that realizes the core processing functions of the selection processing system 1. The input terminal 3 is a terminal that acquires image information obtained by photographing store display shelves, etc. It may also be possible to photograph and acquire products, etc., that will be used as samples in the comparison processing (matching processing) described below.

The management terminal 2 and input terminal 3 in the narrowing down processing system 1 are realized using a computer. FIG. 2 shows an example of a computer hardware configuration. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

If the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be configured as an integrated unit. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

A touch panel display is a device that combines the functions of a display device 72 and an input device 73 in that input can be made directly on the display using a specified input device (such as a touch panel pen) or a finger.

In addition to the above devices, the input terminal 3 may be equipped with a photographing device such as a camera. A portable communication terminal such as a mobile phone, smartphone, or tablet computer can also be used as the input terminal 3. The input terminal 3 photographs image information (photographed image information or product image information, described below) using visible light or the like with the photographing device.

The functions of each means in the present invention are only logically distinct, and may be physically or practically the same area. The order of processing in each means of the present invention may be changed as appropriate. Also, some of the processing may be omitted. Also, some or all of the functions in the management terminal 2 may be executed in the input terminal 3.

The narrowing down processing system 1 has an image information input reception processing unit 20, an image information storage unit 21, an image information processing unit 22, a product information storage unit 23, a narrowing down processing unit 24, a comparison processing unit 25, and a product selection reception processing unit 26.

The image information input reception processing unit 20 receives input of image information (photographed image information) captured by the input terminal 3, and stores it in the image information storage unit 21, which will be described later. For example, input of photographed image information of a store's display shelves is received, and stored in the image information storage unit 21. In addition to the photographed image information, input from the input terminal 3 may also be received including the date and time of shooting, information indicating the subject of the image, for example store identification information such as the store name, information identifying the display shelves, information identifying the shelf levels of the display shelves, and image information identification information that identifies the image information.

The image information storage unit 21 stores captured image information and other information received from the input terminal 3 in association with the captured image information.

The image information processing unit 22 performs an orientation process for the photographed image information received by the image information input reception processing unit 20 to orthogonalize the photographed image information, and a face process for identifying an area (face area) from the photographed image information in which a comparison process (matching process) with the specimen information is to be performed.

When photographing display shelves or products, it is difficult to photograph them facing the right way, so the orthogonalization process is a process that corrects the image to face the display shelves or products facing the right way. The optical axis of the lens of the camera is aligned with the perpendicular direction of the plane of the subject, and the image information is transformed so that it is the same as when the image is photographed from a sufficiently far away. One example of this type of correction process is keystone correction. Note that if the image information is distorted, distortion correction processing may be added.

Note that if the captured image information was captured in a normal position or if there is no distortion, there is no need to perform the normalization process or distortion correction process.

The captured image information may be any image information that is the subject of processing in the present invention. Image information after correction processing, such as normalization processing and distortion correction processing, is performed on the captured image information is also included in the captured image information.

In addition, when multiple images are taken of the subject, the image information processing unit 22 may execute a process of combining the images into one piece of image information, and perform a normalization process and a face process on the image information that has been combined. Alternatively, the image information processing unit 22 may execute a normalization process on each piece of captured image information, combine the combined images into one piece of image information, and perform a face process on the combined images. A publicly known method may be used for the process of combining multiple pieces of image information into one piece of image information. The image information after the process of combining the images into one piece of image information has been executed is also included in the captured image information.

Face processing involves identifying an area (face area) in the captured image information for performing comparison processing with the sample information described below. When identifying products that appear in captured image information of a product display shelf, the area of the product displayed on the display shelf or the area of the product label is identified as the face area. If the product is a bottled drink, the area of the product label is set as the face area, and if the product is a boxed product (such as sweets), the entire product packaging is set as the face area; for example, the face area can be set appropriately depending on the product. Note that even if the product is a bottled drink, the outline of the product or a rectangular area including that outline may be set as the face area.

There are various methods for identifying face areas, and any method can be set according to the characteristics of the product. When identifying the area of a product's label as the face area from the captured image information of a display shelf, the product area can be identified, for example, by identifying thin, narrow vertical shadows that occur between products in the area between shelves of the display shelf (shelf area), identifying repeating patterns in the image, identifying steps on the top edge of the package, identifying dividing positions based on constraints such as the product width being the same, and so on. Then, from within the product area, a specified rectangular area is identified as the label area, and this area is identified as the face area.

The method of identifying the face area can be any method depending on the product category and product form, and is not limited to the above. In addition, the operator may input corrections to the automatically identified face area. Furthermore, input of the position of the face area may be accepted from the operator.

The image information processing unit 22 may identify the face area using deep learning. In this case, the captured image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the face area is identified based on the output value. In addition, the learning model may be one in which various captured image information is provided with the face area as correct answer data.

The following explanation will cover face processing using deep learning to identify face areas.

In face processing, it is preferable to perform processing to identify attribute information of products in the face area. Processing to identify attribute information of products in the face area can be performed by executing the following processing on the image information in the face area. For example, as described above, in addition to identifying the shape and aspect ratio of the product depicted in the face area, character strings in the face area can be identified by performing OCR recognition processing on the image information of the face area, and color information (color information including hue, brightness, and saturation) and the component ratio of its area of the image information in the face area can be identified.

In addition, in the face processing, a logo (manufacturer logo, brand logo, etc.) in the face area may be identified. In the logo identification process, the above-mentioned photographed image information may be input to a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many intermediate layers is optimized, and the logo (manufacturer logo, brand logo, etc.) in the face area may be identified based on the output value. In addition, as the learning model, a manufacturer logo or brand logo in the face area of various photographed image information may be given as correct answer data. Furthermore, in addition to the logo identification process using a neural network, a feature amount (which may be multiple such as color information or local feature amount) in the correct answer data (which is given logos in the face area of various photographed image information) given as the above-mentioned learning model and a feature amount other than the logo in the face area of the photographed image information are extracted in advance, and a judgment module (judgment machine) such as an SVM (support vector machine) is configured as a positive example and a negative example, respectively. Then, each point of the image information of the face region, for example, by dividing the image information of the face region into a mesh of a specified size and inputting the points within that mesh, is made to a judgment module such as an SVM, and the module judges whether each point belongs to the logo region or not. Based on the result of this judgment, the area surrounding the mesh area judged to belong to the logo region is identified as the logo.

In addition, in face processing, logo identification processing in the face area can be performed by pattern matching the image information (or its features) of the face area with the image information (or its features) of the logo.

The process of identifying the logo from the face area is not limited to the above method, and other methods may be used. Also, the above method and other methods may be combined to identify the logo.

The image information processing unit 22 cuts out the face area identified as described above. Cutting out the face area may mean actually cutting out the face area identified from the captured image information, or it may mean setting that area as a processing target for the comparison process described below.

When cutting out a face area, the face area identified by each of the above processes may be cut out as is, or the face area may be identified by multiple methods, and the face area to be cut out may be identified using the results of the face area identified by each method. For example, a method may be used in which the area of a product is identified by identifying a thin, narrow shadow that occurs between products in the area between shelves of a display shelf (shelf area), and a predetermined rectangular area may be identified from that area, or a method may be used to identify a face area by deep learning, and the results of the face areas identified by each of these methods may be combined to identify the face area to be finally cut out by a predetermined calculation. The method of identifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

The product information storage unit 23 is a data collection that comprehensively stores information about products to be compared, and stores, for example, product identification information (product name, model number, product code such as JAN code, etc.), as well as associated information, attribute information, narrowing-down factor information, sample information, etc. of the product in association with each other. An example of the product information storage unit 23 is shown diagrammatically in Figure 4.

The data collection only needs to store information on multiple products to be compared. The associated information includes the product manufacturer's identification information, product classification, content volume, individual size, price information, container form, shelf size, product release date, and production/sales end flags. This associated information can be used from the JICFS data collection described below, for example, but missing information can be added by the operator one by one. The JICFS data collection is a JAN code integrated product information database, a centralized database of JAN codes and associated product information.

The product information storage unit 23 can use, for example, a database of JICFS or ISBN codes of books, but is not limited to these, and any data collection that comprehensively stores products to be compared will suffice. Note that "comprehensive" does not necessarily mean that all products in the genre are stored, but rather, for example, a data collection in which an industry group or a certain organization stores multiple products from the perspective of product sales and management, or a data collection that can be searched during comparison processing in any processing system will suffice. Furthermore, if there are products that are not stored in a publicly known database such as JICFS, the database after adding information on the missing products may be used as a data collection. For example, information on private brand products that are not registered in JICFS may be added to JICFS, and the database may be stored as a data collection in the product information storage unit 23.

The product information storage unit 23 may store product attribute information, specimen information, etc. in association with a data collection such as JICFS, but may not be configured as a single database, but may be configured as multiple associated databases.

Attribute information includes information on the appearance, rating, and price range of a product. Product appearance includes package shape, logo, text, layout information, color information, and texture/pattern. Package shape includes the shape of the product (an attribute related to the product's external appearance (shape), such as bottle, can, box, bag, bottle, etc.), actual size (product height, width), and aspect ratio (product height, width ratio). Logo is the manufacturer logo or brand logo attached to the product. Text is the text attached to the product, including information on the font and color. Layout information includes information on the size, color, and position of a logo, text, figure, illustration, etc. on the product. Rating includes the product's container size (for example, 350 ml, 500 ml, etc.), packaging (six-pack, etc.), ingredient labeling (1 mg, 5 mg, etc.), and function labeling. Price range is information on the product's sales price range, and indicates the range of prices generally expected for the product. The price range may be indicated by a lower limit and an upper limit, or may be indicated as a percentage above or below a certain price (such as a suggested retail price or list price), or a number of yen above or below. Note that attribute information may be stored as additional information about the product.

Attribute information may also include product sales information, such as the top 10 in the market, sales information for specific products in the store's POS system, and regionally limited products.

When repeatedly conducting in-store surveys, attribute information such as product information that was in that sales area, display shelf, or display shelf level the previous time, number of faces, display position, etc. may be included. Furthermore, in the case of the same chain, the product lineup is often similar, so this information may be added to the attribute information. When using this previous information, new products and other products that are being displayed for the first time have already been narrowed down, so adding this information to construct the attribute information makes it possible to achieve high accuracy in processing.

The narrowing-down factor information is information indicating whether the product satisfies the narrowing-down factor in the narrowing-down processing unit 24 described later, or in what state it is. For example, when the narrowing-down factor is the aspect ratio described later, information indicating whether the product is vertical or horizontal, or vertical, horizontal, square, etc. is stored. When the condition for the narrowing-down factor is whether it is satisfied, it may be stored as a binary value such as "T" for satisfied and "F" for not satisfied. In addition, when there are three or more options such as vertical, horizontal, square, etc., it may be stored as corresponding alphanumeric characters or symbols such as "1" for vertical, "2" for horizontal, and "3" for square. In addition, when the narrowing-down factor information cannot be specified due to various reasons such as the shape of the product being unknown because it is a new product, information indicating that may be stored, such as information such as "X". In this case, it is possible to process the condition as being satisfied regardless of the value of the narrowing-down factor as a wildcard.

The sample information is the image information (product image information) of the product and/or its feature information to be compared with the image information and/or feature information of the face region in the comparison processing unit 25 described later. The sample information corresponds to product identification information, for example, product code such as product name, model number, and JAN code, product attribute information, and selection factor information. The product image information may be image information of the entire product, or may be image information of a part of the product, for example, the label part if the product is a PET bottle drink. Note that when feature information of the product image information is used as the sample information, there is no need to extract the feature each time processing is performed.

The product information storage unit 23 may store one piece of sample information for one product, or may store sample information from multiple angles for one product, such as the front, back, top, bottom, and both sides, in order to capture a surface that is not visible when photographing the product from the front. Also, if one product has multiple exterior packaging (packages, etc.), the product may store sample information for each of the exterior packaging.

When feature information is used as sample information, feature information for some or all areas of the product image information is generated and stored by executing processing similar to that in the comparison processing unit 25 described below based on some or all of the product image information obtained by photographing the product to be used as a sample. The area in the product image information for generating features may be extracted automatically, or may be specified by the operator from the product image information whose input has been accepted.

As described above, the product information storage unit 23 stores the identification information, associated information, attribute information, narrowing-down factor information, sample information, etc. of the products to be compared in association with each other.

The narrowing down processing unit 24 executes processing in the comparison processing unit 25 described below to narrow down the sample information of products to be compared with the image information of the face area to be processed and/or its feature quantities. Specifically, for one or more predetermined narrowing down factors, it determines whether the image information of the face area and/or its feature quantities satisfy the conditions set for the narrowing down factors, thereby narrowing down the products to be compared.

Figure 5 shows a conceptual diagram of the processing after the narrowing down process in the narrowing down processing unit 24.

Narrowing factors include, but are not limited to, the aspect ratio of the face area, the brightness and saturation of the dominant color in the face area, and the hue of the dominant color in the face area.

For example, if the aspect ratio of a face area is a filtering factor, thresholds are set for whether the face area is vertically long or not; if the brightness/saturation of the dominant color of the face area is a filtering factor, thresholds are set for whether the face area is a bright image or not, whether the dominant color of the face area is reddish or not, etc. Then, for the face area to be processed, it is determined whether it satisfies the conditions of each filtering factor. Here, there are two options, whether the condition is satisfied or not, but it is also possible to divide it into three or more options, such as vertically long, horizontally long, or the same length and width, or reddish, blueish, and others.

In the above, the aspect ratio of the image information in the face area, the brightness and saturation of the dominant color, and the hue of the dominant color are each converted into a value, and based on that value, it can be determined whether or not the threshold value of each narrowing-down factor is satisfied. In addition, for the dominant color, the color at each pixel of the image information is identified, and the most common color among them can be identified as the dominant color.

Then, the filtering processing unit 24 identifies sample information of products that have the same results (filtering factor information) as the judgment results for each filtering factor for the face area to be processed from the product information storage unit 23.

When the product selection reception processing unit 26, which will be described later, receives that there is no correct product among the candidate products output in the comparison processing in the comparison processing unit 25, which will be described later, the narrowing down processing unit 24 performs narrowing down processing again by identifying products to be compared from the product information storage unit 23 in order to perform the next comparison processing in the comparison processing unit 25. The narrowing down processing again can be performed as follows.

If the narrowing down factors are divided into three levels, and each narrowing down factor is divided into two levels, that is, whether or not it is satisfied, then the narrowing down factor patterns for each product are divided into eight ways (= ^2-3 ways). For example, narrowing down factor 1 is the aspect ratio of the image information in the face area, narrowing down factor 2 is the brightness and saturation of the dominant color, and narrowing down factor 3 is the hue of the dominant color, and the narrowing down factors are set in order of empirical or statistical accuracy. Here, the priority is set as follows: narrowing down factor 1 is the highest, followed by narrowing down factor 2, narrowing down factor 3, and so on. If the narrowing down factor information is stored in the product information storage unit 23 for each product, with "T" indicating that each narrowing down factor is satisfied and "F" indicating that each narrowing down factor is not satisfied, then one of the eight ways, TTT, TTF, TFT, TFF, FTT, FTT, FTF, FFT, and FFF, is stored in association with each product as the narrowing down factor information.

In this case, first, the narrowing down processing unit 24 identifies products from the product information storage unit 23 that have the same narrowing down factor information as the narrowing down factor judgment result for the face area to be processed, and since the product could not be identified even by performing the comparison process, from the second time onwards, the result of the lowest narrowing down factor among the narrowing down factor judgment results for the face area to be processed is changed, and products that have the same narrowing down factor information as the changed judgment result are narrowed down as products to be compared in the comparison processing unit 25, and sample information for those products is identified.

For example, if the screening processing unit 24 first determines that the screening factor for the face area to be processed is TTT, the product having the same screening factor information as the determination result TTF in which the determination result for the lowest screening factor (screening factor 3) is changed from T to F is identified as the product to be compared in the next comparison processing unit 25, and the sample information is identified from the product information storage unit 23.

Then, the comparison processing unit 25 described below processes the sample information of the narrowed-down products (products whose narrowing-down factor information is TTF) in the same manner as the comparison processing described below.

If, in this case, there is no correct product among the products output by the product selection reception processing unit 26, the next candidate product is identified from the product information storage unit 23. That is, as a result of the judgment of the narrowing down factor information, a product having the same narrowing down factor as TFT, in which the judgment result of the second narrowing down factor from the bottom is changed from T to F, is identified as the product to be subjected to the next comparison process in the comparison processing unit 25, and the sample information is identified from the product information storage unit 23, and the comparison process in the comparison processing unit 25 is executed in the same manner as described above.

Even in this case, if the product selection reception processing unit 26 receives that there is no correct product among the products output by the comparison processing unit 25, the filtering process for narrowing down the products to be compared in the comparison processing unit 25 is executed as described above, and the products are identified from the product information storage unit 23. That is, as a result of the judgment of the filtering factor information, a product having the same filtering factor as TFT in which the judgment result of the second lowest filtering factor (filtering factor 2) is changed from T to F is identified as the product to be subjected to the next comparison processing in the comparison processing unit 25, and the sample information is identified from the product information storage unit 23, and the comparison processing in the comparison processing unit 25 is executed in the same manner as described above.

In this way, the second and subsequent narrowing down processes are executed by changing the narrowing down factor information identified in the first narrowing down process based on the priority of the narrowing down factors, and identifying the corresponding products from the product information storage unit 23.

If there are three or more narrowing down factors, you can change the value in the lowest priority order in order, and after changing all the values, go back to the next higher priority order and change the values in that priority order in the same way.

Furthermore, it is preferable to arrange the narrowing down factors in the narrowing down processing unit 24 so that factors with a higher average probability of being the correct answer are processed first. In other words, narrowing down factors that are more likely to be mistaken are arranged so that they are processed lower (later). This increases the likelihood that the correct answer will be included in a group of errors caused by narrowing down factors that are arranged lower and searched earlier, and the correct answer will be presented to the operator earlier. This improves the efficiency of the operator's work. Specifically, when constructing the narrowing down processing system 1 of the present invention, it is preferable to actually narrow down a certain number of image information (such as image information of a display shelf) for which the correct answer is known, measure the accuracy rate, and determine the order (priority) of the narrowing down processing based on the accuracy.

For example, when the narrowing down factors are the aspect ratio of the image information in the face area, the brightness of the dominant color, and the hue of the dominant color, if the accuracy of the aspect ratio of the image information in the face area is 98%, the accuracy of the brightness of the dominant color is 95%, and the accuracy of the hue of the dominant color is 93%, then the narrowing down factors are set in descending order of accuracy as narrowing down factors 1, 2, and 3.

The comparison processing unit 25 executes a comparison process (matching process) between the sample information of the product identified by the narrowing down processing unit 24 and the image information of the face area to be processed and/or its features. For example, the feature in the image information of the face area is extracted. Then, by executing a comparison process between the extracted feature and the sample information such as the feature of the product identified by the narrowing down processing unit 24, the similarity between the feature of the image information of the face area and the sample information is determined. In other words, the similarity between the feature of the image information of the face area and the sample information is calculated, and by sorting based on the similarity, a predetermined number or a predetermined range of product identification information, sample information, etc. from the top are output as candidates. This output may be performed on the display device 72 of the management terminal 2, or may be output to the display device 72 of a computer operated by an operator.

The comparison process may involve not only matching the entire image information of the face region with the sample information, but also extracting the features of part of the image information of the face region, for example the image information of a feature region (block), and performing a comparison process with the features of the sample information.

Comparison processing may also be performed using methods other than feature quantities.

The comparison processing unit 25 may use deep learning to perform the comparison process with the sample information. In this case, all or part of the image information of the face area may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and similar sample information may be identified based on the output value. As a learning model, one in which sample information of various products is given as correct answer data may be used.

The product selection reception processing unit 26 receives from the operator a selection of whether or not there is a correct product among the products output as candidates for products in the face area to be processed, and if there is a correct product, which of the output candidates is the correct product.

When the product selection reception processing unit 26 receives a selection of a correct product from among the output candidate products, it identifies the selected product as the face area to be processed and the product depicted in that face area, and stores the identified product's product identification information, product name, product code, etc. in a specified storage unit in association with the product. The display position of the product can be specified by the vertical and horizontal order of the face area, or it can be specified as a coordinate in the captured image information.

In addition, the specific storage unit mentioned above may store information about which shelf in which store the identified product was displayed on.

If the product selection reception processing unit 26 does not accept a selection that is the correct answer among the products output as candidates, the second or subsequent narrowing down processing is executed in the narrowing down processing unit 24 described above. That is, for the judgment result of the initially judged narrowing down factor, starting from the narrowing down factor with the lowest priority, a product having the same narrowing down factor as the judgment result after the judgment result is changed is identified as the product to be subjected to the comparison processing of the next comparison processing unit 25, and its sample information is identified from the product information storage unit 23.

Next, an example of the processing process in the narrowing down processing system 1 of the present invention will be described with reference to the flowchart in Figure 3. In this embodiment, a case will be described in which a store's display shelves are photographed and the products on the shelves are identified. Therefore, the photographed image information is the store's display shelves on which the products are displayed, and the products in the sample information are the products that may be displayed. In addition, the product information storage unit 23 stores a JICFS data collection, and stores the product name, product identification information, associated information, attribute information, narrowing down factor information, etc.

First, the operator photographs the product to be compared from a frontal position, and accepts input of the photographed product image information from the input terminal 3. The accepted input product image information is then stored in the product information storage unit 23 in association with the product. In addition, feature quantities are generated for some or all areas of the accepted input product image information, and are stored as sample information in association with the product stored in the product information storage unit 23. The area for generating feature quantities may be extracted automatically, or may be specified by the operator from the accepted input product image information.

Furthermore, the product information processing unit accepts input from the operator regarding attribute information such as product appearance, ratings, and price range that is not registered in the JICFS data collection, and stores the information in the product information storage unit 23 as attribute information in association with the product. At this time, in addition to the operator making the input, the unregistered information may be identified using deep learning and stored in the product information storage unit 23. In this case, the product image information may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the information to be registered from the attribute information may be identified based on the output value. In addition, the learning model may be one in which the information to be registered from the attribute information is given as correct answer data to various product image information.

As described above, processing of the products to be compared is performed in advance for multiple products, preferably all products that may be displayed on the shelves.

When the image information input reception processing unit 20 of the management terminal 2 receives image information of the store's display shelves from the input terminal 3 (S100), the image information storage unit 21 stores the image information in association with the date and time of shooting, identification information of the captured image information, etc. At this time, input of information such as which shelf level of which display shelf in which store the captured image information was captured is also received and stored in the image information storage unit 21. Note that the store, display shelf, and shelf level information input here is reflected and stored in the storage unit that stores the comparison results and the product information storage unit 23 when the comparison results are stored in the comparison processing unit 25 described later.

As a pre-processing step, the image information processing unit 22 performs a normalization process on the received captured image information to correct the captured image information so that it is in a normal orientation (S110). Then, it performs face processing to identify a face area from the normalized captured image information (S120).

Then, for each face area identified in the image information processing unit 22, the product depicted in that face area is identified.

First, the narrowing-down processing unit 24 executes a process of narrowing down the product sample information to be compared with the image information of the face area to be processed and/or its feature quantities (S130). Specifically, it determines whether or not the conditions of one or more predetermined narrowing-down factors, such as the aspect ratio of the face area, the brightness and saturation of the dominant color in the face area, and the hue of the dominant color in the face area, are satisfied.

Then, the narrowing down processing unit 24 identifies products having the same result (narrowing down factor information) as the judgment result of whether or not the condition of each narrowing down factor is satisfied for the face area to be processed from the product information storage unit 23 as products to be subjected to comparison processing in the comparison processing unit 25, and identifies sample information of the identified products from the product information storage unit 23. For example, if the judgment result of whether or not the condition of the narrowing down factor is satisfied for the face area to be processed is TTT, the product whose narrowing down factor information is TTT and its sample information are identified from the product information storage unit 23.

When the products and their sample information to be compared in the comparison processing unit 25 are identified in this way, the comparison processing unit 25 executes a comparison process (matching process) between the sample information of the products identified by the narrowing down processing unit 24 and the image information and/or its features of the face area to be processed (S140). Then, the similarity between the features of the image information of the face area and the sample information is calculated, and by sorting based on the similarity, the identification information and sample information of a predetermined number or a predetermined range of products from the top are output to the display device 72 of the management terminal 2 or the display device 72 of the computer operated by the operator as candidates for products that may be captured in the face area (S150).

As a result of the comparison process by the comparison processing unit 25, candidate products that may be shown in the face area are output to the display device 72 of the management terminal 2 or the display device 72 of the computer operated by the operator, and the operator etc. checks the candidate products and visually determines whether the product shown in the face area is included as a candidate product. If the product shown in the face area is included as a candidate product, the correct answer is input by performing a specified operation such as selecting the candidate product. Also, if the product shown in the face area is not included as a candidate product, a specified operation is performed to input to output the next candidate.

When the product selection reception processing unit 26 receives an input operation (operation indicating that there is no correct answer) that prompts the output of the next candidate product shown in the face area (S160), the narrowing down processing unit 24 performs processing to narrow down the products to be compared by the comparison processing unit 25 next (S170). That is, for the judgment result of the narrowing down factor first judged by the narrowing down processing unit 24, a judgment result obtained by changing the judgment result from a narrowing down factor with a lower priority, for example, a judgment result TTF changed from judgment result TTT, and the sample information thereof are identified from the product information storage unit 23.

Then, in S170, the comparison processing unit 25 executes a comparison process (matching process) between the standard information of the product narrowed down as a target for comparison processing and the image information and/or its features of the face area to be processed (S140), and repeats the processes from S140 onwards.

On the other hand, when the product selection reception processing unit 26 receives input of the correct product for the product shown in the face area (S160), it identifies the selected product as the face area to be processed and the product shown in that face area, and stores the identified product's product identification information, product name, product code, etc. in a specified storage unit in association with the product (S180). The display position of the product can be specified by the vertical and horizontal order of the face area, or it can be specified as coordinates in the captured image information.

In addition, the specific storage unit mentioned above stores which shelf and in which store the identified product was displayed on.

By performing the above-mentioned processing on all face areas of the image information, it is possible to identify the products shown in the face areas and to know the display status of the products on the display shelves. In addition, it is possible to reduce the comparison processing (matching processing) in the comparison processing unit 25, which can reduce the processing load and processing time.

Next, a process for further narrowing down the search results using the collection of misjudgment cases will be described. In this embodiment, as shown in FIG. 6, a case will be described in which a collection of misjudgment cases is stored in the product information storage unit 23 in association with the product identification information. However, the collection of misjudgment cases may be stored in a storage unit other than the product information storage unit 23. In that case, the storage unit may store narrowing down factor information and a collection of misjudgment cases in association with the product identification information.

The collection of misjudgment cases stores a correspondence between product identification information and the history of misjudgments. The history of misjudgments stores, in the case of a misjudgment regarding a narrowing-down factor, the identification information of the product and information indicating which narrowing-down factor caused the misjudgment, in correspondence with each other.

An example of the processing process in this embodiment will be described with reference to the flowchart in FIG. 7. Note that the processing from S200 to S280 in FIG. 7 is similar to the processing from S100 to S180 in FIG. 3, and therefore some of it will be omitted to avoid duplication of explanation. Also, FIG. 8 is a schematic conceptual diagram of the processing after the narrowing down process in the narrowing down processing unit 24 in this embodiment.

Let's say that for a certain product AAA, TTT (satisfies all conditions for narrowing down factors 1 to 3) has been registered as the narrowing down factor information.

In the narrowing process of S230, if the result of the judgment on whether the image information of the face area to be processed satisfies the narrowing factor condition is TTT, the product whose narrowing factor information, such as the above-mentioned product AAA, is specified as the target of comparison processing in the comparison processing unit 25. Then, as a result of the comparison processing in the comparison processing unit 25 using the sample information of the specified product and the image information of the face area to be processed and/or its feature amount (S240), if product AAA is included in a predetermined number or within a predetermined range from the top, it will be output to the display device 72 of the management terminal 2 or the display device 72 of the computer operated by the operator as a candidate product that may be captured in the face area (S250). The candidate products output at this time are temporarily stored for the processing of erroneous judgment described later.

When the product selection reception processing unit 26 receives an input operation to prompt the output of the next candidate product shown in the face area (S260), it can be understood that each product output as a candidate is not a correct judgment result (a false judgment) as a condition of the narrowing down factor. Then, as in the first embodiment, the narrowing down processing unit 24 performs a process to narrow down the products to be subjected to the comparison processing of the comparison processing unit 25 next (S270). That is, for the judgment result of the narrowing down factor first judged by the narrowing down processing unit 24, a judgment result in which the judgment result is changed from the narrowing down factor with a lower priority, for example, a judgment result TTF in which the judgment result TTT is changed, and the sample information of the products are identified from the product information storage unit 23. Then, the comparison processing in the comparison processing unit 25 is executed based on the sample information.

The above process is repeated, and for the judgment result of the changed narrowing down factor, for example, the changed judgment result TTF, it is assumed that the input of the correct product is accepted for the candidate products output as a result of the comparison process of the comparison process unit 25 based on the sample information of the products identified by the narrowing down process unit 24 (product BBB, product EEE, etc. in the case of Figure 6) (S260). Then, the product selection acceptance process unit 26 identifies the product whose selection has been accepted as the face area to be processed and the product appearing in that face area, and stores the identified product's product identification information, product name, product code, etc. in a specified storage unit in association with the product (S280).

Then, the product selection reception processing unit 26 stores the history of misjudgments in the misjudgment case collection in the product information storage unit 23 (S290). That is, if the product shown in the face area to be processed is identified as a product with the narrowing factor information of TTF, it can be considered that the narrowing factor information of each product that has already been narrowed down by the narrowing processing unit 24 and output as a candidate that has been temporarily stored is likely to be misjudged. For example, the narrowing factor information of product AAA is TTT, and although the narrowing processing unit 24 narrowed down the product, the product was not identified as the correct answer. Therefore, it can be considered that the third narrowing factor information T, which is the difference from the narrowing factor information of the product that was the correct answer, is likely to be misjudged. Therefore, information indicating that there was a misjudgment of F in narrowing factor 3 of product AAA is stored.

Then, in the second narrowing-down process in the narrowing-down processing unit 24 in S270, when identifying the next product to be subjected to the comparison process in the comparison processing unit 25, the next candidate is identified as a product that has the same narrowing-down factor information as the judgment result obtained by changing the judgment result of a narrowing-down factor with a lower priority, and for which an erroneous judgment has been registered for the changed narrowing-down factor.

In the above example, if the selection factor information in the selection processing unit 24 in the first S230 is TTT, the next product that has TTF as the selection factor information and has a registered erroneous judgment for the changed selection factor is identified as the next candidate. Therefore, the selection processing unit 24 identifies not only products that have TTF as the selection factor information, but also products that have a history of erroneous judgment stored in selection factor 3.

This allows you to further narrow down the products that are prone to misidentification, resulting in more accurate narrowing down.

Furthermore, as a modified example of the second embodiment, instead of simply storing whether or not an erroneous judgment occurred, the number of times an erroneous judgment occurred, or the probability based on the number of times an erroneous judgment occurred among the number of times identified as the narrowing-down factor information, may be stored.

In this case, when the selection processing unit 24 identifies the next product to be subjected to comparison processing in the comparison processing unit 25, the selection processing unit 24 identifies as the next comparison target a product that has the same selection factor information as the judgment result obtained by changing the judgment result of a selection factor with a lower priority, and for which the number or probability of erroneous judgment for the changed selection factor is equal to or greater than a predetermined value.

By performing the above process, it is possible to reduce the number of feature comparison processes that require computational time, thereby reducing the overall computation time.

The processes in each of the above embodiments are not limited to the order described in the present specification, but may be modified as appropriate to the extent that the objectives of the present invention are achieved.

The narrowing down processing system 1 of the present invention is also effective when identifying the display status of a product displayed on a store's display shelves from photographed image information of the display shelves, but is not limited to this. In other words, when a certain product is photographed, the system can be widely used when identifying the area in which the desired product is captured from the photographed image information.

In each of the above-mentioned embodiments, we have described a case where the products shown (displayed on the display shelves) are identified using basic product sample information based on image information of the display shelves, but the narrowing down processing system 1 of the present invention can also be used for different purposes.

For example, an image is taken when a customer visits the store using a camera (security camera) installed in the store, and the area in which the customer's face is captured is identified from the image information. The person captured in that area can then be identified by comparison processing in the comparison processing unit 25 using pre-registered facial image information and/or feature information and attribute information of that person, for example, facial image information and/or feature information registered the previous time the customer visited the store and that person's attribute information, and this information can be used to narrow down the pre-registered facial image information and/or feature information to be used for comparison processing in the comparison processing unit 25. Examples of factors for narrowing down the information include accessories such as glasses and hats, possessions such as satchels and bags, and facial features.

By using the narrowing down processing system 1 of the present invention, when identifying products shown in image information obtained by photographing a display shelf or the like, it is possible to narrow down the comparison targets and perform image recognition processing (matching processing), thereby reducing the processing load and processing time of the computer. In addition, when presenting candidates, it is possible to present the correct product at the top.

1: Narrowing processing system 2: Management terminal 3: Input terminal 20: Image information input reception processing unit 21: Image information storage unit 22: Image information processing unit 23: Product information storage unit 24: Narrowing processing unit 25: Comparison processing unit 26: Product selection reception processing unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device

Claims (2)

A screening processing system for identifying products shown in image information, comprising:
The narrowing down processing system includes:
an image information processing unit for identifying a face area in the image information to be compared with sample information of the product;
a narrowing-down processing unit that narrows down products to be compared by using a result of determining a narrowing-down factor that is predetermined for the face area and narrowing-down factor information of the products that is stored in advance;
a comparison processing unit that compares sample information of the narrowed-down products with image information of the face area and/or its feature amount,
The narrowing down processing unit includes:
determining whether or not the image information of the face area and/or its feature quantity satisfies a condition defined for one or more predetermined narrowing-down factors;
Narrowing down products having the same narrowing-down factor information as the result of the judgment as products to be compared;
A screening processing system comprising: Computer,
an image information processing unit for identifying a face area in the image information to be compared with sample information of the product;
a narrowing-down processing unit that narrows down products to be compared by using a result of determining a narrowing-down factor that is predetermined for the face area and narrowing-down factor information of the products that is stored in advance;
A narrowing-down processing program that functions as a comparison processing unit that compares sample information of the narrowed-down products with image information of the face area and/or its features,
The narrowing down processing unit includes:
determining whether or not the image information of the face area and/or its feature quantity satisfies a condition defined for one or more predetermined narrowing-down factors;
Narrowing down products having the same narrowing-down factor information as the result of the judgment as products to be compared;
A narrowing down processing program comprising:

Images