JP6941331B2 - Image recognition system - Google Patents

Description

The present invention relates to an image recognition system for identifying products displayed in stores and the like.

In the retail industry, it is known that the display status of products affects the sales of products. Therefore, as a product manufacturing company or sales company, it is possible to connect to the development strategy and sales strategy of the company's products by grasping what kind of products of the company or other companies are displayed in the store.

On the other hand, in order to realize this, it is important to accurately identify the products displayed in the store. Therefore, it is conceivable that a person manually identifies the products displayed from the image information of the display shelves of the store. In this case, the product can be identified almost accurately. However, in order to continuously grasp the display status of products, it is necessary to grasp the display status at regular intervals, but humans identify the products every time from the image information taken of the display shelves of the store. Is burdensome and inefficient.

Therefore, it is required to automatically identify the products displayed on the display shelves of the store from the image information taken and grasp the display status of the products. For example, there is a method of using image recognition technology for an image of a store display shelf based on a sample image of each product. As these conventional techniques, for example, there is a system for managing the display status of products by using the techniques shown in Patent Documents 1 to 3 below.

Japanese Unexamined Patent Publication No. 5-342230 Japanese Unexamined Patent Publication No. 5-334409 International release 2012/029548

The invention of Patent Document 1 is a system that supports a person who does not have knowledge about which display shelf a product should be displayed on. Therefore, it is not possible to grasp the products that are actually displayed. Further, Patent Document 2 is a system that supports input of a product image in a shelf allocation support system that supports the display of products. However, the system of Patent Document 2 only supports the input of product images when using the shelf allocation support system, and even if this system is used, it is not possible to grasp the products actually displayed. ..

Patent Document 3 is an invention that specifies a product to be displayed in the empty space when there is an empty space on the display shelf, or notifies the product when the product to be displayed on the display shelf is mistakenly placed. In the present invention as well, image matching processing is performed between the product and the product displayed on the display shelf to identify the product displayed on the display shelf, but the actual situation is that the recognition accuracy is low.

When image recognition technology is used based on a sample image of a product for an image taken on a display shelf, the accuracy of the image recognition and the processing load become problems. For example, in a store where display shelves are installed, if the lighting conditions are not uniform and the difference from the lighting of the sample image is large, the accuracy of image recognition deteriorates. Therefore, it may not be easy to identify the products displayed on the display shelves from the images taken. In addition, a large processing load is generated for image recognition, which requires a large amount of resources. Therefore, the invested capital becomes large.

The present inventor pays attention to a product tag for displaying the price of the product, etc. as one method for identifying the product shown in the image of the display shelf, and accurately displays the displayed product. We have invented an image recognition system that can be identified. In addition, an image recognition system that can accurately identify the products on display by recognizing the products on the display shelves, recognizing the information on the product tags, and collating them. Invented.

The first invention is a first emplacement processing unit that generates a second image information by performing a first emplacement process on the first image information showing a display shelf for displaying products. And the second emplacement processing unit that performs the second emplacement processing on the area including the product tag arrangement area in the second image information, and the image information that has undergone the second emplacement processing. An image having a product tag specifying processing unit that specifies the product tag area from the above, and an information specifying processing unit in the product tag that specifies the information written on the product tag by performing OCR recognition processing in the specified product tag area. It is a recognition system.

By using the image recognition system of the present invention, the information written on the product tag can be accurately recognized, and therefore the products displayed corresponding to the information can be accurately identified.

In the above-described invention, the information specifying processing unit in the product tag identifies the box by binarizing the specified product tag area and performing labeling processing, and among the specified boxes, the height, width, and the like of the box. This is an image recognition system that specifies boxes whose baseline satisfies a predetermined condition as blocks and executes the OCR recognition process on the specified blocks.

The first invention can be realized by loading and executing the computer program of the present invention. That is, the first emplacement processing unit, which generates the second image information by performing the first emplacement processing on the first image information in which the display shelf for displaying the products is shown. A second emplacement processing unit that performs a second emplacement process on an area including a product tag arrangement area in the second image information, and a product tag from the image information that has undergone the second emplacement process. This is an image recognition program that functions as a product tag identification processing unit that specifies an area and an information identification processing unit within a product tag that specifies the information written on the product tag by performing OCR recognition processing in the specified product tag area. ..

By using the image recognition system of the present invention, it is possible to accurately identify the products displayed on the display shelves.

It is a block diagram which shows typically an example of the system structure of the image recognition system of this invention. It is a block diagram which shows typically an example of the hardware composition of the computer used in the image recognition system of this invention. It is a flowchart which shows an example of the whole processing process in the image recognition system of this invention. It is a flowchart which shows an example of the processing process of the product tag arrangement area normalization processing. It is a flowchart which shows an example of the processing process of the specific processing of the product tag image information. It is a flowchart which shows an example of the processing process of the specific processing of the information in a product tag. It is a figure which shows an example of the emplacement image information which performed the emplacement processing on the photographed image information. It is a figure which shows another example of the emplacement image information which performed the emplacement processing on the photographed image information. FIG. 5 is a diagram showing an example of a state in which a product tag placement area is specified for the normal image information with respect to the normal image information of FIG. 7. FIG. 5 is a diagram showing an example of a state in which a product tag placement area is specified for the normal image information with respect to the normal image information of FIG. It is a figure which shows an example of the image information which performed the normalization processing with respect to the product tag arrangement area in FIG. It is a figure which shows the state which detected the straight line in the vertical direction which is the reference of the keystone correction processing in the product tag arrangement area normalization processing part. It is a figure which shows typically the process of specifying the upper side position and the lower side position of the product tag area in the product tag identification processing unit. It is a figure which shows typically the process which specifies the position in the lateral direction of the product tag area in the product tag identification processing unit. It is a figure which shows typically the state which specified the product tag area from the image information of the product tag arrangement area which was put upright in FIG. It is a figure which shows typically the image information which binarized the product tag area in the processing in the product tag information identification processing unit. It is a figure which shows typically the state which generated the box in the processing in the information identification processing part in a product tag. It is a figure which shows typically the state which generated the block in the processing in the information identification processing part in a product tag. It is a figure which shows an example of the product name written in the product tag, the product name which has the smallest edit distance among the product names stored in the product dictionary of the result of OCR recognition, and the edit distance. It is a figure which shows an example of the correspondence relation of the edit distance which may be decided for each number of characters of a final candidate. It is a figure which shows an example of the photographed image information schematically. It is a figure which shows an example of the photographed image information schematically. It is a block diagram which shows typically an example of the system structure of the image recognition system in Example 2. FIG. It is a block diagram which shows typically an example of the structure of the product tag recognition processing part. It is a block diagram which shows typically an example of the structure of the display product recognition processing unit. It is a flowchart which shows an example of the whole processing process in the image recognition system in Example 2. It is a figure which shows typically the Nth processing in the product identification information identification processing unit. An example of the sample image information stored in the sample image information storage unit is shown. It is a figure which shows the state which specified the shelf stage area and the product tag arrangement area. It is a figure which shows the state which specified the shelf stage area and the product tag arrangement area. An example of a state in which the feature amount collection area is set for the N-1th captured image information is shown. It is a figure which shows an example of the state which set the feature amount collection area for the Nth photograph image information. It is a figure which shows the relationship of the pair of the N-1th feature collection area and the Nth feature collection area. It is a figure which shows the state which the shelf position C in the N-1th photographed image information is projected as the shelf position D in the Nth photographed image information by a function F. It is a figure which shows an example of photographed image information. It is a figure which shows the state which set the feature amount collection area in the N-1 first photographed image information. It is a figure which shows the state which set the feature amount collection area in the Nth photograph image information. It is a figure which shows an example of the state which specified the shelf positions D1 to D4 in the Nth photographed image information.

An example of the system configuration of the image recognition system 1 of the present invention is shown in FIG. The image recognition system 1 uses a management terminal 2 and a captured image information input terminal 4.

The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. Further, the photographed image information input terminal 4 is a terminal for inputting image information obtained by photographing the display shelf of the store.

The management terminal 2 and the captured image information input terminal 4 in the image recognition system 1 are realized by using a computer. FIG. 2 schematically shows an example of the hardware configuration of the computer. The computer can input information from an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, and a display device 72 such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic unit 70 and information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The captured image information input terminal 4 may include a photographing device such as a camera in addition to the above-mentioned devices. As the captured image information input terminal 4, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing order of the processing in each means of the present invention may be changed as appropriate. In addition, a part of the processing may be omitted. For example, the emplacement process can be omitted. In that case, it is possible to execute processing on the image information that has not been subjected to the orthostatic processing.

The management terminal 2 in the image recognition system 1 can send and receive information via a network with the captured image information input terminal 4.

The image recognition system 1 includes a photographed image information input reception processing unit 20, a photographed image information storage unit 21, a photographed image information normalization processing unit 22, a position identification processing unit 23, and a product tag placement area cutting processing unit. 24, a product tag arrangement area normalization processing unit 25, a product tag specifying processing unit 26, and an information specifying processing unit 27 in the product tag.

The photographed image information input reception processing unit 20 receives the input of the image information (photographed image information) of the display shelf of the store photographed by the photographed image information input terminal 4, and stores it in the photographed image information storage unit 21 described later. In addition to the photographed image information, the photographed image information input terminal 4 may accept input including the photographed date and time, store identification information such as the store name, and image information identification information for identifying the image information.

The photographed image information storage unit 21 stores the photographed image information, the shooting date and time, the store identification information, the image information identification information, etc. received by the photographed image information input reception processing unit 20 in association with each other. The captured image information may be any image information to be executed for trapezoidal correction processing, and includes image information synthesized as one image information when one display shelf is photographed by a plurality of images. .. In addition, the captured image information also includes the image information after the distortion correction processing is executed.

The captured image information emplacement processing unit 22 generates the emplacement image information in which the photographed image information stored in the photographed image information storage unit 21 is subjected to trapezoidal correction processing and is emplaced. The keystone correction process is a correction process performed so that the shelves of the display shelves shown in the photographed image information are horizontal and the product tags for the products displayed there are vertical.

The keystone correction process executed by the captured image information emplacement processing unit 22 receives input for designating four vertices in the photographed image information, and executes the keystone correction process using each of the vertices. The four vertices that accept the designation may be the four vertices of the shelf stage of the display shelf or the four vertices of the shelf position of the display shelf. Further, it may be four vertices of a group of two or three shelves. Any four points can be specified as the four vertices.

7 and 8 show an example of the photographed image information (normal image information) that has been subjected to the normalization process. FIG. 7 is vertical image information in which beverage can products such as beer are displayed on two upper and lower shelves as display shelves. FIG. 8 is a hanging shelf for displaying products such as toothbrushes on a display shelf, and is vertical image information in which the products are displayed in two upper and lower stages.

The position identification processing unit 23 is an area (product tag arrangement) in which the product tag may be attached to the normal image information in which the trapezoidal correction processing is executed on the photographed image information in the photographed image information normalization processing unit 22. Area) is specified. That is, although the display shelf is shown in the photographed image information and the vertical image information, there is a possibility that the area of the shelf where the products are displayed and the product tag for the products displayed there are attached to the display shelf. There is a product tag placement area with. Therefore, the product tag placement area is specified from the normal image information. To specify the product tag placement area, the operator of the management terminal 2 may manually specify the product tag placement area, and the position identification processing unit 23 may accept the product tag placement area, or the product tag for which input is manually received for the first time. The product tag placement area may be automatically specified from the second time onward based on the placement area information.

FIG. 9 schematically shows a state in which the input of the designation of the product tag placement area is accepted for the normal image information of FIG. 7. Further, FIG. 10 schematically shows a state in which the input of the designation of the product tag arrangement area is accepted for the normal image information of FIG.

The product tag arrangement area cutout processing unit 24 cuts out the image information of the product tag arrangement area specified by the position identification processing unit 23 as the product tag arrangement area image information. The product tag arrangement area cutout processing unit 24 may actually cut out as image information, or may cut out virtually without actually cutting out as image information. Virtually cutting out image information means executing processing with the specified area, for example, the range of the product tag placement area as the processing target.

The product tag placement area normalization processing unit 25 executes trapezoidal correction processing for normalizing the product tag placement area image information cut out by the product tag placement area cutting processing unit 24. While the surface of the display shelf is vertical, the surface of the product tag is often facing upward from the vertical surface so that the customer can easily see it. Therefore, the recognition accuracy is improved by making the image information of the product tag placement area normal. FIG. 11 shows an example of the image information of the product tag arrangement area that has been rectified with respect to the image information of the product tag arrangement area of FIG. FIG. 11 (a) shows the image information of the product tag placement area in which the image information of the product tag placement area of the upper shelf in FIG. 9 is rectified, and FIG. 11 (b) shows the image information of the lower shelf in FIG. This is the image information of the product tag placement area in which the image information of the product tag placement area is rectified.

The product tag placement area normalization processing unit 25 normalizes the image information of the product tag placement area by executing the following processing. That is, in the image information of the product tag placement area, edge detection is performed, and a nearly vertical contour line of a certain length or more (for example, 70 degrees to 110 degrees) is vertical (90 degrees) at a location near the left and right ends. ) To specify the contour line) within a predetermined range of angles. It is preferable, but not limited to, to extract the contour lines near the left and right ends. In the case of the image information of the product tag arrangement area of FIG. 9, L1 to L4 are specified, respectively, as shown in FIG. FIG. 12 (a) is the product tag arrangement area of the upper shelf in FIG. 9, and FIG. 12 (b) is the product tag arrangement area of the lower shelf in FIG. The contour lines L1 to L4 to be specified are not actually drawn in the image information of the product tag arrangement area. Then, the trapezoidal correction processing for the image information of the product tag arrangement area is executed so that L1 and L2 in FIG. 12A and L3 and L4 in FIG. 12B are vertical lines, respectively. By executing such a process, the image information of the product tag arrangement area is made normal, and the image information of the product tag arrangement area shown in FIG. 11 can be obtained. By executing the processing of the product tag placement area normalization processing unit 25, the accuracy of the product tag specifying processing unit 26 and the information specifying processing unit 27 in the product tag can be improved. Therefore, the processing is executed. It is preferable, but it can be omitted. In that case, the product tag identification processing unit 26 and the product tag in-information identification processing unit 27 execute the product tag arrangement area cut out by the product tag arrangement area cutout processing unit 24.

The product tag identification processing unit 26 specifies an area (product tag area) of each product tag from the product tag arrangement area image information that has been placed upright. Two main methods can be used for the process of specifying the product tag area. The first method is a method of specifying the product tag area based on the contour line, and the second method is to match the overall features such as the distribution of the entire light and darkness with the image information of the product tag template. This is a method of specifying the product tag area. A method other than the first method and the second method can also be used.

The first method is a method that utilizes the fact that the background color (background color) of the product tag is often white (brighter than the background). That is, first, among the image information of the product tag placement area that has been placed upright, a histogram is generated in which the brightness information of the image information is integrated in the horizontal direction. Then, the rising and falling positions of the histogram are specified, and the upper side position A and the lower side position B of the product tag are specified. FIG. 13 schematically shows this process. The rising edge is the point where the histogram increases sharply from black to white (more than a predetermined ratio), and the falling edge is the point where the histogram sharply decreases from white to black. ..

Then, out of the image information of the product tag placement area that has been placed upright, a histogram is generated in which the brightness information of the image information of the product tag placement area that has been placed upright is integrated in the vertical direction by cutting out between the upper side position A and the lower side position B. do. Then, the rising and falling positions of the histogram are specified, and the rising edge and the falling edge within a predetermined distance range to the right of the rising edge are paired, and the product tag area is specified by setting each as the left side position and the right side position. Then, if there is no product tag area within a predetermined distance, it is specified as a product tag area on the right side of the rising edge where the pair could not be formed and on the left side of the falling edge where the pair could not be formed. FIG. 14 schematically shows this process.

Further, FIG. 15 shows a state in which the product tag area is specified from the image information of the product tag placement area that is placed upright in FIG. 11 by the first method. A rectangular area composed of each of the upper side position A, the lower side position B, the left side position (rising) U, and the right side position (falling) D is the specified product tag area.

The second method is so-called template matching. That is, the product tag area is specified by registering the image information of the product tag as a template in advance and matching the template with the image information of the product tag placement area that has been placed upright.

The product tag includes the tax-excluded price, the tax-included price, the product identification information (product name, etc.), the manufacturer name, the rating, and the like. Therefore, if the image information of the product tag used as a template includes product identification information such as the product name, specific price numbers, and characters, that part is also included in the judgment target of the image matching process, so that part is mosaicked. , It is preferable to neutralize or exclude it from the judgment target by deleting it. Neutralization means not allocating high and low points for any input, and exclusion means excluding that part from the matching target during image matching processing.

The product tag information identification processing unit 27 executes a process of specifying the information described in the product tag area specified by the product tag identification processing unit 26 by OCR recognition or the like. When OCR recognition is performed, it can be performed for all or a part of the area specified as the product tag area. If it is applied to all the areas specified as the product tag area, there is a high possibility that erroneous recognition will occur due to noise or the like, so it is preferable to limit the target area for OCR recognition. In this case, the processes corresponding to the first method and the second method used in the product tag specifying processing unit 26 are executed.

When the first method is used in the product tag specifying processing unit 26, binarization processing is first performed in the specified product tag area. Then, the labeling process is executed on the binarized image information. The labeling process is a process of assigning the same number (identification information) to pixels in which white or black is continuous in binary image information, thereby forming consecutive pixels into one island (group). be. Then, a rectangular area (box) including the islands detected by the labeling process is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable, but not limited to, to generate the smallest rectangle surrounded by vertical and horizontal line segments that surrounds the area labeled with the same number. When generating a box, islands that do not satisfy the height and width that are the predetermined thresholds are not generated as noise and are removed from the processing target as they are. For example, an island that is too small may have horizontal ruled lines or dust on the image, and an island that is too wide may have a logo, etc., which are removed as noise.

The characters used in product tags are generally in bold, such as Gothic. Therefore, even if the image information is slightly out of focus, the character group forming one character string can be detected as an island having the same height as the baseline.

Then, the information identification processing unit 27 in the product tag merges adjacent boxes having predetermined similarities to form a block. That is, a block is formed by merging consecutive boxes whose baseline and height match within a predetermined range and whose height and width are within a certain threshold range. At this time, small boxes and the like between the blocks to be merged are also collectively configured as one block. As a result, voiced sound marks, semi-voiced sound marks, hyphens, etc. are also incorporated into one block. The block is an area that is the target of OCR recognition. Then, among the blocks, the block with the highest height is estimated as the price range (price range excluding tax), and OCR recognition is performed. Similarly, OCR recognition is performed for other blocks. By executing the above processing, character recognition processing can be executed more accurately than performing OCR recognition corresponding to a manuscript freely laid out over a plurality of lines in the product tag area. 16 to 18 schematically show this process. FIG. 16 is binarized image information, and FIG. 17 is a diagram showing a state in which a box (area shown by a broken line) is generated. Further, FIG. 18 is a diagram showing a state in which a block (region shown by a broken line) is generated.

As described above, the information identification processing unit 27 in the product tag can recognize the information described in the product tag as characters.

When the second method is used in the product tag identification processing unit 26, the product identification information such as the tax-excluded price, the tax-included price, the manufacturer name, and the product name, and the rating are written in advance in the image information of the product tag used as the template. The position and size (height, width) of each character frame are set. Therefore, the image information of the corresponding portion is cut out from the product tag area specified by the product tag specifying processing unit 26, and the OCR recognition process is executed. At this time, the OCR recognition process can be performed by defining the product identification information such as price, manufacturer name, and product name, and the character type used by the rating (for example, numbers, Roman characters, symbols, character strings, etc.) as constraints. The accuracy can be improved.

Further, the information identification processing unit 27 in the product tag executes a process of confirming the consistency of the read information. As the consistency confirmation process, it is preferable to perform two types of consistency confirmation process by dictionary collation and logical check.

The consistency confirmation process by dictionary collation is executed as follows, for example. The image recognition system 1 stores product identification information such as product names of products that may be displayed on display shelves in association with corresponding code information (for example, JAN code) (shown in the figure). It is equipped with. Then, the editing distance (Levenshtein) between the character string other than the character string read from the area indicating the price recognized by the information identification processing unit 27 in the product tag and the product identification information such as all the product names registered in the product dictionary. Distance) is calculated. If the minimum editing distance is one of the obtained editing distances, the product identification information such as the product name of the editing distance is used as the final candidate. Then, the allowable editing distance is determined in advance for the length of the character string of the product identification information such as the product name that is the final candidate, and the product identification information such as the product name is identified if it is within the allowable editing distance. do. If the edit distance exceeds the allowable editing distance, the read character string is undetermined. If there are multiple minimum editing distances, the read character string is undetermined.

The edit distance is a type of distance that indicates how different the two character strings are. Specifically, one character string is transformed into the other character string by inserting, deleting, or replacing one character. The minimum number of steps required to do this. FIG. 19 shows an example of the product name and the editing distance with the shortest editing distance among the product names stored in the product dictionary as a result of the product name and OCR recognition written on the product tag. Further, FIG. 20 shows a table of correspondences of edit distances that may be determined for each number of characters of the final candidate. In this specification, the case where the processing is performed using the editing distance will be described, but the distance function may be a distance function that has been processed to shorten the replacement distance in the calculation of the editing distance. This distance function is a function that calculates the distance indicating how different the two character strings are, and includes the above-mentioned editing distance.

Then, after obtaining the product identification information such as the product name with the minimum editing distance described above, a rating dictionary (shown) is separately prepared for the character string excluding the relevant part of the product identification information such as the product name from the OCR recognition result. The editing distance with each rating stored in (not shown) and the editing distance with each manufacturer name stored in the manufacturer name dictionary (not shown) are obtained. For example, in the example of "350 ml of throat raw" in FIG. 19, the part of "350 ml" matches "350 ml" in the rated dictionary with an editing distance of 0, and it is identified that it is a character string of the rated part. Similarly, the editing distance is obtained for the maker name, and the character string with the minimum editing distance is identified as the maker name. Then, each character string of the rated part and the manufacturer name part is removed from the OCR recognition result, and the character string having the shortest editing distance in the product name dictionary is obtained from the removed remaining character string, and is it an acceptable editing distance? Judge whether or not. If it is within the allowable editing distance, determine the product identification information such as the product name, the manufacturer name, and the rating. By performing such processing, even if the product identification information such as the product name includes the rating and the manufacturer name, it can be appropriately determined. The rating dictionary is a dictionary that stores the ratings (capacity, etc.) of products that may be displayed on the display shelves. The manufacturer name dictionary is a dictionary that stores the manufacturer names of products that may be displayed on the display shelf.

When displaying the recognition result to the user and displaying it for selection, it is preferable that the confirmed character string and the unconfirmed character string are displayed in a method that can be identified by each. For example, the color of a confirmed character string and an unconfirmed character string may be separated, and information indicating confirmed or unconfirmed may be added to a confirmed character string and / or an unconfirmed character string. For undetermined character strings, if there are multiple candidates for product identification information such as the product name with the minimum editing distance, each candidate may be displayed.

The logical check (the processing for determining the logical consistency of the recognized information) in the information specifying processing unit 27 in the product tag can be performed as follows. For example, if you read two prices as the price range, one will be the tax-excluded price and one will be the tax-included price, but if the tax-excluded price is higher than the tax-included price, they will be reversed and recognized. Let me. If the value obtained by multiplying the tax-excluded price by the consumption tax rate does not match the tax-included price, it is assumed that either or both are misrecognized. Furthermore, when the product identification information such as the product name is recognized, it is determined whether the product or the product category is within the normal price range. In addition, it may be determined whether the correspondence between the product identification information such as the product name, the manufacturer name, and the rating matches.

By confirming the consistency of the information in the product tag as described above, the information included in the product tag reflected in the photographed image information can be determined. The information determined in this way can be output in tabular format, for example.

Next, an example of the processing process of the image recognition system 1 of the present invention will be described with reference to the flowcharts of FIGS. 3 to 6.

The photographed image information photographed on the display shelf of the store is input from the photographed image information input terminal 4, and the input is accepted by the photographed image information input reception processing unit 20 of the management terminal 2 (S100). 21 and 22 show an example of captured image information. It also accepts input of image information identification information such as shooting date and time, store identification information, and shooting image information. Then, the captured image information input reception processing unit 20 stores the captured image information, the shooting date and time, the store identification information, and the image information identification information of the captured image information in association with each other in the captured image information storage unit 21.

When the management terminal 2 receives a predetermined operation input, the normal image information normalization processing unit extracts the photographed image information stored in the photographed image information storage unit 21, and is the apex for performing the keystone correction processing. The input of four points of the position (position of the display shelf) is received, and the keystone correction process is executed (S110). 7 and 8 show an example of the captured image information (normal image information) in which the keystone correction process is executed in this way.

Then, the position specifying processing unit 23 specifies the product tag placement area by accepting a predetermined operation input in the management terminal 2 for the normal image information (S120). That is, the input of the product tag placement area in the normal image information is accepted. 9 and 10 are diagrams showing a state in which the product tag arrangement area is specified. Then, the product tag placement area cutting processing unit 24 cuts out the image information of the product tag placement area specified in S120 (S130), and the product tag placement area normalization processing unit 25 executes the trapezoidal correction processing to perform the product. The normalization process for the image information in the tag placement area is executed (S140).

As the normalization process for the image information in the product tag placement area, first, edge detection is performed in the image information in the product tag placement area. Then, among the detected edges, a contour line having a certain length or more and within an angle within a predetermined range from the vertical is specified (S200). Then, among the specified contour lines, the leftmost and rightmost contour lines are specified (S210). An example of the contour line thus specified is shown in FIG. Then, the trapezoidal correction process is executed for the image information of the product tag arrangement area so that the specified contour lines (L1, L2 in FIG. 12A and L3 and L4 in FIG. 12B) become vertical lines, respectively. (S220). By executing such a process, the image information of the product tag arrangement area is made normal, and the image information of the product tag arrangement area shown in FIG. 11 can be obtained.

When the process of normalizing the image information of the product tag placement area is completed in S140, the product tag identification processing unit 26 uses the first method or the second method to obtain individual product tags from the image information of the product tag placement area. The region is specified (S150).

In the case of the first method, among the image information of the product tag placement area that has been placed upright, a histogram that integrates the brightness information of the image information in the horizontal direction is generated (S300), and the rising and falling positions of the histogram are set. Identify. Then, the rising position of the specified histogram is specified as the upper side position A of the product tag, and the falling position of the histogram is specified as the lower side position B of the product tag (S310).

Next, of the image information of the product tag placement area that has been placed upright, a histogram that cuts out between the upper side position A and the lower side position B and integrates the brightness information of the image information of the product tag placement area that has been placed upright in the vertical direction. Is generated (S320).

In the generated histogram, the rising position U and the falling position D are specified, the rising position U (left side position) and the falling position D (right side position) within a predetermined distance range to the right of the rising position U are specified as a pair, and the product tag is specified. It is specified as an area (S330).

If there is no product tag area within a predetermined distance, specify it as a product tag area on the right side of the rising position U where the pair could not be formed, and on the left side of the falling position D where the pair could not be formed. (S340).

FIG. 15 shows a state in which each product tag area is specified by the above processing.

When the second method is used, the product tag identification processing unit 26 performs image matching processing between the image information of the product tag of the template registered in advance and the image information of the product tag arrangement area that has been placed upright. By executing, the product tag area is specified.

When the product tag specifying processing unit 26 specifies the product tag area as described above, the information specifying processing unit 27 in the product tag specifies the information in the product tag (S160).

The first method in the product tag information specifying processing unit 27 obtains binarized image information by binarizing the specified product tag area (S400). Then, the box is specified by the labeling process on the binarized image information (S410). Islandes that do not satisfy the height and width that are the predetermined thresholds are removed from the processing target as they are without generating a box as noise.

Then, by finding the height, width, and baseline of the generated box, and identifying adjacent boxes whose baseline and height match within a predetermined range and whose height and width are within a certain threshold range, The specified boxes are merged to form a block (S420). Then, among the blocks, the block with the highest height is estimated as the price range (price range excluding tax), and OCR recognition is performed (S430). If the price information cannot be obtained as a result of OCR recognition (if the numerical character string cannot be recognized) (S440), the block with the next highest height is estimated to be the price range (tax-excluded price range), and the OCR is also obtained. Recognize.

In the OCR recognition of S430, the reading accuracy of the price can be improved by adding the character recognition constituting the price display such as "number" and "comma" used for the price as a constraint condition.

When the price information is acquired by OCR recognition from the block estimated as the price area (tax-excluded price area) in this way (S440), the blocks other than the price information acquired by estimating the price area (tax-excluded price area) in S430 are obtained. It is specified (S450), and OCR recognition is performed for each specified block (S460). For this OCR recognition, it is preferable to perform two types of OCR recognition processing. That is, it is an ordinary standard OCR recognition process and an OCR recognition process in which character recognition constituting a price display is added as a constraint condition.

The block specified in S450 includes a block in the price area (price area including tax) and a block in the area of information such as product identification information such as manufacturer name and product name, and rating. Then, two types of OCR recognition processing are executed for each block. For blocks in the area of product identification information such as manufacturer name and product name, and information such as rating, character strings are recognized by standard OCR recognition processing, and many characters containing errors in OCR recognition processing with constraints. Become a column. In this case, it is determined whether the recognition results of the two recognition processes differ by a predetermined value or more, and if they differ, the character string executed by the standard OCR recognition process is used as the recognition result, and the price area (tax-included price area). It is determined that the block is other than the block of. On the other hand, for blocks in the price area (price area including tax), the standard OCR recognition process recognizes the price information character string, and the constrained OCR recognition process also recognizes the price information character string. In this case, it is determined whether the recognition results of the two recognition processes differ by a predetermined value or more, and if they do not differ, it is determined that the block is in the price range (price range including tax), and the recognition process with constraints added. The character string is used as the price information of the recognition result.

As described above, the information identification processing unit 27 in the product tag can recognize the information described in the product tag as characters.

When the second method is used in the product tag specifying processing unit 26, the product tag used as a template has the product identification information such as the tax-excluded price, the tax-included price, the manufacturer name, and the product name, and the rating, respectively. The position and size of the displayed character frame are set. Therefore, the image information of the corresponding portion may be cut out from the product tag area specified by the product tag specifying processing unit 26, and the OCR recognition process may be executed.

Then, the product tag information specifying processing unit 27 executes a dictionary collation process with the specified product name or the like (S470). That is, the edit distance between the read character string and the product identification information such as each product name in the product dictionary is obtained, the product identification information such as the product name with the minimum edit distance is specified, and the edit distance is within a predetermined distance. If so, it is identified as product identification information such as a product name. Then, the product name part with the shortest edit distance is removed from the read character string, the edit distance with each rating in the rating dictionary is obtained for the remaining substring, and whether the minimum edit distance is within the predetermined distance is determined. Judgment is made, and if it is within a predetermined distance, that part is identified as a rated character string. Similarly, the product name part and the rated part with the minimum editing distance are removed from the read character string, the editing distance with each manufacturer name in the manufacturer name dictionary is obtained for the remaining partial character string, and the minimum editing distance is determined. It is determined whether it is within the distance, and if it is within the predetermined distance, that part is identified as a character string of the manufacturer name.

Further, the product tag information identification processing unit 27 executes a logical check process for the character-recognized character string (S480). That is, it is judged whether or not the character strings recognized as characters are logically inconsistent.

If there is no inconsistency as a result of the logical check, identify the product identification information such as tax-excluded price, tax-included price, product name, manufacturer, and rating for each recognized character string, and identify them as the shooting date and time and store identification. Information and captured image information is stored and output in a predetermined storage area in association with the image information identification information. For example, output in tabular format.

In the first embodiment, the case where the information described in the product tag is read from the photographed image information has been described, but further, it may be associated with the image recognition of the product displayed on the display shelf. That is, the product identification information such as the product name of the product displayed on the display shelf may be specified from the information described in the product tag and collated with each other.

In this case, the image recognition system 1 of the first embodiment constitutes a part of the functions of the image recognition system 1 of the second embodiment. The image recognition system 1 in the second embodiment includes a photographed image information input reception processing unit 20, a photographed image information storage unit 21, a photographed image information storage unit 21, a photographed image information emplacement processing unit 22, and a position identification process. Unit 23, product tag recognition processing unit 28, display product recognition processing unit 29, sample image information storage unit 30, product identification information storage unit 31, consistency determination processing unit 32, and recognition result matching processing unit 33. And have. FIG. 23 shows an example of the system configuration of the image recognition system 1 in the second embodiment. Further, FIG. 24 shows an example of the configuration of the product tag recognition processing unit 28, and FIG. 25 shows an example of the configuration of the display product recognition processing unit 29.

The management terminal 2 in the image recognition system 1 can send and receive information via a network with the captured image information input terminal 4.

The photographed image information input reception processing unit 20, the photographed image information storage unit 21, and the photographed image information emplacement processing unit 22 are the same as those in the first embodiment.

The position identification processing unit 23 specifies a shelf area (shelf area) in addition to the function of the position identification processing unit 23 in the first embodiment. That is, among the display shelves shown in the photographed image information and the vertical image information, there is a shelf area where products are displayed and a product tag arrangement area where product tags for the products displayed there are attached. .. Therefore, the product tag placement area and the shelf stage area are specified from the normal image information. To specify the product tag placement area and the shelf level area, the operator of the management terminal 2 may manually specify the product tag placement area and the shelf level area, and the position identification processing unit 23 may accept the product tag placement area and the shelf level area for the first time. The product tag placement area and the shelf area may be automatically specified from the second time onward based on the information of the product tag placement area and the shelf area for which the input is manually received.

The product tag recognition processing unit 28 executes the product tag recognition process. That is, each process of the product tag arrangement area cutting processing unit 24, the product tag arrangement area normalization processing unit 25, the product tag identification processing unit 26, and the product tag in-information identification processing unit 27 in the image recognition system 1 of the first embodiment. To execute. The processing in each processing unit is the same as in the case of the image recognition system 1 of the first embodiment.

The display product recognition processing unit 29 executes a process of recognizing the products displayed on the shelves in the display shelf shown in the photographed image information. The display product recognition processing unit 29 includes a shelf stage area cutting processing unit 291, a face identification processing unit 292, a product identification information identification processing unit 293, and a shelf stage image matching processing unit 294.

The shelf area cutout processing unit 291 cuts out the image information of the shelf area specified by the position specifying processing unit 23 as the shelf area image information. The shelf area cutout processing unit 291 may actually cut out as image information, or may cut out virtually without actually cutting out as image information. If there are multiple shelves on the display shelf, each is cut out as shelf area image information.

The face specifying processing unit 292 identifies the face on which the product is placed (the area on which the product is placed) for each shelf in the shelf area in the normal image information. The face identification processing unit 292 is divided into a first face identification process and a second and subsequent face identification processes.

The first face identification process in the face identification processing unit 292 is performed in the area (preferably a rectangular area) in which the product is placed within the area composed of the coordinates of the shelf step specified by the position identification processing unit 23 (preferably a rectangular area). Face) is identified. Specifically, it is separated based on restrictions such as identifying thin and narrow shadows that occur between products, identifying repeating patterns of images, identifying steps on the upper side of the package, and having the same product width. Specify the area of the face by specifying the position. As the face identification process, any method can be adopted depending on the product category and the product form, and the process is not limited to the above. In addition, the person in charge may accept correction input for the automatically identified face. Furthermore, the input of the position of the face may be accepted from the person in charge. The coordinates of the area constituting the specified face are the coordinates of the face area in the normal image information, the shooting date / time information, the store information, the image information identification information of the shot image information, the image identification information of the normal image information, and the face. It manages the face identification information for identification in association with each other. The coordinates indicating the face area are the coordinates of the vertices required to specify the rectangular area, and may be, for example, the coordinates of four points, or the coordinates of two points, upper right and lower left, upper left and lower right.

In the second and subsequent face identification processes in the face identification processing unit 292, the coordinates of the face area specified in the previous (N-1th) normal image information for the same shelf on the same display shelf are used this time (N-1). It is the coordinates of the face area specified by the normal image information of the Nth time).

Similar to the coordinates of the position of the shelf, the coordinates of the face area are relative coordinates based on a predetermined position in the image information (for example, the upper left vertex C1 of the display shelf) such as the display shelf in the vertical image information.

The product identification information identification processing unit 293 specifies the product identification information of the product displayed on the face for each shelf of the display shelf. The product identification information includes, but is not limited to, the product name and the JAN code assigned to the product. Any information that can identify the product may be used.

The product identification information identification processing unit 293 executes the following processing. That is, by matching the image information of the face with the sample image information of the product stored in the sample image information storage unit 30 described later for each face, the product identification information of the product displayed on the face is specified. Specifically, first, the similarity between the image information of the region composed of the coordinates of the face to be processed and the sample image information stored in the sample image information storage unit 30 is determined, and the similarity is the highest. The product identification information corresponding to the sample image information is specified as the product identification information of the product displayed on the face composed of the above coordinates.

Here, in order to determine the similarity between the face image information and the sample image information, the following processing is performed. First, in the processing before the product identification information identification processing in the product identification information identification processing unit 293, the directions of the image information of the face area on the shelf of the orthodox image information and the sample image information are the same (rollover or rollover). It is not inverted), and the size of each image information is approximately the same (if the size of the image information is different within a predetermined range, each before determining the similarity. Adjust the size so that the size of the image information in is within the specified range). The similarity between the face image information and the sample image information is the image feature amount (for example, local feature amount) between the feature point based on the image feature amount (for example, local feature amount) of the face image information and the sample image information. ) Is extracted. Then, the pair having the highest similarity between the feature points of the face image information and the feature points of the sample image information is detected, and the difference in the coordinates of the corresponding points is obtained for each pair. Then, find the average value of the difference. The average value of the difference indicates the total average moving amount of the face image information and the sample image information. Then, the coordinate difference of all the feature point pairs is compared with the average coordinate difference, and the pair with a large degree of deviation is excluded. Then, the similarity is ranked by the number of remaining corresponding points.

The similarity between the face image information and the sample image information can be calculated by the above method. Further, in order to improve the accuracy, the EMD (Earth Movers Distance) between the color histograms may be obtained and used as a measure of similarity. As a result, it is possible to compare the similarity of the captured image information, which is relatively strong against environmental changes such as the brightness information, and to specify it with high accuracy. The determination of similarity is not limited to the above. The identified product identification information corresponds to the shooting date / time information, the store information, the image information identification information of the shooting image information, the image identification information of the normal image information, and the face identification information for identifying the face, and the product identification information storage unit. Store in 31.

The product identification information identified as described above is associated with the shooting date / time information, the store information, the image information identification information of the shooting image information, the image identification information of the orthodox image information, and the face identification information for identifying the face. It is stored in the product identification information storage unit 31.

The shelf image matching processing unit 294 includes the image information of the shelf area in the previous (N-1st) normal image information and the image information of the shelf area in the current (Nth) normal image information. If the similarity is high, it is determined that the product identification information of each face on the shelf is the same. As described above, this similarity determination process is performed on the image feature amount of the image information of the shelf area in the previous (N-1st) orthogonal image information and the current (Nth) orthogonal image information. The similarity may be determined based on the image information of the shelf area, or the EMD of the color histograms may be used. Moreover, it is not limited to them. Then, instead of the specific processing for each face in the product identification information identification processing unit 293, the product identification information identification processing unit 293 is provided with the product identification information of each face on the shelf in the Nth orthodox image information. It is stored in the product identification information storage unit 31 as the same as the product identification information of each face on the same shelf for the first time. As a result, it is possible to omit the processing for the shelves that hardly change, such as the shelves where the products do not move much and the shelves that are managed in an extremely short cycle. It is not necessary to provide the processing by the shelf image matching processing unit 294.

The sample image information storage unit 30 stores the sample image information for identifying which product is the product of each face on the shelf of the display shelf shown in the normal image information. Specimen image information is image information obtained by photographing products that may be displayed on a display shelf from a plurality of angles such as up and down, left and right, and diagonally. FIG. 28 shows an example of the sample image information stored in the sample image information storage unit 30. FIG. 28 shows the case where the canned beer is photographed from various angles as the sample image information, but the sample image information is not limited to the canned beer. The sample image information storage unit 30 stores the sample image information and the product identification information in association with each other.

In the sample image information storage unit 30, information necessary for calculating similarity, for example, an image feature amount and its position, which is extracted from the sample image information together with the sample image information or instead of the sample image information, is stored in the sample image information storage unit 30. The pair information may be stored. It is assumed that the sample image information also includes the information necessary for calculating the similarity. In this case, the product identification information identification processing unit 293 does not have to calculate the image feature amount for the sample image information every time when performing the matching process between the image information in the face area and the sample image information, and the calculation is performed. You can save time.

The product identification information storage unit 31 stores the product identification information of the product displayed on each face of the shelf of the display shelf. For example, product identification is associated with product identification information, such as shooting date / time information, store information, image information identification information of shot image information, image identification information of orthodox image information, and face identification information for identifying faces. It is stored in the information storage unit 31.

The consistency determination processing unit 32 determines each shelf and each shelf regarding the recognition result of the information displayed on the product tag by the product tag recognition processing unit 28 and the recognition result of the product (product identification information) by the display product recognition processing unit 29. Determine the consistency of the product that is likely to be included in the stage. For example, in the recognition result of the information displayed on the product tag or the recognition result of the product, the product is recognized as having a rating of 350 ml, but the product of 500 ml is displayed on the display shelf or the shelf of the product. If it is stipulated, change the rating of the same product name to "500 ml". It is preferable that the products to be placed on the display shelf or the shelf are preset and associated with the photographed image information stored in the photographed image information storage unit 21.

The recognition result collation processing unit 33 collates the product identification information of the product for each face recognized by the display product recognition processing unit 29 with the product information (product identification information) recognized by the product tag recognition processing unit 28, and the recognition result. Check if they match.

Specifically, first, as a result of the recognition process by the display product recognition processing unit 29, the number of groups on one shelf is specified as a group of sections in which faces having high similarity are lined up. Also, identify where the left and right positions of the shelves in each group are. Then, the information of the product tag recognition result by the product tag recognition processing unit 28, whose left and right positions are the same as those of each group, is associated with each group.

When the group by face and the product tag are associated, the following processing is executed. First, as a result of product tag recognition by the product tag recognition processing unit 28, the read product name (product identification information) is converted into a candidate product list with likelihood. In this case, the probability of being the product A1 is converted to p1, the probability of being the product A2 is p2, and so on. If the manufacturer name and rating information are obtained from the recognition result of the product tag, the likelihood of the product of that manufacturer and the product having the rating is set high. If price information is obtained from the recognition result of the product tag, the likelihood of the product having that price range as the selling price is increased. Furthermore, if two prices, the tax-excluded price and the tax-included price, are read and the ratio between them exactly matches the presence or absence of consumption tax, and if it matches the logical check, the likelihood is further increased. Set high. In addition, at this time, if the genres of the products displayed on the shelves of the display shelves are known, the likelihood of the products belonging to those genres is set high or low.

Then, as a result of the recognition by the display product recognition processing unit 29, the information of the recognized product is also given a candidate product list with a likelihood based on the degree of image similarity. In this case, the probability of being the product B1 is converted to Pb1, the probability of being the product B2 is Pb2, and so on.

Then, the list of each candidate product such as products A1 and A2 based on the recognition result of the product tag is compared with the list of each candidate product such as products B1 and B2 based on the recognition result of the display product recognition processing unit 29, and both products are both. It is classified into one of (Case 1), which appears only in Group A (Case 2), and (Case 3) which appears only in Group B. For Case 1, the likelihood of the products in Group A and Group B, respectively. Is synthesized, and the likelihood is higher than any of them. For cases 2 and 3, a composite function that reflects the consensus of the accuracy of the product tag recognition result by the product tag recognition processing unit 28 and the accuracy of the product information recognition result by the display product recognition processing unit 29 is applied. By doing so, a final list of candidate products with likelihood is generated.

By generating the final list of candidate products with likelihood as described above, the candidate products can be ranked and specified. Therefore, for example, the highest (first) candidate product is used as the product. It may be confirmed, or the candidate products from the first place to the predetermined order may be displayed and the selection input of the visual judgment result may be accepted.

Then, the likelihood of the read price is calculated again for the confirmed product. That is, the ratio of the tax-excluded price to the tax-included price, whether it is within the range of the selling price of the product, or the consistency or divergence with the frequent price of the product is determined, and the likelihood of the price is determined. Then, if this likelihood is higher than a predetermined threshold value, the price may be automatically determined, and if it is lower, that fact may be reflected in the display and input by selection may be accepted.

Next, an example of the processing process of the image recognition system 1 in the second embodiment will be described with reference to the flowchart of FIG. The same processing as in Example 1 will be omitted.

The photographed image information (FIGS. 21 and 22) photographed on the display shelf of the store is input from the photographed image information input terminal 4, and the input is accepted by the photographed image information input reception processing unit 20 of the management terminal 2 (S100). .. It also accepts input of image information identification information such as shooting date and time, store identification information, and shooting image information. Then, the captured image information input reception processing unit 20 stores the captured image information, the shooting date and time, the store identification information, and the image information identification information of the captured image information in association with each other in the captured image information storage unit 21.

When the management terminal 2 receives a predetermined operation input, the photographed image information emplacement processing unit 22 extracts the photographed image information stored in the photographed image information storage unit 21, and is a shelf which is a vertex for performing trapezoidal correction processing. The input of four points of the position (position of the display shelf) is received, and the keystone correction process is executed (S110). 7 and 8 show an example of the captured image information (normal image information) in which the keystone correction process is executed in this way.

Then, the position specifying processing unit 23 specifies the shelf stage area and the product tag placement area by accepting a predetermined operation input in the management terminal 2 for the normal image information (S120). That is, the input of the shelf area and the product tag arrangement area in the normal image information is accepted. 29 and 30 are diagrams showing a state in which the shelf area and the product tag arrangement area are specified.

When the shelf area and the product tag arrangement area are specified as described above, the display product recognition processing unit 29 performs the display product recognition process in the shelf area, and the product tag recognition process in the product tag arrangement area performs the product tag recognition process. Each unit 28 executes. The display product recognition processing unit 29 may perform the display product recognition processing and the product tag recognition processing unit 28 may perform the product tag recognition processing in parallel or at different timings.

The product tag recognition process (S130 to S160) in the product tag recognition processing unit 28 is the same as in the first embodiment. That is, the product tag placement area cutting processing unit 24 in the product tag recognition processing unit 28 cuts out the image information of the product tag placement area specified in S120 (S130), and the product tag placement area normalization processing unit 25 performs keystone correction processing. Is executed to execute the orthostatic processing (S140).

When the product tag placement area normalization processing is completed in S140, the product tag identification processing unit 26 in the product tag recognition processing unit 28 identifies each product tag area from the image information of the product tag placement area. (S150).

When the product tag specifying processing unit 26 specifies the product tag area, the information specifying processing unit 27 in the product tag specifies the information in the product tag (S160). By this identification, the information described in the product tag, for example, the information such as the tax-excluded price, the tax-included price, the product name (product identification information), and the rating can be recognized in characters.

Next, the display product recognition processing by the display product recognition processing unit 29 will be described.

The shelf area cutout processing unit 291 cuts out the image information of the shelf area from the normal image information based on the area of the shelf that has received the input in S120 (S170). Then, a process of specifying the face is executed for each shelf in the shelf area image information (S180). Specifically, for the shelf in the shelf area, within the range of the rectangular area composed of the coordinates of four points, the narrow and narrow shadows generated between the products are specified, and the repeating pattern of the image is specified. , The face is specified by specifying the step on the upper side of the package, specifying the dividing position based on restrictions such as the same product width. Face identification information for identifying the face is attached to the specified face. Then, the coordinates of each of the specified faces are stored in association with the shooting date and time, store identification information, image information identification information of the shot image information, image information identification information of the orthodox image information, and face identification information for identifying the face. Let me. The coordinates of the face may be two points that can specify the rectangular area without storing four points.

When each face of each shelf in the shelf position area image information of the normal image information is specified as described above, the product identification information identification processing unit 293 stores the sample image in the sample image information storage unit 30 for each face. The information and the matching process are executed, and the product identification information of the product displayed on the face is specified (S190). That is, from the image information in the rectangular region of the face of a certain shelf (the face identification information of the face in this region is X) and each sample image information stored in the sample image information storage unit 30, each image feature amount is obtained. Is calculated and a pair of feature points is obtained to determine the similarity. Then, the sample image information having the highest similarity is specified, and if the similarity at that time is equal to or higher than a predetermined threshold value, the product identification information corresponding to the sample image information is obtained based on the sample image information storage unit 30. Identify. Then, the specified product identification information is used as the product identification information of the product displayed on the face of the face identification information X. Then, the product identification information identification processing unit 293 associates the identified product identification information with the shooting date / time, store identification information, image information identification information of the shot image information, image information identification information of the normal image information, and face identification information. It is stored in the product identification information storage unit 31 (S200).

Not all face product identification information can be specified. Therefore, for faces that have not been identified, the input of product identification information is accepted, and the product identification information for which the input is accepted is used as the shooting date and time, store identification information, image information identification information of shot image information, and image information of orthodox image information. It is stored in the product identification information storage unit 31 in association with the identification information and the face identification information. Similarly, input may be accepted for the correction process of the specified product identification information.

By performing the above processing, it is possible to identify the product identification information of the product displayed on the shelf of the display shelf shown in the photographed image information.

In this way, there is a possibility that the product tag recognition processing unit 28 includes the product tag recognition result, the display product recognition processing unit 29 recognizes the displayed product, and the consistency determination processing unit 32 is included in each shelf and each shelf. The consistency of whether or not the product is expensive is determined (S210). For example, in the product tag recognition result or the product recognition result, it is recognized that the product has a rating of 350 ml, but it is stipulated that a product of 500 ml is displayed on the display shelf or shelf of the product. In that case, the rating of the same product name is changed to "500 ml".

Further, the recognition result collation processing unit 33 collates the product identification information of the product for each face recognized by the display product recognition processing unit 29 with the product information (product identification information) recognized by the product tag recognition processing unit 28. It is collated whether the recognition results match (S220).

That is, as a result of the recognition processing by the display product recognition processing unit 29, the recognition result collation processing unit 33 specifies how many groups are on one shelf, with the sections in which faces having high similarity are lined up as a group. Also, identify where the left and right positions of the shelves in each group are. Then, the information of the product tag recognition result by the product tag recognition processing unit 28, whose left and right positions match with each group, is associated with the group.

After associating the group with the product tag by the face in this way, the product identification information (product name) corresponding to the face or group stored in the product identification information storage unit 31 and the product tag recognition by the product tag recognition processing unit 28. Compare the results, identify if those recognition results match, and determine the read price.

By the above processing, it is possible to compare and collate the result of recognizing the product displayed on the display shelf by the image recognition processing and the result of recognizing by the character recognition processing by the product tag.

In the above-mentioned Examples 1 and 2, the keystone correction process is executed by designating four points, but it is burdensome to specify and input the reference vertex each time. Therefore, it may be configured to automatically identify the vertex that is the reference of the keystone correction process. The processing in this case will be described.

In this case, the captured image information normalization processing unit 22 is divided into the first keystone correction process and the second and subsequent keystone correction processes. In addition to the first time, the first time includes the case of manually performing at an arbitrary timing in order to correct the deviation when automatically identifying the vertices. The second and subsequent times are other than the first time.

In the first keystone correction processing in the captured image information emplacement processing unit 22, the input of the designation of the four vertices of the rectangular area of the display shelf is accepted as in the first embodiment. The four vertices of the rectangular area of the display shelf may be the four vertices of the shelf position of the display shelf, the four vertices of the shelf stage, or the four vertices of the area to which the product tag is attached. Further, it may be four vertices of a group of two or three shelves. The four vertices that have been designated here are stored in association with the image information identification information of the shooting date / time information, store information, and shooting image information. Then, the captured image information normalization processing unit 22 executes trapezoidal correction processing on the captured image information based on the coordinates of the four vertices that have received the designation, and obtains the captured image information as the normal image information.

It is desirable that the captured image information be captured in a similar area at a similar angle at regular intervals. However, it is not possible to shoot the exact same area at the same angle. Therefore, the captured image information normalization processing unit 22 executes the second and subsequent keystone correction processing as follows.

First, the captured image information emplacement processing unit 22 sets the vertex coordinates of the N-1st captured image information obtained by photographing the same (almost the same) region corresponding to the Nth captured image information at the time of the previous processing. Identify from the stored information. The vertex coordinates of the N-1st shot image information obtained by shooting the same (almost the same) area corresponding to the Nth shot image information are used as store identification information, image identification information, shooting date and time information, etc. corresponding to the shot image information. Identify based on. Then, with respect to the N-1th captured image information, a rectangular area having a predetermined size including the coordinates of the specified four vertices, for example, a square having a width of about 1/5 of the shelf level is used as the feature collection area 220. Set as. FIG. 31 shows an example of a state in which the feature amount collection area 220 is set for the N-1th captured image information. The feature amount collection area 220 may be a rectangular area including the coordinates of the vertices. On the other hand, if the backgrounds of the display shelves match each other, the backgrounds will shift significantly even if the shooting position shifts slightly. Therefore, it is preferable to set the feature quantity collection area 220 at a position that includes as much of the inside of the display shelf as possible. That is, it is preferable that the apex coordinates are located outside the display shelf in the feature collection area 220 with respect to the center point of the feature collection area 220.

For example, the four points of vertex coordinates are located at the upper left, upper right, lower left, and lower right. Then, when the rectangular area of the feature amount collection area 220 is divided into two areas at the vertical and horizontal centers, a total of four areas are divided. In the feature amount collection area 220 including the upper left vertex coordinates, the vertex coordinates are the upper left area of the rectangular area. The feature amount collection area 220 is set so as to be located at. Similarly, in the feature amount collection area 220 including the upper right vertex coordinates, the feature amount collection area 220 is set so that the vertex coordinates are located in the upper right area of the rectangular area, and the feature amount collection including the lower left vertex coordinates is set. In the area 220, the feature amount collection area 220 is set so that the vertex coordinates are located in the lower left area of the rectangular area, and in the feature amount collection area 220 including the lower right vertex coordinates, the vertex coordinates are in the rectangular area. The feature amount collection area 220 is set so as to be located in the lower right area. As a result, the apex coordinates are located in the feature collection area 220 in the direction outside the display shelf from the center point of the feature collection area 220.

Next, the captured image information emplacement processing unit 22 includes the feature amount collection area 220 set in the N-1st captured image information in the Nth captured image information, and the N-1th captured image information. A feature collection area 221 having a size of 220 or more is set. The feature amount collection area 221 set in the Nth captured image information does not exceed half the size of the short side. Furthermore, if it goes outside the captured image information, the range is trimmed. FIG. 32 shows an example of a state in which the feature amount collection area 221 is set for the Nth captured image information.

Then, the captured image information emplacement processing unit 22 collects the local feature amount in each feature amount collection area 220 set for the N-1th photographed image information, and the feature points based on the local feature amount and their coordinates. Memorize the set. Further, in each feature amount collection area 221 set for the Nth captured image information, the local feature amount is collected, and the feature point based on the local feature amount and the set of its coordinates are stored.

The captured image information emplacement processing unit 22 corresponds to the local feature amount of the feature point in the feature amount collection area 220 of the N-1st shot image information and the feature amount collection area 220 of the N-1th shot image information. The feature amount of the Nth photographed image information at the position where the image is to be taken is compared with the local feature amount of the feature point in the feature collection area 221. Then, the feature point of each local feature amount of the Nth shot image information, which is the closest to each local feature amount of each feature point of the N-1th shot image information, is specified. Then, the feature points of the closest local features are paired, and the coordinates of the feature points according to the paired local features are associated. At this time, pairs whose proximity (similarity) between local features is less than a predetermined threshold are excluded. Thereby, the pair of the feature points of the local feature amount in the N-1th feature amount collection area 220 and the feature points of the closest local feature amount in the Nth feature amount collection area 221 can be specified. FIG. 33 shows the relationship between the feature points of the local feature amount of the N-1th feature amount collection area 220 and the feature points of the local feature amount of the Nth feature amount collection area 221. In FIG. 33, the point cloud of the feature point by the local feature amount in the N-1th feature amount collection area 220 is A, and the point cloud of the feature point by the local feature amount in the Nth feature amount collection area 221 is B, N-. The vertices used for the first keystone correction process are indicated by C.

The coordinates of the point cloud A of the feature points due to the local feature amount in the N-1st feature amount collection area 220 and the point cloud B of the feature points due to the local feature amount in the Nth feature amount collection area 221 corresponding to the point group A. Based on the coordinates of, the function F (affine transformation) that projects the point cloud A onto the point cloud B is obtained. The function F has methods such as repeating sampling estimation and using RANSAC of OpenCV, which is a kind of robust estimation, but is not limited thereto. Pairs with a large deviation from the relational line that is in the projection relationship are excluded from the processing target.

After the function F is obtained by the captured image information normalization processing unit 22, the captured image information normalization processing unit 22 determines the coordinates of the vertices C used in the N-1th keystone correction processing based on the function F. It is projected onto the Nth captured image information and specified as the coordinates of the vertex D for the Nth trapezoidal correction process. FIG. 34 schematically shows this.

By performing the above processing for each feature amount collection area 220 and 221, the four vertices of the shelf position for the keystone correction processing in the Nth captured image information are specified. Then, the captured image information normalization processing unit 22 executes trapezoidal correction processing for the Nth captured image information based on the specified four vertices to normalize the captured image information, and generates and stores the normal image information. At this time, the captured image information normalization processing unit 22 corresponds to the shooting date / time information, the store information, the image information identification information of the shooting image information, and the image identification information of the normal image information in association with the normal image information. Let me remember it. The coordinates of the vertices corresponding to the specified Nth shot image information are stored in association with the shooting date / time information, store information, and image information identification information of the shot image information.

Note that the vertex identification process used in the trapezoidal correction processing in the captured image information emplacement processing unit 22 is not limited to the case of identifying a product from the image information obtained by photographing the display shelf as in the present invention, and is the same object to be photographed. It can also be applied to the image recognition system 1 in which a plurality of image information obtained by capturing the image is rectified and the rectified image information is generated. As a result, it is possible to generate the orthodox image information of the same object to be photographed by rectifying each of a plurality of image information of the same object to be photographed.

Further, in order to specify the four vertices of the shelf position for the trapezoidal correction processing in the Nth shot image information, the feature points based on the local feature amount in the feature amount collection area 220 in the N-1th shot image information are described above. The function F was obtained by using the point cloud A of No. 1 and the point cloud B of the feature points based on the local feature amount in the feature amount collection area 221 in the Nth shot image information, and used in the N-1th trapezoidal correction process. The process of projecting the coordinates of the vertex C onto the captured image information of the Nth time by the function F and specifying the coordinates of the vertex D as the coordinates of the vertex D for the Nth trapezoidal correction process has been described. However, in such processing, it is sufficient to find the coordinates (positions) of the corresponding points of similar image information in the N-1st shot image information and the Nth shot image information. However, any feature quantity of a type that specifies a part in the image information may be used. For example, there are sharp points and highlight points in the image information. In the present specification, a feature amount that identifies a location in image information, such as a local feature amount, is referred to as an image feature amount (position-specific image feature amount). In the description of the present specification, the case where the local feature amount is used as the image feature amount as described above will be described.

Next, the vertex identification process for performing the keystone correction process in the third embodiment will be described. In this case, the apex for performing the keystone correction processing has already been specified in the photographed image information obtained by photographing an arbitrary display shelf, and after a predetermined period (for example, one week) has elapsed, a similar area is formed for the same display shelf. A case will be described in which the captured image information captured at the same angle is used.

The photographed image information photographed on the display shelf of the store is input from the photographed image information input terminal 4, and the input is accepted by the photographed image information input reception processing unit 20 of the management terminal 2. FIG. 35 shows an example of captured image information. It also accepts input of image information identification information such as shooting date and time, store identification information, and shooting image information. Then, the photographed image information input reception processing unit 20 stores the photographed image information, the shooting date and time, the store identification information, and the image information identification information that received the input in the photographed image information storage unit 21 in association with each other.

When the management terminal 2 receives a predetermined operation input, the captured image information normalization processing unit 22 extracts the captured image information stored in the captured image information storage unit 21 and executes the keystone correction processing at the shelf position. A process for identifying the vertices D (D1 to D4) is executed.

The vertex coordinates (for example, vertex coordinates C1 to C4) of the previous (N-1st) captured image information (FIG. 36) in which the same or almost the same area corresponding to the captured image information (FIG. 35) of this time (Nth) was captured. To identify). The vertex coordinates of the previously captured image information may be specified based on the shooting date and time, the store identification information, the image identification information of the captured image information, and the like.

The captured image information emplacement processing unit 22 extracts the N-1th captured image information from the captured image information storage unit 21, and for each of the vertices C1 to C4, a rectangle having a predetermined size including one vertex. The area is set as the feature amount collection area 220 and set as the N-1th captured image information. FIG. 36 shows a state in which the feature amount collection area 220 is set in the N-1th captured image information.

Further, the captured image information emplacement processing unit 22 extracts the Nth captured image information (FIG. 35) from the captured image information storage unit 21, and features in a wider range than the N-1th feature amount collection area 220. The amount collection area 221 is set as the Nth captured image information. FIG. 37 shows a state in which the feature amount collection area 221 is set for the Nth captured image information. Each feature amount collection area 221 in the Nth photographed image information includes one N-1th feature amount collection area 220. In FIG. 37, the inclusion relationship is shown by showing the N-1th feature collection area 220 in the Nth feature collection area 221.

Then, the captured image information emplacement processing unit 22 collects the local feature amount in each feature amount collection area 220 set for the N-1th photographed image information, and sets the feature points and coordinates according to the local feature amount. Remember. In addition, the local feature amount is collected in each feature amount collection area 221 set for the Nth captured image information, and the feature point and the set of coordinates according to the local feature amount are stored.

The captured image information emplacement processing unit 22 is the feature amount collection area 221 of the Nth shot image information closest to each local feature amount of each feature point in the feature amount collection area 220 of the N-1th shot image information. Identify the feature points of the local features in, and associate them with the respective coordinates as the feature points of the paired local features. FIG. 33 shows the relationship between the N-1st feature amount collection area 220 and the Nth feature amount collection area 221.

Then, the point cloud of the feature point by the local feature amount in the N-1th feature amount collection area 220 is A, and the point cloud of the feature point by the local feature amount in the Nth feature amount collection area 221 is B, N-1th time. Assuming that the vertices used for the trapezoidal correction processing of the above are C (C1 to C4), the captured image information emplacement processing unit 22 sets the point cloud A to the point cloud B based on the coordinates of the point cloud A and the point cloud B. Find the function F (Affin transformation) that projects to.

Then, the captured image information emplacement processing unit 22 projects the coordinates of the vertices C (C1 to C4) used in the N-1th keystone correction process based on the obtained function F, and performs the Nth keystone correction process. It is specified as the coordinates of the vertices D (D1 to D4) for.

By performing the above processing for each feature amount collection area 220 and 221, the four vertices D (D1 to D4) for the Nth keystone correction processing can be automatically specified. The coordinates of the specified Nth vertex D (D1 to D4) are stored in association with the image information identification information of the shooting date and time, the store identification information, and the shooting image information. FIG. 38 shows the identified vertices D1 to D4.

As described above, when the vertices D (D1 to D4) of the shelf position for the keystone correction processing for the Nth captured image information are specified, the captured image information emplacement processing unit 22 determines the vertices D (D1 to D4). ), The keystone correction process is executed for the Nth captured image information.

By executing the above processing, it becomes possible to automatically identify the corresponding vertices for the second and subsequent keystone correction processing without specifying the four vertices used in the keystone correction processing. The burden on the person can be reduced.

Further, a modification of the face identification process in the face identification process unit 292 of the second embodiment will be described. In this embodiment, the face identification process of the second embodiment may be performed as the first process, and the face identification process of the second and subsequent times may be performed as the face identification process. The processing in this case will be described.

In addition to the first time, the first time includes the case where the processing of the second embodiment is performed at an arbitrary timing in order to correct the deviation at the time of automatic identification. The second and subsequent times are other than the first time.

The face identification processing unit 292 executes the same processing as the processing of the second embodiment as the first face identification processing. Then, in the second and subsequent face identification processes in the face identification processing unit 292, the coordinates of the face area specified by the previous (N-1th) normal image information are used for the same shelf on the same display shelf. Extract and use the coordinates as the coordinates of the face area specified by the normal image information of this time (Nth time).

Similar to the coordinates of the position of the shelf, the coordinates of the face area are relative coordinates based on a predetermined position in the display shelf (for example, the apex C1 on the upper left of the display shelf) in the orthodox image information.

Further, as a modification of Example 2, the process of Example 2 is first performed as a process of specifying the product identification information of the product displayed on the face for each shelf of the display shelf in the product identification information identification processing unit 293. The following processing is executed as the identification processing of the product identification information of the above, and as the identification processing of the product identification information from the second time onward.

The product identification information identification processing unit 293 first identifies the face identification information of the face to be processed in the identification process of the product identification information of the face in the Nth normal image information. Let X be the specified face identification information. Then, the image information of the area of the face identification information X in the Nth normal image information is compared with the image information of the area in the N-1th normal image information at the position corresponding to the face identification information X. Regarding the determination of similarity, it is preferable, but not limited to, to obtain the EMD between color histograms and use it as a measure of similarity. If the similarity is equal to or higher than a certain threshold value, the product identification information corresponding to the face of the region in the N-1th normal image information is extracted from the product identification information storage unit 31, and the Nth normal image information. It is the product identification information of the face identification information X in the above. Thereby, the product identification information of the face identification information X in the Nth normal image information to be processed can be specified. If the similarity is less than a certain threshold, the image information in the area of the face identification information X in the Nth orthodox image information and the sample image information stored in the sample image information storage unit 30 are the same as in the first case. The product identification information having the similarity equal to or higher than a predetermined threshold value and having the highest similarity is specified as the product identification information of the face of the face identification information X in the Nth orthodox image information.

In the comparison with the image information of the face area in the N-1th orthodox image information, not only the comparison with the position of the corresponding face but also the faces in a predetermined range may be included as the comparison target. For example, when comparing with the image information in the area of face identification information X in the Nth normal image information, the comparison target is from the area of face identification information X in the N-1th normal image information and from that area. The area of the face within a predetermined range, for example, the area of the face located one or more apart in the left-right direction thereof, and the area of the face located on the upper and lower shelves may be included. Further, in addition to the area of the face identification information X in the N-1th orthodox image information, the area of a plurality of adjacent faces such as the face identification information X-2, X-1, X, X + 1, X + 2 is included. You may.

In this case, the image information in the area of the face identification information X in the Nth normal image information and the image information in the area of the face range in the N-1th normal image information to be compared. To identify the face identification information of the N-1th orthodox image information having the highest similarity. The similarity may be conditional on the similarity being equal to or higher than a certain threshold value. Then, the product identification information corresponding to the face identification information of the specified N-1th normal image information is extracted from the product identification information storage unit 31, and the product identification information of the face identification information X in the Nth normal image information is obtained. do. FIG. 27 schematically shows this process. FIG. 27 (a) is the previous (N-1st) normal image information, and FIG. 27 (b) is the current (Nth) normal image information. Then, the image information of the area of each face on the shelf 1 of the N-th vertical image information is compared with the image information of the area of each face of the shelf 1 of the N-1th normal image information. The product identification information of the face of the shelf 1 of the N-1th normal image information, which has the highest similarity, is used as the product identification information of the face of the shelf 1 of the N-1th normal image information. Indicates that it is specified as. FIG. 27 shows a case where, in addition to the face of the N-1th normal image information at a position corresponding to the face of the Nth normal image information, comparison is performed with two faces on the left and right of the face. .. Not only the same shelf, but also the image information of the face positions of the upper and lower shelves may be compared. For example, in the case of FIG. 27, when specifying the product identification information of the face position at the center of the shelf 2 of the Nth normal image information, the center of the shelf 2 of the N-1th normal image information and its left and right sides thereof. Not only comparing with the image information of two face areas, but also the image information of the center of the shelf 1 of the N-1th orthodox image information and the two face areas on the left and right, N-1st. You may compare the image information and the similarity with the image information of the center of the shelf 3 and the two face regions on the left and right of the vertical image information of the above.

As a result of comparison with the image information of the face of the N-1th orthodox image information, if the product identification information cannot be specified because the similarity does not satisfy the threshold, the face in the Nth orthodox image information is not specified. By comparing the image information in the region of the identification information X with the sample image information stored in the sample image information storage unit 30, the product identification information having a similarity equal to or higher than a predetermined threshold and having the highest similarity is obtained as N. The face identification information X in the first orthodox image information is specified as the product identification information of the face. The similarity determination process in this case can be performed in the same manner as the initial product identification information identification process.

The product identification information identified as described above is associated with the shooting date / time information, the store information, the image information identification information of the shooting image information, the image identification information of the orthodox image information, and the face identification information for identifying the face. It is stored in the product identification information storage unit 31.

In the first to fifth embodiments of the image recognition system 1 of the present invention, the accuracy of the image information to be processed in each processing unit is improved by executing the orthostatic processing as the preprocessing. In each of the above examples, the case was described. However, it is not always necessary that the image information to be processed has been subjected to the orthostatic processing. In that case, the processing target of each processing unit is the image when the orthostatic processing is not executed. Information. For example, the position identification processing unit 23, the product tag arrangement area cutting processing unit 24, the product tag arrangement area normalization processing unit 25, the product tag identification processing unit 26, the product tag in-information identification processing unit 27, and the product tag recognition processing unit. 28, Display product recognition processing unit 29, shelf area cutout processing unit 291, face identification processing unit 292, product identification information identification processing unit 293, shelf image matching processing unit 294 have not executed the orthotopic processing. Image information may be processed. In this case, even if the emplacement processing is not executed, the processing for the image information showing the display shelf on which the products are displayed may be executed. The image information showing the display shelves on which the products are displayed includes both the image information that has been subjected to the orthostatic processing and the image information that has not been orientated.

By using the image recognition system 1 of the present invention, it is possible to accurately identify the products displayed on the display shelves.

1: Image recognition system 2: Management terminal 4: Photographed image information input terminal 20: Photographed image information input reception processing unit 21: Photographed image information storage unit 22: Photographed image information emplacement processing unit 23: Position identification processing unit 24: Product tag placement area cutout processing unit 25: Product tag placement area normalization processing unit 26: Product tag identification processing unit 27: Product tag information identification processing unit 28: Product tag recognition processing unit 29: Display product recognition processing unit 30 : Specimen image information storage unit 31: Product identification information storage unit 32: Consistency determination processing unit 33: Recognition result collation processing unit 220: N-1 Feature amount collection area in the first image information 221: Features in the Nth image information Quantity collection area 291: Shelf stage area cutout processing unit 292: Face identification processing unit 293: Product identification information identification processing unit 294: Shelf stage image matching processing unit 70: Computing device 71: Storage device 72: Display device 73: Input device 74: Communication device

Claims (3)

A first emplacement processing unit that performs a first emplacement process on the first image information showing a display shelf for displaying products and generates a second image information, and a first emplacement processing unit.
A second emplacement processing unit that performs a second emplacement process on the area including the product tag arrangement area in the second image information, and
A product tag identification processing unit that specifies a product tag area from the image information that has undergone the second orthostatic processing, and a product tag identification processing unit.
An information identification processing unit in the product tag that specifies the information written on the product tag by performing OCR recognition processing in the specified product tag area, and
An image recognition system characterized by having. The information identification processing unit in the product tag
The box is specified by binarizing the specified product tag area and performing labeling processing.
Among the specified boxes, the boxes whose height, width, and baseline satisfy the predetermined conditions are specified as blocks.
The OCR recognition process is executed for the specified block.
The image recognition system according to claim 1. Computer,
A first emplacement processing unit, which performs a first emplacement process on the first image information showing a display shelf for displaying products and generates a second image information.
A second emplacement processing unit that performs a second emplacement process on an area including a product tag arrangement area in the second image information.
Product tag identification processing unit that specifies the product tag area from the image information that has undergone the second orthostatic processing,
Information identification processing unit in the product tag that specifies the information written on the product tag by performing OCR recognition processing in the specified product tag area,
An image recognition program characterized by functioning as.

Images