JP7660819B2 - Information Processing System - Google Patents

Description

The present invention relates to an information processing system that detects POPs attached to display shelves installed in stores and to products displayed there, and executes processing related to their effects.

In convenience stores, supermarkets, and other stores, it is common for products to be sold by placing them on display shelves. Therefore, by displaying multiple products on a display shelf, even if one product is purchased, another person can purchase the same product. Furthermore, managing where and how many products are displayed on the display shelf is important in terms of product sales strategy.

POPs may also be attached to display shelves or products to promote or advertise the products on display. POPs are promotional materials such as advertisements, testers, and sample products used to sell the products on display.

POPs are important items for product manufacturers and retailers in selling products. For this reason, systems that print POPs according to the sales status of products have been disclosed, for example, in Patent Document 1 below.

JP 2022-20094 A Patent No. 6877806

The invention of Patent Document 1 allows POPs to be printed according to sales status, so for example, POPs can be printed to advertise products with unfavorable sales.

On the other hand, since POPs are important items for selling products, it is desirable to understand the effects of POPs, such as what effect they have, which products they should be attached to, and where they should be attached to be effective. However, the above-mentioned Patent Document 1 does not allow such understanding.

Therefore, Patent Document 2 discloses a system that identifies POP advertisements that affect store sales.

However, while the patented invention in Patent Document 2 describes how to identify whether image data contains a POP advertisement, it does not describe how to identify the POP advertisement.

It is not easy to identify products to be displayed from image data of display shelves, and because POPs are required to be conspicuous, a variety of shapes and designs are sometimes adopted. For this reason, it is not easy to consistently identify POPs from images, and the above-mentioned Patent Document 2 does not state at all how to identify them, so the system is not feasible.

Thus, to understand the effectiveness of POPs, it is necessary to be able to automatically and stably detect POPs from image data captured on display shelves, but no such system has existed in the past. Furthermore, it has not been possible to estimate the effectiveness of POPs detected from image data.

In view of the above problems, the inventors have invented an information processing system that detects POPs from image data captured of display shelves and performs processing related to the effects of the POPs.

The first invention is an information processing system that displays the effects of a POP, the information processing system having a POP detection processing unit that detects a POP from image data showing a display shelf, a linking processing unit that links the POP with the product corresponding to the POP, and an effect display processing unit that displays the effect of the POP, wherein the POP detection processing unit determines a POP or a POP candidate by determining the type of an object detected from the image data using the area of the detected object and the type of the detected object, and the effect display processing unit identifies a POP to be processed and calculates the effect of the POP by performing a calculation using an index value based on the type of the identified POP.

By configuring the present invention as described above, it is possible to link POPs with products and then calculate the effect of the POP.

In the above-mentioned invention, the effect display processing unit can be configured as an information processing system that identifies a POP corresponding to the product for which a selection has been accepted, and classifies the product into a predetermined rank by performing a calculation using an index value based on the type of the identified POP.

In the above-mentioned invention, the effect display processing unit can be configured as an information processing system that displays and outputs part or all of the area of the product or its corresponding POP in a display method corresponding to the rank .

By executing the processes of these inventions, it is possible to display and output the effects of POP.

In the above-mentioned invention, the effect display processing unit can be configured as an information processing system which divides the POP effect items and one or more of the customer feedback, recognition or sales items into ranks, and displays and outputs the items in a switchable manner for each of the items in a display method corresponding to the rank in part or all of the area of the product or its corresponding POP.

In the above-mentioned invention, the effect display processing unit can be configured as an information processing system which divides the POP effect items and one or more of the customer feedback, recognition or sales items into ranks, and displays and outputs two or more items simultaneously in part or all of the area of the product or its corresponding POP in a display method according to the rank.

By executing the processes of these inventions, the effects of POP and other items can be displayed in a switchable or simultaneous manner.

The information processing system of the first invention can be realized by loading and executing the program of the present invention into a computer. That is, the information processing program causes a computer to function as a POP detection processing unit that detects a POP from image data showing a display shelf, a linking processing unit that links the POP with a product corresponding to the POP, and an effect display processing unit that displays an effect of the POP, in which the POP detection processing unit determines a type of an object detected from the image data using an area of the detected object and the type of the detected object to determine a POP or a POP candidate, and the effect display processing unit specifies a POP corresponding to a product whose selection has been accepted, and calculates the effect of the POP by performing a calculation using an index value based on the type of the specified POP.

By using the information processing system of the present invention, it is possible to detect POPs from image data captured of display shelves and perform processing related to the effects of the POPs.

1 is a block diagram illustrating an example of a configuration of an information processing system according to the present invention. 4 is a block diagram illustrating an example of a configuration of a POP detection processing unit in the information processing system of the present invention. FIG. 4 is a block diagram illustrating an example of a configuration of an effect estimation processing unit in the information processing system of the present invention. FIG. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. 4 is a flowchart showing an example of the entire process in the information processing system of the present invention. 10 is a flowchart showing an example of a POP detection process in the information processing system of the present invention. 11 is a flowchart showing an example of an effect estimation process in the information processing system of the present invention. FIG. 2 is a diagram showing an example of photographed image data. FIG. 13 is a diagram showing an example of a screen for accepting selection of a POP or product. FIG. 13 is a diagram showing an example of the relationship when there are multiple POPs. FIG. 13 is a diagram showing an example of a display of the contribution of POP for each product. FIG. 11 is a diagram showing an example of a result of executing an object detection process from image data. 13A and 13B are diagrams illustrating an example of a process for determining whether a region of an object detected from image data satisfies a predetermined condition and thereby determining a POP region. This figure shows an example of each area after the area of an object detected as either a product, product tag, or shelf component in the object detection results of Figure 12 is determined to be a POP area if the area satisfies certain conditions. FIG. 11 is a block diagram illustrating an example of a configuration of a POP detection processing unit according to a second embodiment. 13 is a flowchart illustrating an example of a POP detection process according to the second embodiment. FIG. 11 is a diagram illustrating an example of an outline of a process according to a second embodiment. FIG. 11 is a diagram illustrating an example of processing by an individual discrimination processing unit in the second embodiment. 13 is a diagram showing a schematic diagram of a state in which input of designation of a product display area, a product tag area, and a shelf top area has been received for image data. FIG. 13 is a flowchart illustrating an example of a POP detection process according to the third embodiment. FIG. 13 is a block diagram illustrating an example of a configuration of an information processing system according to a fourth embodiment. FIG. 13 is a block diagram illustrating an example of a configuration of a linking processing unit in a fourth embodiment. FIG. 13 is a block diagram illustrating an example of a configuration of an effect display processing unit in a fourth embodiment. 13 is a flowchart illustrating an example of an entire process in an information processing system according to a fourth embodiment. 13 is a flowchart illustrating an example of a linking process in the fourth embodiment. 13 is a flowchart showing an example of an effect display process in the fourth embodiment. FIG. 13 is a diagram showing an example of a screen for accepting selection of a product for which a POP effect is to be estimated in the fourth embodiment. FIG. 10 is a diagram illustrating an example of index values according to the type of POP. FIG. 10 is a diagram illustrating an example of colors displayed according to ranks. FIG. 13 is a diagram showing an example in which the area of a product is output based on a color determined by a display control processing unit in the fourth embodiment. 13 is a diagram showing an example of a case where a POP area is output based on a color determined by a display control processing unit in the fourth embodiment. FIG. FIG. 13 is a diagram showing an example of a case where the product area is divided into categories and a color corresponding to the rank of each category is output in Example 5.

Figure 1 shows a block diagram of an example of the overall processing function of the information processing system 1 of the present invention, Figure 2 shows an example of the processing function of the POP detection processing unit 20, and Figure 3 shows an example of the processing function of the effect estimation processing unit 22. The information processing system 1 uses a management terminal 2 and an image data input terminal 3. The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The image data input terminal 3 is a terminal that inputs image data of photographs of display shelves in a store.

The management terminal 2 and image data input terminal 3 in the information processing system 1 are realized using a computer. FIG. 4 shows a schematic example of a computer hardware configuration. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

If the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be configured as an integrated unit. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

A touch panel display is a device that combines the functions of a display device 72 and an input device 73 in that input can be made directly on the display using a specified input device (such as a touch panel pen) or a finger.

The image data input terminal 3 may be equipped with a photographing device such as a camera in addition to the above devices. A portable communication terminal such as a mobile phone, smartphone, or tablet computer may also be used as the image data input terminal 3. The image data input terminal 3 photographs the display shelves in the store and inputs image data of the display shelves (photographed image data) into the management terminal 2. Products are displayed on the display shelves, and POPs are attached to the display shelves, their vicinity, or the products. Therefore, the image data includes images of the displayed products, such as the products displayed on the display shelves, product tags, shelf components, and POPs. It is preferable that unnecessary objects such as people are not photographed.

The functions of each means in the present invention are only logically distinct, and may be physically or practically the same area. The order of processing in each means of the present invention may be changed as appropriate. Also, some of the processing may be omitted. For example, the rectification processing described below may be omitted. In that case, processing can be performed on image data that has not been rectified.

The information processing system 1 has a POP detection processing unit 20, a data storage unit 21, and an effect estimation processing unit 22.

The POP detection processing unit 20 executes a process to detect POPs or candidates for POPs that appear in the photographed image data or image data that has been subjected to the normalization process described below (normalized image data) based on the photographed image data. The data of the image to be processed, including the photographed image data and normalized image data, is collectively referred to simply as "image data".

The data storage unit 21 stores image data received from the image data input terminal 3, the shooting date and time, store identification information, display shelf identification information that identifies the display shelf, image data identification information, and the like, in association with each other. The image data may be any image data that is the subject of processing in the present invention. In general, when simply shooting, it is difficult to shoot the subject in a straight-on state, so it is a good idea to perform a correction process to correct it to a straight-on state, such as a keystone correction process. When multiple images of a single display shelf are taken, image data in which these are combined into one image data is also included. In addition, image data after distortion correction process has been performed is also included. The data storage unit 21 also stores various data (including information) used in the processing of the information processing system 1 of the present invention.

The effect estimation processing unit 22 estimates the effect of the POP using the POP detected in the POP detection process.

The POP detection processing unit 20 has an image data input reception processing unit 200, an image data alignment processing unit 201, an object detection processing unit 202, and an individual discrimination processing unit 203.

The image data input reception processing unit 200 receives input of image data (photographed image data) of a store's display shelves photographed by the image data input terminal 3, and stores the data in the data storage unit 21. In addition to the photographed image data, the image data input terminal 3 may also receive input of store identification information such as the date and time of photographing and the store name, display shelf identification information for identifying the display shelf, and image data identification information for identifying the photographed image data. An example of photographed image data is shown in FIG. 8. Note that FIG. 8 shows a case where the photographed image data shows one display shelf, but multiple display shelves may be photographed, or a portion of a display shelf may be photographed. In addition, although the processing is not specified in the present invention, display shelves and shelf levels may be long in the horizontal direction. Therefore, in the processing, the image may be divided into sections of a certain width, and each section may be the processing target.

The image data orthogonalization processing unit 201 performs a process (orthogonalization process) on the captured image data stored in the data storage unit 21 to correct the captured object so that it is facing upright, for example, keystone correction process, to generate orthogonalized image data. Keystone correction process is a correction process performed so that the shelves of the display shelves captured in the captured image data are horizontal, and the products displayed there and the product tags (e.g. price tags) for those products are vertical. Orthogonalization is a process of transforming the captured image data so that the optical axis of the lens of the imaging device is aligned perpendicular to the plane of the captured object, so that it is the same as if it were captured from a sufficiently distant location, for example, keystone correction process. The image data that has been orthogonalized (orthogonalized image data) may be stored in the data storage unit 21 described above.

The keystone correction process executed by the image data alignment processing unit 201 accepts input of the designation of four vertices in the captured image data, and executes the keystone correction process using each of those vertices. The four vertices that are accepted for designation may be the four vertices of a shelf on a display shelf, or the four vertices of the shelf position on a display shelf. They may also be the four vertices of a group of two or three shelf levels. Any four points can be designated as the four vertices. The image data alignment processing unit 201 can use various known processing methods.

The object detection processing unit 202 executes a process of detecting objects appearing in the captured image data stored in the data storage unit 21 or image data rectified by the image data rectification processing unit 201. The object detection process is preferably performed using so-called deep learning. In this case, the image data is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the type (class) of the object appearing in the input image data is output as an output value.

As a learning model, products, product tags (price tags), shelf components, etc. that may appear in the image data are used as annotation data, and a predetermined learning process is performed using the annotation data to learn the above-mentioned learning model. For example, data that associates image data of a product with the fact that it is a product or with the product name or product code (JAN code, etc.) is used as annotation data, and a learning process for the learning model is performed. Image data is image data in the annotation data, and is a class indicating that it is a product, or that it is a product and the product identification information such as the product name or product code is a "product type". Similarly, image data of a product tag is image data in the annotation data, and is a class indicating that it is a product tag, or that it is a product tag and the product name or product identification information is a "product tag type". Furthermore, image data of a shelf component is image data in the annotation data, and is a class indicating that it is a shelf component or the part name of the shelf component is a "shelf component". The object types (classes) used as annotation data by the object detection processing unit 202 include things that are typically displayed on display shelves, such as merchandise, product tags, and shelf components, but are not limited to these, and the system may be trained to detect any object.

The individual discrimination processing unit 203 discriminates that the object types (classes) detected from the image data by the object detection processing unit 202 as types other than merchandise, product tags, and shelf components are POPs or candidates for POPs. Furthermore, when an object detected as a type (class) of merchandise, product tags, or shelf components satisfies a predetermined condition, the individual discrimination processing unit 203 discriminates that the object type (class) is POP or a candidate for POP.

For example, for an object detected as a type of commodity, it is determined whether or not there is a product-specific advertisement (POP) attached to the area, and if it is determined that a POP is present, the type of object in that area is determined to be a POP or a POP candidate. In this case, there is a method of detecting a POP attached to a displayed product from the area of an object detected as a type of commodity, and a method of detecting a POP by comparing the image data of the area of an object detected as a type of commodity with the image data of a product without a POP.

In the former method, a large number of image data including POP parts attached to displayed products are first collected in advance, and the POP parts are annotated for each image data including a POP part. A learning model that has been previously trained using image data of normal displayed products without POPs as annotation data is then trained using the image data annotated with the POP parts to generate a new learning model. Image data of the area of an object detected as a type of product can be input to this new learning model, and during product recognition processing, in addition to outputting product identification information, it can also additionally output whether or not a POP is attached to the displayed product in the image data.

In the latter method, after performing product recognition processing on image data of the area of an object detected as a type of merchandise, standard image data (e.g., sample data) without a POP of the recognized product is compared with the image data of the area of the object, and the similarity is compared. If the similarity is equal to or greater than a predetermined value, it is determined that the object detected as a type of merchandise does not have a POP, and if the similarity is less than the predetermined value, it is determined that the object detected as a type of merchandise has a POP or a candidate for a POP.

In the above-mentioned method, by comparing image data of an area of an object detected as a commodity with standard image data of the commodity without a POP recognized by the commodity recognition process, it is possible to determine whether or not it is a POP based on the location of the differing area, which allows for more accurate determination. For example, if it is determined that the area that differs between the two image data is localized in a specific position, such as near the top, shoulder, or tail of the commodity, it can be determined that it is a POP or a candidate for a POP.

Various known processing methods can be used for the product recognition process. For example, a process of identifying product identification information can be executed for image data of the object area using deep learning. In this case, the image data of the object area can be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the product identification information can be identified based on the output value. As a learning model, a learning model in which product identification information is given as correct answer data to various image data in which the product is captured can be used. In addition, product sample data (image data of the product captured from one or more directions or data such as features based thereon) can be compared with image data of the object area, and the product identification information can be identified using the similarity. Product recognition processes are not limited to these.

In addition, for objects detected as being of a product tag type, it is determined whether the image data of the object's area meets the requirements of a product tag type. For example, OCR recognition processing is performed on the area of an object detected as being of a product tag type, and if it contains characters typically found in product tags, such as the price and product name, it is determined to be a product tag, and if not, it is determined to be a POP or a candidate for POP. Alternatively, if the results of the OCR recognition processing show that it contains specific words typically used in POPs, such as "member discount," "tester," "sample," or "sample," it may be determined to be a POP or a candidate for POP. Also, if handwritten characters are included, it may be determined to be a POP or a candidate for POP.

For objects detected as being of a shelf component type, the system determines whether the image data of the object's area meets the requirements for a shelf component. For example, the system performs OCR recognition processing on the area of an object detected as being of a shelf component type, and if it contains a specific word, such as a member discount or other word that is typically used in POPs, it is determined to be a POP or a candidate for POP, and if not, it is determined to be a shelf component.

In addition, if the individual discrimination processing unit 203 determines that the type (class) of the image information of the object area is a candidate for POP, the image information of the object area may be displayed on the display device 72 of the management terminal 2, etc., and input as to whether it is POP or not may be accepted. If the input is POP, the type (class) of the image information of the object area may be set to POP.

When the individual discrimination processing unit 203 discriminates a POP area, it assigns POP identification information for identifying the POP and stores it in the data storage unit 21 in association with the image data of the POP area. It is also preferable that the image data in which the POP is shown be associated with the shooting date and time, store identification information, and display shelf identification information.

The effect estimation processing unit 22 has a POP selection reception processing unit 220, a product selection reception processing unit 221, an analysis processing unit 222, and an effect output processing unit 223.

The POP selection reception processing unit 220 extracts image data of the POP area detected by the POP detection processing unit 20 from the data storage unit 21, and receives the selection of the POP to be subjected to effect estimation.

The product selection reception processing unit 221 extracts products identified from the image data from the data storage unit 21 and receives the selection of products for which effects are to be estimated. Various known technologies can be used for the process of identifying products from image data and storing them in the data storage unit 21. In addition, image data of products from the image data may be stored in advance in the data storage unit 21.

For example, the area of the product is identified by identifying thin, narrow shadows that occur between products from the image data, identifying repeating patterns in the image, identifying steps on the top edge of the package, identifying dividing positions based on constraints such as the product width being the same, etc. Then, an image matching process is performed on the image information of the product area with pre-stored image data of a reference product (sample data), and the product identification information (a specified code such as the product name or JAN code) of the sample data that is most similar is identified as the product identification information of the product in that area. Through this process, the product shown in the image data can be identified, and the product image data and product identification information can be associated and stored in the data storage unit 21.

As another method, machine learning such as deep learning may be used to identify the product area and the product identification information. In this case, the area to be processed, for example image data, may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product area and product identification information may be identified based on the output value. Furthermore, the learning model may be one in which the product area and its product identification information are given as correct answer data to image data of various areas to be processed, for example display shelves and shelf areas.

The product identification process may be performed by the object detection processing unit 202, the individual discrimination processing unit 203, etc. Image data of the area of the identified product is associated with the product identification information and stored in the data storage unit 21. It is also preferable that the shooting date and time of the image data showing the product, the store identification information, and the display shelf identification information are also associated.

Figure 9 shows an example of a screen that accepts POP selection at the POP selection reception processing unit 220 and product selection at the product selection reception processing unit 221. The effect estimation processing unit 22 accepts the selection of some or all of the store, display shelf, date, etc. for which POP effect estimation is desired, using a predetermined method. Upon accepting the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on the screen shown in Figure 9 above.

The analysis processing unit 222 executes a process for estimating the effect of a POP. For example, when the "Analysis" button on the screen shown in FIG. 9 is pressed, the estimation processing unit receives the POP selected by the POP selection reception processing unit 220 and the product selected by the product selection reception processing unit 221, and extracts from the data storage unit 21 the POP identification information corresponding to the image data of the selected POP, the shooting date and time of the image data in which the POP appears, the store identification information, and the display shelf identification information. In addition, the estimation processing unit extracts from the data storage unit 21 the product identification information corresponding to the image data of the selected product, the product identification information corresponding to the image data in which the product appears, and the shooting date and time of the image data in which the product appears, the store identification information, and the display shelf identification information.

Then, the analysis processing unit 222 acquires sales information (for example, various information related to sales of the product, such as sales volume, sales amount, sales volume, reservation amount, and number of reservations) of the product having the product identification information at the store on the date selected by the effect estimation processing unit 22 from a product sales management system such as a POS system. Note that this date and time is not limited to the date, but may include multiple dates before and after the date. In other words, it is sufficient to acquire sales information of the product having the product identification information at the store on the date on which the POP was installed. Also, input of a date on which the POP was not installed is accepted, and sales information of the product having the product identification information is acquired. Note that the number of days on the date on which the POP was installed and the date on which the POP was not installed should be the same for comparison. It may be one day, one week, or any appropriate number of days.

The analysis processing unit 222 compares the sales information of the product on a date with a POP and the sales information of the product on a date without a POP, as described above, and calculates the contribution of the POP (an index showing the improvement in the number of sales of the product due to the POP, for example, the rate of increase). For example,
(Number of items sold on days with POP) ÷ (Number of items sold on days without POP)
The contribution degree is calculated by calculating

As another calculation formula, the contribution of the POP may be calculated by taking into account the number of faces (the number of image data of the product area). The number of faces is the number of products displayed on the display shelf shown in the image data, and can be calculated for each product. Since the object detection processing unit 202 detects image data of the area having the type of product, the area is regarded as a face. Then, in the above-mentioned product identification process, the number of image data of the product area having the same identification information as the product identification information of the selected product in the same image data in which the selected product is shown is identified. This number is the number of faces. The information on the number of faces may be stored in the data storage unit 21 in association with the image data, product identification information, etc. Therefore, the number of faces of the product on the display shelf on the date on which the selected POP is installed and the number of faces of the product on the same display shelf (display shelf with the same display shelf identification information) on the date on which the POP is not installed are extracted from the data storage unit 21.

Then, the analysis processing unit 222 performs, for example,
(Number of items sold on days with POP / Number of faces of items on days with POP) ÷ (Number of items sold on days without POP / Number of faces of items on days without POP)
The contribution degree is calculated by calculating

The formula for calculating the contribution rate is not limited to the above, and any formula may be used as long as it is possible to compare sales information such as the number of sales on days with POP and the number of sales on days without POP.

Furthermore, when there are multiple POPs, the contribution degree may be calculated taking into account the influence of the multiple POPs. For example, when two POPs (POPa, POPb) are selected, there are combinations of the presence or absence of each POP as shown in FIG. 10. Assuming that the number of faces is the same for all four cases according to the presence or absence of POPa and POPb, the number of faces for each case is normalized, and the contribution degree of POPa can be calculated by dividing C by D in FIG. 10, and the contribution degree of POPb can be calculated by dividing B by D. In addition, the contribution degree when POPa and POPb are present can be calculated by dividing A by D, and this value may be used to correct the contribution degree of POPa and the contribution degree of POPb. When the number of POPs is three or more, the number of faces for each case can be normalized similarly, assuming that the number of faces is the same for each case, and the calculation can be performed in the same manner.

The effect output processing unit 223 outputs the contribution degree calculated by the analysis processing unit 222. For example, FIG. 11 shows an example of the POP contribution degree output for each product.

As described above, it is possible to detect POP from image data and estimate the effect of the POP on product sales.

Next, an example of a processing process using the information processing system 1 of the present invention will be described with reference to the flowcharts in Figures 5 to 7.

First, the process of detecting POPs contained in image data (S100) will be explained using the flowcharts in Figures 5 and 6.

Image data of a store's display shelves is input from the image data input terminal 3, and the input is accepted by the image data input acceptance processing unit 200 of the management terminal 2 (S200). The image data input acceptance processing unit 200 also accepts input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data. The image data input acceptance processing unit 200 then stores the accepted input image data, shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data in association with each other in the data storage unit 21. Note that the store identification information, display shelf identification information, and image data identification information may be assigned automatically.

When a specific operation input is received at the management terminal 2, the image data alignment processing unit 201 extracts the image data to be stored in the data storage unit 21, receives input of the four shelf positions (positions of the display shelves) that are the vertices for performing alignment processing such as keystone correction processing, and executes the alignment processing (S210).

Then, by receiving a predetermined operation input at the management terminal 2, the object detection processing unit 202 executes an object detection process on the normalized image data (S220) and classifies the type of object depicted in the image data (S230). An example of the result of executing the object detection process from the image data is shown in FIG. 12. In FIG. 12, "a" means products, "b" means product tags, "c" means shelf parts, and "d" means the type of object detected other than a to c. Note that the type of object detected from the image data only needs to be associated with each area, and is not written on each image. Also, the area of the detected object may be cut out from the image data, but it does not have to be actually cut out as long as the image data of the area is in a processable state. Note that cutting out the image data of the area includes not only actually cutting out the image data of the area from the image data, but also specifying the image data of the area as a processable state without cutting out it.

The individual discrimination processing unit 203 discriminates and cuts out areas of objects of types other than products, product tags, and shelf components (areas of object "d") from the object areas of the image data detected by the object detection processing unit 202 as POP areas (S240, S250).

The individual discrimination processing unit 203 also determines whether a region of an object in the image data detected by the object detection processing unit 202 (S240) that is detected as a type of commodity satisfies a predetermined condition, and determines whether the region is a POP region (S260).

For example, for an area of an object detected as a type of commodity, it is determined whether or not the area has a product-specific advertisement (POP), and if it is determined that a POP is present, the type of the area is determined to be POP. For an area of an object detected as a type of commodity, a process is performed to detect POPs attached to displayed products, or POPs are detected by comparing image data of the area of an object detected as a type of commodity with image data of products without POPs (sample image data of the products).

Similarly, for an object area detected as a type of product tag, a determination is made as to whether the image data of the object area meets the requirements for a product tag. Furthermore, for an object detected as a shelf component, a determination is made as to whether the image data of the object area meets the requirements for a shelf component.

As described above, for the area of an object detected as a type of merchandise, product tags, or shelf components (S240), it is determined whether the area satisfies a specific condition for each object (S260), and if the condition is satisfied, all or part of the area is determined to be a POP (S270). An example of this process is shown in Figure 13. Also, for the object detection results in Figure 12, an example of each area after determining that the area satisfies the specific condition in the area of an object detected as a type of merchandise, product tags, or shelf components is a POP area is shown in Figure 14.

By performing the above-described processing in the individual discrimination processing unit 203, it is possible to detect POP areas in image data.

When the POP detection processing unit 20 detects image data of a POP area as described above, it assigns POP identification information and stores it in the data storage unit 21 in association with the image data of the POP area. At this time, the shooting date and time of the image data in which the POP was detected, store identification information, display shelf identification information, and image data identification information may also be stored in association with each other. Note that for image data other than the image data of the POP area, the shooting date and time, store identification information, display shelf identification information, and image data identification information of each detected image data may also be stored in association with each other. In addition, for image data of an object area detected as a commodity or a commodity tag, information such as the commodity identification information of the commodity shown in the image data of that area, the commodity identification information shown in the commodity tag, the price, and the manufacturer may be detected using publicly known technology, and further stored in association with each other in the image data storage unit 21.

Then, when the POP detection processing unit 20 detects a POP, the effect estimation processing unit 22 executes a POP effect estimation process using the POP detected by the POP detection processing unit 20 (S110).

First, the effect estimation processing unit 22 receives the selection of the store, display shelf, and date for which the effect of the POP is to be estimated using a predetermined method. Upon receiving the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on a screen such as that shown in FIG. 9.

The selection of the POP for which the effect is to be estimated from this screen is accepted by the POP selection acceptance processing unit 220 (S300), and the selection of the product is accepted by the product selection acceptance processing unit 221 (S310).

The analysis processing unit 222 extracts, based on the POP selected and accepted by the POP selection acceptance processing unit 220, POP identification information and other information corresponding to that POP from the data storage unit 21, and based on the product selected and accepted by the product selection acceptance processing unit 221, product identification information and other information corresponding to that product from the data storage unit 21.

Then, the analysis processing unit 222 acquires sales information of the product corresponding to the product identification information from a product sales management system such as a POS system based on the date information, product identification information, store identification information, etc., on which the selection was accepted by the effect estimation processing unit 22 (S320). Note that if the selection of date, store identification information, or display shelf identification information is not accepted in the effect estimation process, the date, store identification information, and display shelf identification information corresponding to the POP selected by the POP selection acceptance processing unit 220 and the product selected by the product selection acceptance processing unit 221 may be extracted.

As described above, the analysis processing unit 222 compares the sales information of the product on a date when the POP is present with the sales information of the product on a date when the POP is not present, and calculates the contribution of the POP (an index showing the improvement in the number of product sales due to the POP, for example, the rate of increase) (S330).
for example,
(Number of items sold on days with POP) ÷ (Number of items sold on days without POP)
The contribution degree is calculated by calculating

The calculated contribution level is output by the effect output processing unit 223 (S340).

By checking the output results output by the effect output processing unit 223 through the above process, such as the output results shown in Figure 11, the estimated POP effect can be recognized.

Next, another embodiment of the detection process in the POP detection processing unit 20 will be described. The detection processing unit in this embodiment further includes an area detection processing unit 204 in addition to the configuration of embodiment 1. An example of the configuration of the POP detection processing unit 20 in this embodiment is shown in FIG. 15, and an example of the POP detection process is shown in the flowchart of FIG. 16. Also, an example of the process overview is shown in FIG. 17, and an example of the individual discrimination process is shown in FIG. 18.

The area detection processing unit 204 detects the product display area where products are displayed, the product tag area where product tags are placed, and the upper shelf area which is the area above the shelf, from the image data (including image data that has been normalized) stored in the data storage unit 21. The product display area, product tag area, and upper shelf area may be detected by the operator of the management terminal 2 manually specifying the product display area, product tag area, and upper shelf area, which the area detection processing unit 204 accepts, or the product display area, product tag area, and upper shelf area may be automatically detected from the second time onwards based on the information on the product display area, product tag area, and upper shelf area that was manually input the first time. The area detection processing unit 204 may also detect areas other than these and process them.

Figure 19 shows a schematic diagram of the state in which input for designating the product display area, product tag area, and top shelf area has been received for image data that has been normalized from image data of a display shelf on which products are displayed.

When detecting the product display area, product tag area, and upper shelf area, the area detection processing unit 204 may use deep learning to identify the product display area, product tag area, and upper shelf area. In this case, the above-mentioned orthogonalized image data may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product display area, product tag area, and upper shelf area may be detected based on the output value. In addition, the learning model may be one in which the product display area, product tag area, and upper shelf area are given as correct answer data to various image data.

The area detection processing unit 204 may use deep learning as described above, or may use various known methods to detect areas.

The area detection processing unit 204 may actually cut out the detected area as image data, or may virtually cut out the area by specifying the range of the area without actually cutting out the area as image data. In this case, the range of the area can be specified by coordinates.

The object detection processing unit 202 in this embodiment may perform object detection processing from the range of image data as in the first embodiment, or may perform the processing for each area detected by the area detection processing unit 204.

The individual discrimination processing unit 203 in this embodiment discriminates the type of each object detected by the object detection processing unit 202 using the type (class) of the object and the area in which the object is located. For example, the discrimination is performed according to predetermined individual discrimination conditions such as the discrimination tables shown in Figs. 17 and 18.

"a1" to "a4", "b1" to "b10", and "d1" to "d5" in Figure 17 correspond to "a1" to "a4", "b1" to "b10", and "d1" to "d5" in Figure 18.

Note that Figures 17 and 18 are merely examples, and the object types are also merely examples. Also, these are merely examples of POP types, and are not limited to these, and can be set arbitrarily.

Next, an example of the POP detection process in the POP detection processing unit 20 of this embodiment will be explained using the flowchart in Figure 16.

Image data of a store's display shelves is input from the image data input terminal 3, and the input is accepted by the image data input acceptance processing unit 200 of the management terminal 2 (S400). The image data input acceptance processing unit 200 also accepts input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data. The image data input acceptance processing unit 200 then stores the accepted input image data, shooting date and time, store identification information, display shelf identification information, and image data identification information in the data storage unit 21 in association with each other.

When a specific operation input is received at the management terminal 2, the image data alignment processing unit 201 extracts the image data to be stored in the data storage unit 21, receives input of four shelf positions (positions of display shelves) that are the vertices for performing alignment processing such as keystone correction processing, and executes alignment processing (S410).

Then, by receiving a specified operation input on the management terminal 2, the object detection processing unit 202 performs area detection processing on the aligned image data to detect the product display area, product tag area, and shelf top area (S420), and determines the range of each area.

In addition, by receiving a predetermined operation input at the management terminal 2, the object detection processing unit 202 executes an object detection process on the normalized image data (S430) and classifies the type of object appearing in the image data (S440). An example of the result of executing the object detection process on the image data is classified as shown in FIG. 12, similar to the first embodiment.

Then, for each object in the image data detected by the object detection processing unit 202, the individual discrimination processing unit 203 classifies the detected object according to predetermined discrimination conditions using the type of the object and the area in which the object is located (the area determined in S420).

The individual discrimination processing unit 203 also uses the type of object in the image data detected by the object detection processing unit 202 and the area in which the object is located to determine whether a predetermined condition is satisfied (S450), and determines whether the type of object is a POP or something other than a POP (products, product tags, shelf components, etc.) (S460, S470). Note that the area in which each object is located can be determined by any appropriate method, for example, by determining whether the center of gravity of the object's area is included in the area of products, product tags, or the top of the shelf.

For example, when an object in a product display area is classified as a commodity, the individual discrimination processing unit 203 judges whether or not there is a product-adjacent POP in the area of the object. For example, as in the first embodiment, discrimination is made by a method of detecting a POP attached to a displayed commodity from the area of the object detected as a commodity, and a method of detecting a POP by comparing image data of the area of the object detected as a commodity with image data of a commodity without a POP. If a product-adjacent POP is detected, the area is discriminated as a POP (a2), and other areas are discriminated as products (a1). If a product-adjacent POP is not detected, OCR recognition processing is performed on the area to discriminate whether it contains a specific word used in POPs such as "tester," "sample," or "sample." If it does not, the object in the area is discriminated as a commodity (a1). If it does, the object in the area is discriminated as a POP such as a tester or sample (a3).

In addition, when an object in the product display area is a type of product tag, the individual discrimination processing unit 203 determines that the object in that area is a POP (d1).

In addition, when an object is in the product display area and the type of the object is a shelf member, the individual discrimination processing unit 203 determines that the object in that area is a shelf member (c).

In addition, if an object is in the product display area and the type of the object is other than products, product tags, or shelf components, it is determined to be a POP (d2).

If an object is in the product tag area and the type of the object is a product, the individual discrimination processing unit 203 determines that the object is a POP such as a tester or sample (d4).

If the object is in the product tag area and the type of the object is a product tag, OCR recognition processing is performed on the object area, and if it is determined that there is handwriting in the area, it is determined that the object is a POP (retail POP) created by a store (retailer) (b4). If the result of OCR recognition processing on the object area is that it is determined that there is no handwriting in the area, and if it is determined that the price and product name are displayed, and if it contains a word indicating a condition such as "member", it is determined that the type of the object is a product tag indicating a set product display (b2). On the other hand, if it is determined that the price and product name are displayed, but the word indicating a condition such as "member" is not displayed, it is determined that the type of the object is a product tag (b1). If it is determined that the price and product name are not displayed and there is a point display, it is determined that the type of the object is a POP with a "point display" (b3). Furthermore, if there is no point display in the above, it is determined that the object is a POP created by the product manufacturer (manufacturer POP) (b9). Note that, although we have shown a case where manufacturer POP and retail POP are distinguished from each other, it is also possible to simply distinguish them as POP.

Furthermore, if an object is in the product tag area and the type of the object is a shelf member, the individual discrimination processing unit 203 determines that the object in that area is a shelf member (c).

If an object is in the product tag area and the type of the object is other than a product, product tag, or shelf component, OCR recognition processing is performed on that area. It is then determined whether it contains specific words used in POPs such as "tester," "sample," or "sample," and if so, it is determined to be a POP such as a tester or sample (d4). On the other hand, if it does not contain specific words used in POPs such as "tester," "sample," or "sample," it is determined to be a POP such as a promotional item (d3).

If the object is in the upper shelf area and the type of the object is a commodity, the individual discrimination processing unit 203 determines that it is a stock commodity (a4).

If an object in the upper shelf area is a product tag, the area of the object is subjected to OCR recognition processing, and if it is determined that there is handwriting in the area, it is determined to be a retail POP (b8). If the result of OCR recognition processing on the area of the object is that it is determined that there is no handwriting in the area, and if it is determined that the area contains a price and product name display and a word indicating a condition such as "member", the type of the object is determined to be a product tag indicating a set product display (b6). On the other hand, if it is determined that the area contains a price and product name display but does not contain a word indicating a condition such as "member", the type of the object is determined to be a product tag (b5). If it is determined that the price and product name display are not included and there is a point display, the type of the object is determined to be a "point display" POP (b7). Furthermore, if there is no point display in the above, it is determined to be a manufacturer POP (b10).

Furthermore, if an object is in the upper shelf area and the type of the object is a shelf member, the individual discrimination processing unit 203 determines that the object in that area is a shelf member (c).

In addition, if an object is in the upper shelf area and the type of the object is other than a product, product tag, or shelf component, it is determined to be a POP such as a large promotional item (d5).

As described above, the individual discrimination processing unit 203 can use the area detected by the area detection processing unit 204 and the type of object detected by the object detection processing unit 202 to determine whether the object is a POP or not, and detect the area of a POP in the image data.

As a modified example of the processing of the first and second embodiments, the processing shown in the flowchart of FIG. 20 may be performed. In this case, if the type of the object detected by the object detection processing unit 202 is other than merchandise, merchandise tags, or shelf members, the individual discrimination processing unit 203 determines that the object is a POP (S560). Then, for other objects, the same processing as the processing of S450 to S470 may be performed. That is, using the type of the object in the image data detected by the object detection processing unit 202 and the area in which the object is located, it is determined whether a predetermined condition is satisfied (S570), and it is determined whether the object is a POP or not a POP (such as merchandise, merchandise tags, or shelf members) (S580, S590). Note that the processing from S500 to S540 may be the same as the processing from S400 to S440.

The processing of Examples 2 and 3 may be used to link POPs with products and estimate the effectiveness of the POPs.

The information processing system 1 of this embodiment has a POP detection processing unit 20, a data storage unit 21, a linking processing unit 23, and an effect display processing unit 24. An example of the configuration of the information processing system 1 in this case is shown in FIG. 21, an example of the configuration of the linking processing unit 23 is shown in FIG. 22, and the configuration of the effect display processing unit 24 is shown in FIG. 23. The processing of the POP detection processing unit 20 may be similar to that of the second or third embodiment.

The linking processing unit 23 links a POP with a product corresponding to the POP (a product advertised by the POP). The linking processing unit 23 has a POP determination processing unit 230 and a corresponding product determination processing unit 231.

The POP determination processing unit 230 determines the text information displayed in the POP area. The POP determination processing unit 230 extracts the image data of the POP area detected by the POP detection processing unit 20 from the data storage unit 21, and determines the text information of the image data displayed in that area using a known method such as OCR recognition processing or deep learning.

The POP determination processing unit 230, for example, binarizes the image data of the POP area and performs OCR recognition processing on the binarized image data. If necessary, it may also perform a determination process such as checking against a dictionary of pre-stored product identification information.

As another process of the POP determination processing unit 230, machine learning such as deep learning may be used to determine the character information of the image data of the POP area. In this case, the image data of the POP area may be input to a learning model in which the weighting coefficients between the neurons of each layer of a neural network consisting of multiple intermediate layers are optimized, and the character information contained in the image data of the area may be determined based on the output value. In addition, the learning model may be one in which the image data of various POP areas and the character information contained in those areas are given as correct answer data.

As a further process of the POP determination processing unit 230, instead of using text information, image data of the product contained therein may be determined. For example, image matching processing or deep learning may be used to determine that the image data displayed in the POP area contains sample data of a product that has been stored in advance.

The corresponding product determination processing unit 231 determines the product that the POP is corresponding to. In other words, it identifies the product that the POP is trying to advertise and links that product to the POP. The corresponding product determination processing unit 231 uses part or all of the text information displayed in the area of the POP determined by the POP determination processing unit 230 to correspond to the product displayed on the display shelf where the POP is located.

The process of matching POPs with products can be performed, for example, as follows. First, store identification information, display shelf identification information, and date information that are associated with the image data of the POP area are extracted from the data storage unit 21. Then, from among the objects of merchandise detected by the object detection processing unit 202 or the like and stored in the data storage unit 21, products that have the same store identification information, display shelf, and date information and whose product identification information matches some or all of the text information determined by the POP determination processing unit 230, in particular the product identification information indicating the product written on the POP, are identified.

This allows the POP to be associated with the product. In this case, the POP identification information of the POP can be associated with the product identification information of the product.

In addition, in order to identify products that correspond to a POP, instead of processing the entire display shelf, the type of POP determined by the individual discrimination processing unit 203 (Figures 17 and 18) may be used to change the range in which to search for corresponding products.

For example, if the product corresponds to the POP "a2" in FIG. 18, it is a POP attached to the product, so it is sufficient to search for the product in the range adjacent to the POP. Also, if the product corresponds to the POP "a3" in FIG. 18, it is a "sample/tester" POP, so it is sufficient to search for the product in the vicinity of the POP, for example, within a range of 2 or 3 faces from the range where the POP is located. Also, if the product corresponds to the POP "b3", "b4", and "b9" in FIG. 18, it is a POP located in the product tag area, so it is normal for the product to be located in the vicinity of the POP, so it is sufficient to search for the product in the range of 2 or 3 faces from the range where the POP is located. "b7", "b8", "b10", and "d5" in FIG. 18 are POPs installed at the top of the display shelf, so the entire product displayed on the display shelf is the search range. "d1" and "d2" in FIG. 18 are POPs installed in the product display area, and products are located nearby, so it is sufficient to search for products near the POPs, for example, within a range of 2 or 3 faces from the range where the POPs are located. Furthermore, "d3" in FIG. 18 is a POP in the product tag area, and "d4" is a "sample/tester" POP in the product tag area, so it is sufficient to search for products near the POPs, for example, within a range of 2 or 3 faces from the range where the POPs are located.

Regarding the positions of the POP and the product, when an object is detected by the object detection processing unit 202, the position of the object can be treated as a face (a certain image area) and the positional relationship in the image data can be stored as coordinates, order, and position.

When the corresponding product determination processing unit 231 determines the product that the POP corresponds to, if the determination is made using image data in the POP determination processing unit 230, the corresponding product determination processing unit 231 can link the POP to the product by associating the product identification information corresponding to the image data of the product contained in the image data displayed in the POP area with the POP identification information.

By performing the above-mentioned processing, the corresponding product determination processing unit 231 can link the POP and the product. Linking the POP and the product can be done, for example, by associating the POP identification information with the product identification information.

The effect display processing unit 24 estimates and displays the effect of the POP associated with (corresponding to) the product. The effect display processing unit 24 has a product selection reception processing unit 240, a POP identification processing unit 241, an index processing unit 242, a display control processing unit 243, and a display output processing unit 244.

The product selection reception processing unit 240 of this embodiment accepts the selection of a product for which the effect of a corresponding POP is to be estimated. There are no limitations on the method of accepting the product selection, and for example, a selection of some or all of the store, display shelf, date, etc. for which the POP effect is to be estimated is accepted in a predetermined manner. Upon accepting the selection, the effect estimation processing unit 22 extracts image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays it as a list as shown on the screen in Figure 27. Then, from this list, the selection of a product for which the effect of a POP is to be estimated is accepted.

The product selection reception processing unit 240 may receive the selection of one product, or multiple products. It may also receive the selection of products on a shelf of a display shelf, or the products on the entire display shelf. When the selection of multiple products is received, the processing in the effect display processing unit 24 is performed for each selected product.

The POP identification processing unit 241 identifies a POP corresponding to a product whose selection has been accepted by the product selection reception processing unit 240. By identifying POP identification information corresponding to the product identification information of a product whose selection has been accepted by the product selection reception processing unit 240, it is possible to identify a POP corresponding to the selected product. Note that a product may be associated with multiple POPs, and in this case, each POP is identified. Note that the POP identification processing unit 241 executes the process by identifying a POP corresponding to a product whose selection has been accepted by the product selection reception processing unit 240, but the POP to be processed may also be identified by accepting a POP selection.

In this case, a selection of some or all of the store, display shelf, date, etc. for which a POP effect estimation is desired is accepted in a specified manner. When the selection is accepted, the effect estimation processing unit 22 extracts image data of the relevant POP from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on the screen. Then, from this list, a selection of a POP for which a POP effect estimation is to be performed is accepted. At this time, the product corresponding to the POP whose selection has been accepted is specified. Through this processing, the same processing can be performed even if a POP selection is accepted instead of a product selection.

The index processing unit 242 performs calculations using index values based on the type of POP, such as summing up index values based on the type of POP corresponding to the selected product.

As shown in FIG. 28, the index processing unit 242 sets index values according to the type of POP, and performs calculations such as adding up the index values according to the type of POP corresponding to the selected product. The calculated index values are then divided into ranks divided into a predetermined number of stages, for example, 10 stages.

The display control processing unit 243 determines the display method according to the ranks divided by the index processing unit 242. For example, it determines the color to be displayed. Rank 1 (low total index value) is displayed in "blue" (or "white" when there is no index value), and rank 10 (high total index value) is displayed in "red", for example, as shown in FIG. 29, with the display divided into 10 colors. This color is displayed in the area of the product or POP. By displaying according to such ranks, it is possible to display something like a heat map.

The display output processing unit 244 of this embodiment displays and outputs the area of the selected product or POP based on the color determined by the display control processing unit 243. An example of this display is shown in Fig. 30 and Fig. 31. Fig. 30(a) shows a case where the display of the product area is changed when one product is selected, and Fig. 30(b) shows a case where the display of the area of each product is changed when multiple products are selected. Fig. 31(a) shows a case where the display of the POP area corresponding to the product is changed when a product is selected, and Fig. 31(b) shows a case where the display of the POP area corresponding to each product is changed when multiple products are selected. In addition to displaying the color of the entire product or POP area, part or all of the product or POP area may be divided into a specified rectangle and the color may be changed within the range of the rectangle.

For convenience of illustration, Figures 30 and 31 in this specification show changes in the display method using patterns rather than colors. In addition to colors, various display methods can be used, such as shades of color, brightness of color, different patterns, different highlighting, and the presence or absence of blinking patterns. Also, instead of displaying on the image data, the rank itself can be displayed as text, etc.

Next, an example of the processing in this embodiment will be explained using the flowcharts in Figures 24 to 26.

First, similar to the second or third embodiment, the POP detection processing unit 20 executes a POP detection process to detect a POP in the image data read from the image data input terminal 3 (S100). The POP detection process can be the same as in each embodiment. The POP detected here is stored in the data storage unit 21. Similarly, the product in the image data is also detected by the object detection processing unit 202, and is also stored in the data storage unit 21.

After executing the POP detection process, the linking process is executed in the linking processing unit 23 (S120).

The POP determination processing unit 230 in the linking processing unit 23 extracts image data of the POP area stored in the data storage unit 21 at any timing, and determines the text information displayed in that area using known methods such as OCR recognition processing and deep learning (S600).

Then, using some or all of the text information determined by the POP determination processing unit 230, the corresponding product determination processing unit 231 determines the product that the POP corresponds to (S610).

The corresponding product determination processing unit 231 extracts the store identification information, display shelf identification information, and date information associated with the image data of the POP area from the data storage unit 21. Then, among the objects of merchandise detected by the object detection processing unit 202 or the like and stored in the data storage unit 21, it identifies products that have the same store identification information, display shelf, and date information, and whose product identification information matches some or all of the text information determined by the POP determination processing unit 230, in particular the product identification information indicating the product written on the POP, thereby associating the POP with the product. The association between the POP and the product can be achieved by storing the POP identification information and the product identification information in the data storage unit 21 in association with each other.

After the linking process in the linking processing unit 23 is completed as described above, the effect display processing unit 24 executes the effect display processing (S130).

The product selection reception processing unit 240 in the effect display processing unit 24 receives, in a predetermined manner, the selection of some or all of the store, display shelf, date, etc. for which the POP effect is to be estimated. Upon receiving the selection, the effect estimation processing unit 22 extracts image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays it as a list as shown on the screen in Figure 27. Then, from this list, the selection of one or more products for which the POP effect is to be estimated is received (S700).

The POP identification processing unit 241 identifies one or more POPs that correspond to the product selected by the product selection reception processing unit 240 (S710).

Then, the index processing unit 242 performs a calculation using an index value based on the type of POP identification information identified by the POP identification processing unit 241, and ranks the POP according to a predetermined rank (S720).

When the index processing unit 242 has performed the ranking, the display control processing unit 243 determines the color to be displayed in the area of the product or the POP corresponding to that product (S740), and the display output processing unit 244 displays the result on the display device 72 of the management terminal 2 as shown in Figures 30 and 31.

By performing the above process, the effectiveness of the POP and the product can be determined.

As a variation of the fourth embodiment, the effect display processing unit 24 may be capable of displaying any other items in addition to the effect estimation based on the link between the POP and the product. The other items may be, for example, customer feedback, recognition, sales, etc., but are not limited to these, and may be other items.

For example, as for customer feedback, the effect display processing unit 24 receives evaluations of a product selected by the product selection reception processing unit 240 through customer questionnaires or the like, converts the evaluations into index values, calculates a total index value for the product based on this, and ranks the product according to a predetermined ranking. The display control processing unit 243 then determines the color to be displayed in the area of the product or the POP corresponding to that product, and the display output processing unit 244 displays this on the display device 72 of the management terminal 2.

For a product whose selection has been accepted by the product selection acceptance processing unit 240, the effect display processing unit 24 accepts the recognition of that product through a recognition questionnaire or the like, converts that recognition into an index value, calculates a total index value for that product based on that, and ranks the product according to a predetermined ranking. The display control processing unit 243 then determines the color to be displayed in the area of the product or the POP corresponding to that product, and the display output processing unit 244 displays that on the display device 72 of the management terminal 2.

Furthermore, sales information for the product corresponding to the product identification information is obtained from a product sales management system such as a POS system based on at least two dates to be compared, the product identification information of the product selected and accepted by the product selection acceptance processing unit 240, store identification information, etc. Then, the sales increase rate is calculated. The increase rate is ranked among predetermined ranks. The display control processing unit 243 then determines the color to be displayed in the area of the product or the POP corresponding to that product, and the display output processing unit 244 displays this on the display device 72 of the management terminal 2.

When the display output processing unit 244 displays the estimated effect of linking a POP with a product, customer feedback, awareness, sales, etc. in the area of the product or the POP corresponding to the product, the display may be switched for each item or may be displayed simultaneously.

For example, by performing a specified operation, a color corresponding to the rank of each item (estimated effectiveness, customer feedback, awareness, and sales) may be displayed in the product or in the area of the POP corresponding to the product, in that order.

The product or the POP area corresponding to the product may be divided into categories, and colors corresponding to the ranks of the categories of effectiveness estimates, customer feedback, awareness, and sales may be displayed simultaneously. For example, areas according to the number of items may be provided in the product area, and colors corresponding to the ranks of each item may be displayed simultaneously there. An example of this is shown diagrammatically in FIG. 32. FIG. 32(a) shows a case in which rectangular areas corresponding to items are provided in the product area and the respective colors are displayed, and FIG. 32(b) shows a case in which the product area is divided according to items, and the respective colors are displayed in each divided area. FIG. 32 also shows an enlarged view of the area of one product, and a similar display process may be performed in the POP area.

By performing the above process, multiple items can be displayed at the same time.

By using the information processing system 1 of the present invention, it is possible to detect POPs from image data captured of display shelves and estimate the effectiveness of the POPs.

1: Information processing system 2: Management terminal 3: Image data input terminal 20: POP detection processing unit 21: Data storage unit 22: Effect estimation processing unit 23: Linking processing unit 24: Effect display processing unit 70: Computing device 71: Storage device 72: Display device 73: Input device 74: Communication device 200: Image data input reception processing unit 201: Image data alignment processing unit 202: Object detection processing unit 203: Individual discrimination processing unit 204: Area detection processing unit 220: POP selection reception processing unit 221: Product selection reception processing unit 222: Analysis processing unit 223: Effect output processing unit 230: POP determination processing unit 231: Supported product determination processing unit 240: Product selection reception processing unit 241: POP identification processing unit 242: Index processing unit 243: Display control processing unit 244: Display output processing unit

Claims (6)

An information processing system for displaying a POP effect,
The information processing system includes:
a POP detection processing unit that detects a POP from image data showing a display shelf;
a linking processing unit that links the POP with a product corresponding to the POP;
An effect display processing unit that displays an effect of the POP,
The POP detection processing unit is
determining a type of the object detected from the image data using an area of the detected object and the type of the detected object, thereby determining whether the object is a POP or a candidate for a POP;
The effect display processing unit:
Identifying a POP to be processed, and calculating the effect of the POP by performing a calculation using an index value based on the type of the identified POP.
An information processing system comprising: The effect display processing unit:
Identifying a POP corresponding to the product selected;
Classifying the POPs into predetermined ranks by performing a calculation using index values based on the identified POP types;
2. The information processing system according to claim 1 . The effect display processing unit:
displaying and outputting the product or a part or the whole of the area of a POP corresponding to the product in a display method according to the rank;
3. The information processing system according to claim 2 . The effect display processing unit:
The POP effect item and one or more items of customer feedback, recognition, or sales are classified into the ranks,
displaying and outputting the product or a part or all of the area of the POP corresponding to the product in a manner that allows switching between the items in a display method according to the rank;
4. The information processing system according to claim 3 . The effect display processing unit:
The POP effect item and one or more items of customer feedback, recognition, or sales are classified into the ranks,
displaying and outputting two or more items simultaneously in a part or all of an area of the product or a POP corresponding to the product in a display method according to the rank;
5. The information processing system according to claim 4. Computer,
a POP detection processing unit that detects POPs from image data showing a display shelf;
a linking processing unit for linking the POP with a product corresponding to the POP;
An information processing program that causes an effect display processing unit to function as an effect display processing unit that displays the POP effect,
The POP detection processing unit is
determining a type of the object detected from the image data using an area of the detected object and the type of the detected object, thereby determining whether the object is a POP or a candidate for a POP;
The effect display processing unit:
Identifying a POP corresponding to the product whose selection has been accepted, and calculating the effect of the POP by performing a calculation using an index value based on the type of the identified POP.
2. An information processing program comprising: