JP7567232B2 - Information processing system, information processing device, information processing method, and program - Google Patents

Description

The present invention relates to displaying product information using a mobile terminal.

A system has been proposed in which products are read in stores using mobile terminals such as smartphones owned by customers. For example, Patent Document 1 proposes a method of reading barcodes attached to products using a mobile information terminal with a camera and displaying the product price. Patent Document 2 describes a method of taking an image of a product at a cash register where the product is paid for, and displaying the image of the purchased product along with the product name, unit price, quantity, total price, etc. of each product.

JP 2004-310274 A JP 2012-043371 A

In a typical store, price tags are attached to the products displayed on the shelves, so customers can find out the price of each product by looking at the price tag. However, it takes a lot of effort for the store to create price tags for each product and attach price tags to the shelves. In particular, when new products are put on display, the price of a product changes, or the tax rate changes, it is necessary to add or update the price tags for the relevant products, which places a heavy burden on the store.

The objective of the present invention is to provide an information processing system that can realize a store where customers can view product prices and product information using mobile devices without attaching price tags to product shelves.

One aspect of the present invention is an information processing system, comprising:
An imaging means for generating an image of an external appearance of a product;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
a display control means for displaying on a display unit a price superimposed screen in which the price is superimposed on the captured image, and then, when a predetermined time has elapsed with the same product displayed on the price superimposed screen, displaying on the price superimposed screen a character string informing a customer that at least one product can now be selected from among the products included on the price superimposed screen ;
Equipped with.

Another aspect of the present invention is an information processing device capable of communicating with a terminal device,
A receiving means for receiving an image of an external appearance of a product from the terminal device;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
and a transmitting means for generating a price overlay screen in which the price is overlayed on the captured image and transmitting the generated price overlay screen to the terminal device, and then, when a predetermined time has elapsed with the same product displayed on the price overlay screen, transmitting a display control signal to the terminal device for causing the price overlay screen to display a character string informing a customer that at least one product can now be selected from among the products included on the price overlay screen.

Another aspect of the present invention is a computer-implemented information processing method, comprising:
receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
A price overlay screen in which the price is overlayed on the captured image is generated and transmitted to the terminal device, and then, when a predetermined time has elapsed with the same product displayed on the price overlay screen, a display control signal is transmitted to the terminal device for causing the price overlay screen to display a character string informing the customer that at least one product can now be selected from among the products included on the price overlay screen .

Another aspect of the present invention is a program,
receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
The computer executes a process of generating a price overlay screen in which the price is overlayed on the captured image and transmitting the generated price overlay screen to the terminal device, and then, when a predetermined time has elapsed with the same product displayed on the price overlay screen, transmitting a display control signal to the terminal device to cause the price overlay screen to display a character string informing the customer that at least one product can now be selected from among the products included on the price overlay screen .

This invention makes it possible to create a store where customers can view product prices and product information using mobile devices without attaching price tags to product shelves.

1 shows a configuration of an information processing system according to an embodiment. 2 shows a schematic configuration of an image processing server. 1 shows an example of a data structure of product information. 13 shows another example of the data structure of product information. 2 shows a schematic configuration of a mobile terminal. A method for obtaining price information using captured images is shown. An example of a product details screen is shown. 13 shows an example of a product details screen for comparing multiple products. This shows a method for obtaining product information using a barcode. 13 is a flowchart of a display process. 13 shows an example of a message display when a captured image contains a large number of products. 13 shows another example of the product details screen.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[Overall system configuration]
FIG. 1 shows the overall configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes an image processing server 2 and a mobile terminal 3. A customer purchases a product in a store 8. In the store 8, the products are displayed on a product shelf 4 and sold. In a normal store, the product shelves have price tags attached thereto, and customers can know the price of the product by looking at the price tag. In contrast, in this embodiment, it is assumed that the product shelves 4 in the store 8 do not have price tags attached thereto. That is, each product is simply displayed on the product shelf 4. Note that, in this embodiment, a description will be given using medicine as an example of a product. As shown in the figure, different types of medicines A to H are displayed on the product shelf 4.

In this embodiment, since there are no price tags on the product shelves 4, customers use their mobile terminals 3 to obtain price information and detailed information about the products (hereinafter referred to as "product details"). Note that "product details" refers to information about product specifications, and includes at least one of the following: product name, manufacturer name, price, ingredients, place of origin, capacity, usage, efficacy, and images of the product or product packaging. Specifically, customers can view price information and product details by displaying the product on their mobile terminals 3 within the store 8. Note that customers can also view product details by using their mobile terminals 3 to read barcodes attached to the products.

The image processing server 2 can communicate with the mobile terminal 3 via a network or the like, the mobile terminal 3 stores information necessary for display control, and transmits display information necessary for the mobile terminal 3 to display various screens to the mobile terminal 3. The image processing server 2 may obtain information necessary for the display control of the mobile terminal 3 from an external device every time the mobile terminal performs display control, or may obtain the information periodically from the external device and store it internally. The image processing server 2 may internally store only a portion of the information necessary for the display control of the mobile terminal, and obtain the remaining information from an external device. The image processing server 2 of this embodiment is described on the assumption that the mobile terminal 3 periodically obtains information necessary for display control from an external device, or that the mobile terminal 3 has previously stored information necessary for display control internally.

The mobile terminal 3 is a terminal used by customers who shop at the store 8, and has functions such as taking images of products displayed on the product shelves 4, reading product barcodes attached to products, and acquiring and displaying price information and detailed product information by communicating with the image processing server 2. Note that for ease of explanation, only one mobile terminal 3 is shown in FIG. 1, but in reality there are as many mobile terminals as there are customers in the store 8. Note that the mobile terminal 3 may be a terminal such as a smartphone owned by the customer, or may be a store-owned terminal that is temporarily lent to the customer when shopping at the store 8.

[Device configuration]
Next, the configurations of the image processing server 2 and the portable terminal 3 will be described with reference to FIGS.
(Image processing server)
2 shows a schematic configuration of the image processing server 2. The image processing server 2 includes a storage unit 23, a communication unit 24, and a control unit 25.

The storage unit 23 is composed of a memory such as a hard disk or a flash memory. The storage unit 23 stores the program executed by the control unit 25, and information necessary for the control unit 25 to execute a predetermined process by executing the program. When the storage unit 23 is provided within the image processing server 2, it is connected to the communication unit 24 and the control unit 25 via the bus line 20. Instead of providing the storage unit 23 within the image processing server 2, information used by the image processing server 2 may be stored in an external device outside the image processing server 2, and the control unit 25 may acquire the information from the external device and perform processing. In this case, the external device is connected to the communication unit 24 and the control unit 25 via a fiber channel, a LAN cable, or the like.

Specifically, the memory unit 23 stores customer information, product information, and image feature information. "Customer information" is information about an individual customer. In this embodiment, in order to receive a service of displaying product information using a mobile terminal 3 at one of the stores 8 belonging to a chain, the customer performs user registration with the chain headquarters in advance. The memory unit 23 stores information about the registered customer, such as the customer ID, the terminal ID of the mobile terminal 3 used by the customer, the customer's past purchase history, and member rank, as customer information. The member rank is set based on the customer's past purchase history at that store and the entire chain of stores, and generally, the higher the past purchase amount, the higher the member rank.

"Product information" is information about each product, and includes price information and product details for each product identification information such as a product ID. "Price information" indicates the price of each product. Product prices are determined for each store, and the price information includes the price for each product for each store. "Product details" is information about product specifications, as described above, and includes at least one of the following: product name, manufacturer name, price, ingredients, place of origin, capacity, usage, efficacy, and images of the product or product packaging. Product details are associated with product identification information. Product identification information is information for identifying the product type, and is, for example, the JAN code or product name indicated by the barcode attached to the product.

Figure 3 shows an example of the data structure of product information. In this example, product information linking product identification information, price information, and detailed product information is prepared for each store and stored in the storage unit 23. Figure 4 shows another example of the data structure of product information. In this example, product information is divided into price information and detailed product information, and stored in the storage unit 23. As shown in Figure 4 (A), the price information indicates the price of each product for each store. Meanwhile, as shown in Figure 4 (B), detailed product information is stored for each product.

"Image feature information" is information on features used for image recognition of products. Specifically, image feature information is information indicating features extracted by image processing from an image of the exterior of each product sold at store 8. Image feature information is stored in association with product identification information. Note that instead of image feature information, the image of the product itself may be stored.

The communication unit 24 communicates with the mobile terminal 3 via wired or wireless communication. The control unit 25 includes a CPU, ROM, RAM, etc. (not shown), and performs various controls on each component in the image processing server 2. The processing executed by the control unit 25 will be described later.

In the above configuration, the image processing server 2 is an example of an information processing device, the communication unit 24 is an example of a receiving means and a transmitting means, the memory unit 23 is an example of a price memory unit, and the control unit 25 is an example of a product recognition means and a price acquisition means.

(Mobile device)
5 shows a schematic configuration of the mobile terminal 3. The mobile terminal 3 includes a display unit 31, an input unit 32, a storage unit 33, a communication unit 34, a control unit 35, and a camera 36. These elements are connected to each other via a bus line 30.

The display unit 31 displays product images captured by the camera 36, product prices, detailed product information, etc., based on the control of the control unit 35. The input unit 32 is an interface that accepts user input, and includes, for example, a touch panel, buttons, and a voice input device.

The storage unit 33 is composed of a memory such as a hard disk or a flash memory. The storage unit 33 stores the programs executed by the control unit 35, and information necessary for the control unit 35 to execute a predetermined process by executing the programs. For example, the storage unit 33 stores a dedicated application program (hereinafter referred to as a "product information display app") that is activated when shopping at the store 8 and controls the display of price information and detailed product information.

The communication unit 34 communicates with the image processing server 2, for example, via a Wi-Fi (registered trademark) device installed in the store. The control unit 35 includes a CPU, ROM, RAM, etc. (not shown), and performs various controls on each component in the mobile terminal 3. The control performed by the control unit 35 will be described later. In the above configuration, the camera 36 is an example of an imaging means, a reading means, and a store identification information acquisition means, and the control unit 35 is an example of a display control means and a customer identification information acquisition means.

The camera 36 is used to capture images of products. When a customer sets the camera 36 to shooting mode and captures an image of a product displayed on the product shelf, the captured image of the product is displayed on the display unit 31. Here, "capturing an image" refers to the state in which an image of the product is captured within the shooting range of the camera 36, and it is not necessary to store or record the image as image data in the mobile terminal 3. In other words, when an image of a product on a product shelf is captured by the camera 36, that product is displayed on the display unit 31, but if the customer changes the position or orientation of the mobile terminal 3, a different product is captured and that product is displayed on the display unit 31. Hereinafter, an image captured by the mobile terminal 3 is referred to as a "captured image."

The camera 36 also functions as a barcode reader. When the captured image includes a barcode, the aforementioned product information display application operates as a barcode reader, and obtains product identification information such as a JAN code from the barcode captured by the camera 36 and outputs it to the control unit 35.

[Product information display method]
Next, a method for displaying product information will be described.
(Method using captured images)
(1) Display of price superimposed screen In this embodiment, as described above, the product shelves 4 in the store 8 do not have price tags, and customers obtain product information using the mobile terminal 3. FIG. 6 shows a method of obtaining price information using a captured image. A customer operates a product information display application installed on the mobile terminal 3 to start the camera 36, and captures a product of interest with the mobile terminal 3. As shown in the example of FIG. 6, when a customer captures an image of the product shelf 4 within the range indicated by the dashed line 42 with the mobile terminal 3, the mobile terminal 3 displays the captured image within the range of the dashed line 42 as shown in a display screen 61, and transmits the captured image to the image processing server 2. When the image processing server 2 receives the captured image from the mobile terminal 3, it performs image recognition of each product included in the captured image, obtains price information of each product, and transmits it to the mobile terminal 3. The mobile terminal 3 displays the received price information superimposed on the corresponding product. As a result, a screen (called a "price superimposed screen") in which the price 51 of each product is displayed on each product as shown in a display screen 62 is displayed on the mobile terminal 3. In this way, when a customer takes an image of a product of interest with the mobile terminal 3, the price of the product included in the image is displayed superimposed on each product. Therefore, even if there are no price tags on the product shelves 4, the customer can use the mobile terminal 3 to find out the price of the product of interest.

(2) Display of Product Details Screen When a predetermined time has elapsed with the same product displayed on the price overlay screen, the mobile terminal 3 displays a message 52 informing the customer that a product can be selected, as illustrated in FIG. 7(A). When the customer taps on one of the products on the screen to select it, a product details screen for the selected product is displayed. Specifically, when the customer selects a product, the mobile terminal 3 transmits product identification information of the selected product to the image processing server 2. The image processing server 2 acquires product details information of the product corresponding to the received product identification information from the memory unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received product details information, and displays it on the display unit 31.

Figure 7 (B) is an example of a product details screen. This example shows the case where drug B is selected on the price overlay screen. The product details screen 63 includes a package image 63a of the product and detailed information 63b about the product. Note that the package image 63a is not an image captured by the mobile terminal 3, but an image that has been stored in advance as part of the product details in the memory unit 23 of the image processing server 2. In this way, a customer can select a product of interest on the price overlay screen and view detailed information about that product.

In addition, when a customer selects multiple products while message 52 is displayed as shown in FIG. 7(A), the mobile terminal 3 displays a product details screen that comparatively displays detailed product information about the multiple products. Specifically, the mobile terminal 3 transmits product identification information of the multiple products selected by the customer to the image processing server 2. The image processing server 2 acquires detailed product information for the product corresponding to the received product identification information from the memory unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received detailed product information for the multiple products, and displays it on the display unit 31.

Figure 8 shows an example of a product details screen for comparing multiple products. Figure 8 shows a product details screen 64 when medicine B and medicine G are selected as multiple products. In this way, when a customer selects multiple products, a product details screen that displays detailed product information for those products for comparison is displayed on the mobile terminal 3. In this case, it is preferable to set a limit on the number of products that can be displayed for comparison at one time so that the product details screen does not become difficult to view.

When the image captured by the mobile terminal 3 contains only one product, it can be assumed that the product has been selected, and the mobile terminal 3 simply displays a product details screen for that product without displaying a price overlay screen. Specifically, the mobile terminal 3 transmits the captured image to the image processing server 2, and the image processing server 2 performs image recognition of the product contained in the captured image. The image processing server 2 then determines whether the captured image contains only one product, and if the captured image contains only one product, obtains product details information for that product from the memory unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen containing the received product details information and displays it on the display unit 31.

(Method using barcodes)
In this embodiment, in addition to the method of acquiring product information using a captured image as described above, product information can also be acquired using a barcode. FIG. 9 shows a method of acquiring product information using a barcode. A customer operates a product information display application on the mobile terminal 3 to start the camera 36 and captures a barcode 43 printed or attached to a package 41 of a product. When the mobile terminal 3 determines that the captured image includes a barcode, the mobile terminal 3 functions as a barcode reader, reads the barcode, acquires product identification information, and transmits the information to the image processing server 2. Based on the product identification information, the image processing server 2 acquires product details of the corresponding product from the storage unit 23 and transmits the information to the mobile terminal 3. The mobile terminal 3 generates a product details screen 65 including the received product details, and displays it on the display unit 31. Note that the barcode to be read from the product may be a one-dimensional barcode exemplified in FIG. 9, or a two-dimensional barcode such as a QR code (registered trademark). The barcode is an example of code information.

[Display Processing]
Next, the display process by the information processing system 100 will be described. The display process is a process in which a customer uses the mobile terminal 3 in the store 8 to display product information. FIG. 10 is a flowchart of the display process. The display process is realized by the image processing server 2 and the mobile terminal 3 executing programs prepared in advance. It is assumed that a customer has previously registered as a user with the chain headquarters to which the store 8 belongs in order to receive a product display service using the mobile terminal 3 in the store 8, and that the customer's information is registered as customer information in the storage unit 23 of the image processing server 2. It is also assumed that the customer has installed a product information display application for displaying products in the store 8 in the mobile terminal 3.

First, when a customer enters the store 8, the customer launches a product information display app installed on the mobile terminal 3 and reads the store barcode using the mobile terminal 3 (step S101). The store barcode is attached to a predetermined location in the store 8, for example, a wall near the store entrance or on a card used for shopping. When the customer reads the store barcode, the mobile terminal 3 obtains a store ID from the read store barcode and sends it to the image processing server 2 (step S102). When sending any information to the image processing server 2, the mobile terminal 3 sends the customer ID or the terminal ID of the mobile terminal 3 as necessary to identify itself. The image processing server 2 receives the store ID from the mobile terminal 3 and identifies the store (step S103).

In step S101 above, the mobile terminal 3 identifies the store by reading the store barcode, but instead, the store may be identified using GPS information indicating the position of the mobile terminal 3 and the store ID may be transmitted to the image processing server 2. Also, instead of the mobile terminal 3 transmitting store information to the image processing server 2, the image processing server 2 may identify the store in which the customer is located by identifying the store's WiFi via which the information from the mobile terminal 3 is transmitted to the image processing server 2.

Next, the mobile terminal 3 determines whether the product barcode attached to the product has been read using the camera 36 (step S201). If the product barcode has been read (step S201: Yes), the mobile terminal 3 transmits the product identification information of the read product to the image processing server 2 based on the read product barcode (step S202). The image processing server 2 receives the product identification information from the mobile terminal 3 (step S203), acquires product details of the product by referring to the storage unit 23, and transmits it to the mobile terminal 3 (step S204). If the product details of each product are stored in the storage unit 23 for each store, in step S204, the image processing server 2 acquires the store specified in step S103 and the product details associated with the product identification information received in step S203 from the storage unit 23, and transmits it to the mobile terminal 3. Following step S204, the mobile terminal 3 receives the product details information and displays a product details screen on the display unit 31 (step S205). In this way, as described with reference to FIG. 9, the product information is displayed using the barcode, and the process proceeds to step S206. Note that if the product barcode has not been read (step S201: No), the process also proceeds to step S206.

In step S206, the mobile terminal 3 determines whether or not a captured image of the product has been acquired using the camera 36. If a captured image has been acquired (step S206: Yes), the mobile terminal 3 transmits the captured image to the image processing server 2 (step S207). The image processing server 2 receives the captured image from the mobile terminal 3 (step S208) and acquires product identification information of the product included in the captured image using the image feature information in the storage unit 23.

Next, the image processing server 2 refers to the product information corresponding to the store identified in step S103 among the product information in the storage unit 23, acquires the price of each product corresponding to the product identification information acquired in step S208, and transmits it to the mobile terminal 3 (step S209). The mobile terminal 3 receives the price of each product included in the captured image and displays it on the display unit 31 (step S210). In this way, as described with reference to FIG. 6, a price superimposition screen is displayed on the mobile terminal 3 using the captured image of the product. Then, the process proceeds to step S211. If the captured image is not displayed (step S206: No), the process also proceeds to step S211. Note that, if the captured image received in step S208 contains only one product, the image processing server 2 acquires the product identification information of the product based on the captured image, acquires product information corresponding to the store identified in step S103, acquires product details information of the product, and transmits it to the mobile terminal 3. In this case, the mobile terminal 3 displays a product details screen including the product details information of the product.

In step S211, the mobile terminal 3 determines whether or not a product has been selected by the customer on the price superimposition screen. If a product has been selected (step S211: Yes), the mobile terminal 3 transmits information identifying the selected product (hereinafter referred to as "selected product information") to the image processing server 2 (step S212). Note that since the image processing server 2 has already identified each product at the time the price superimposition screen is displayed, the mobile terminal 3 can transmit product identification information of the selected product as selected product information. Note that instead of product identification information, the mobile terminal 3 may use, for example, position information of the selected product on the captured screen as the selected product information.

The image processing server 2 receives the selected product information from the mobile terminal 3 (step S213), refers to the storage unit 23, acquires product details of the selected product, and transmits the product details to the mobile terminal 3 (step S214). If product details of each product for each store are stored in the storage unit 23, in step S214 the image processing server 2 acquires the product details by referring to the product information in the storage unit 23 corresponding to the store identified in step S103. If multiple products are selected, the image processing server 2 transmits the product details of the multiple products to the mobile terminal 3.

Next, the mobile terminal 3 receives the product details information of the selected product and displays the product details screen on the display unit 31 (step S215). In this way, as described with reference to Figures 7 and 8, the product details screen of the product selected by the customer is displayed on the price overlay screen.

Next, the mobile terminal 3 determines whether the product information display app has been terminated by the customer's operation (step S216). If the product information display app has been terminated (step S216: Yes), the process ends. On the other hand, if the product information display app has not been terminated (step S216: No), the process returns to step S201.

As described above, according to the product information display process of this embodiment, customers can obtain price information, detailed product information, and the like by reading a barcode using the mobile terminal 3 or capturing an image of a product. Therefore, even if there is no price tag on the product shelf in the store, customers can easily obtain the information necessary to purchase the product.

[Modification]
Next, modified examples of the present embodiment will be described. The following modified examples can be implemented in appropriate combination.
(1) Processing when a captured image contains a large number of products When a large number of products are captured at one time by the mobile terminal 3, the area of each product on the price superimposition screen becomes small, which may make the price superimposition screen difficult to see. Therefore, when a large number of products are included in the captured image, the mobile terminal 3 displays a message requesting the user to reduce the number of products captured before displaying the price superimposition screen. Fig. 11 (A) shows an example of a message display when a large number of products are included in the captured image. Now, assume that the upper limit of the number of products that can be displayed at one time on the price superimposition screen (hereinafter referred to as the "upper limit number") is six. In the example of Fig. 11 (A), the mobile terminal 3 captures products that exceed the upper limit number. In this case, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products.

In specific processing, the mobile terminal 3 transmits an image captured by the camera 36 to the image processing server 2. The image processing server 2 recognizes products from the captured image, and if the number of recognized products exceeds the upper limit, notifies the mobile terminal 3 of this fact. Upon receiving this notification, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products, as exemplified in FIG. 11 (A).

Instead of using the maximum number of products that can be displayed at one time, it is also possible to determine the maximum size of the area that each product occupies in the captured image. For example, the maximum size of each product may be set to 1/6 of the total area of the captured image. In this case, if the size of each product recognized from the captured image is equal to or larger than the maximum size, the image processing server 2 notifies the mobile terminal 3 of this fact. Upon receiving this notification, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products, as shown in FIG. 11 (A).

In addition, when the number of products included in the captured image is large, instead of requesting the customer to reduce the number of products as described above, the customer may be requested to select one of the products. FIG. 11(B) is an example of displaying a message requesting the customer to select one of the products. In this example, since the number of products included in the captured image exceeds the upper limit of six, a message 57 is displayed requesting the customer to select one of the products included in the captured image. In this case, as a specific process, the mobile terminal 3 transmits the captured image to the image processing server 2. The image processing server 2 recognizes products from the captured image, and if the number of recognized products exceeds the upper limit or if the size of each product exceeds the upper limit, it notifies the mobile terminal 3 of this fact. Upon receiving the notification, the mobile terminal 3 displays a message 57 requesting the customer to select one of the products, as exemplified in FIG. 11(B). When the customer selects one of the products while the message 57 is displayed, the mobile terminal 3 displays a product details screen for the selected product, as exemplified in FIG. 7(B). Note that instead of requesting the customer to select one of the products, the customer may be prompted to approach the product shelf, and the number of products displayed on the mobile terminal 3 may be reduced to or below the upper limit. In this case, the mobile device 3 may display a message such as, "Please move closer to the products to reduce the number of products displayed on the screen to six or less."

(2) Changing Prices for Each Customer In this embodiment, customers who use the mobile terminal 3 to display product information at the store 8 have registered in advance as users with the chain headquarters to which the store belongs. Therefore, the price of products may be changed for each customer. In this case, the storage unit 23 of the image processing server 2 stores different price information for each store and for each customer or for each customer rank. For example, in addition to a standard price list applied to general customers, a special price list for good customers who have a high purchase history at the store may be stored. Also, a member rank may be set for customers based on their purchase history, and a price list for each member rank may be stored. Note that instead of preparing a price list for good customers and each member rank, a discount rate corresponding to good customers and each customer rank may be stored.

In actual processing, when the image processing server 2 receives a customer ID from the customer's mobile terminal 3, it refers to the customer information in the memory unit 23 to determine whether the customer is a general customer or a preferred customer, and the customer's membership rank. The image processing server 2 then refers to the product information corresponding to the store and the customer stored in the memory unit 23, determines the price of each product, and transmits it to the mobile terminal 3.

(3) Display Information on Product Details Screen The product details screens shown in Figs. 7B and 8 are screens that display detailed product information for each product, but sales information may be added to them. For example, as shown in Fig. 12A, a coupon mark 53 may be added to the product details screen to inform the customer of the existence of a coupon that can be used for that product. The coupon that can be applied to each product may be determined according to the customer's purchase history and membership rank. In another example, sales information for a given period of time and information such as the number of units sold may be added to the product details screen. In this case, the image processing server 2 may access a POS server that manages the sales of each product in the entire chain or for each store, and obtain information such as the sales and number of units sold of the product. Information such as the sales and number of units sold of each product may be periodically transmitted from the POS server to the image processing server 2, and the information may be used. In the example of Fig. 12B, recent sales information 54 of the product is displayed.

In addition, some or all of the above embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

(Appendix 1)
An information processing system,
An imaging means for generating an image of an external appearance of a product;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
a display control means for displaying on a display unit a price superimposition screen in which the price is superimposed on the captured image;
An information processing system comprising:

(Appendix 2)
The information processing system described in Appendix 1, wherein the display control means, when a product is selected on the price overlay screen, obtains product details information for the selected product and displays a product details screen including the product details information on the display unit.

(Appendix 3)
The information processing system according to claim 2, wherein the display control means, when multiple products are selected on the price overlay screen, displays a product details screen on the display unit that comparatively displays product detailed information for the selected multiple products.

(Appendix 4)
A reading means for reading code information attached to the product is provided,
The information processing system according to any one of appendices 1 to 3, wherein the display control means obtains product details information for a product corresponding to the read code information, and displays a product details screen including the product details information on the display unit.

(Appendix 5)
The information processing system according to any one of appendices 2 to 4, wherein the product details screen includes coupon information determined for each customer or for each customer rank.

(Appendix 6)
The information processing system according to any one of appendices 2 to 5, wherein the product details screen includes sales information or sales quantity of the product for a specified period.

(Appendix 7)
The information processing system described in any one of Appendix 1 to 6, wherein the display control means displays a message on the display unit requesting the user to reduce the number of products included in the captured image if the size of each product included in the captured image is equal to or larger than a predetermined size, or if the number of products included in the captured image is equal to or larger than a predetermined number.

(Appendix 8)
The information processing system described in any one of Appendices 1 to 6, wherein the display control means displays a message on the display unit requesting the selection of a product included in the captured image when the size of each product included in the captured image is equal to or larger than a predetermined size, or when the number of products included in the captured image is equal to or larger than a predetermined number, and when a product is selected, displays a product details screen on the display unit including product information about the selected product.

(Appendix 9)
A customer identification information acquisition means for acquiring customer identification information;
a price storage unit that stores different prices for each of the customers or for each rank of the customers,
9. The information processing system according to any one of claims 1 to 8, wherein the price acquisition means acquires a price corresponding to the customer from the price memory unit based on the customer's identification information.

(Appendix 10)
A store identification information acquisition means for acquiring store identification information;
A price storage unit that stores different prices for each store,
10. The information processing system according to any one of claims 1 to 9, wherein the price acquisition means acquires a price corresponding to a store from the price memory unit based on identification information of the store.

(Appendix 11)
The commodity recognition means detects the number of commodities included in the captured image,
An information processing system as described in any one of appendices 1 to 10, wherein the display control means displays a product details screen including product information about the product on the display unit when the number of products is one, and displays the price overlay screen on the display unit when the number of products is multiple.

(Appendix 12)
An information processing device capable of communicating with a terminal device,
A receiving means for receiving an image of an external appearance of a product from the terminal device;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
a transmission means for generating a price superimposed screen in which the price is superimposed on the captured image and transmitting the price superimposed screen to the terminal device;
An information processing device comprising:

(Appendix 13)
receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
An information processing method for generating a price superimposed screen in which the price is superimposed on the captured image, and transmitting the price superimposed screen to the terminal device.

(Appendix 14)
receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
A program that causes a computer to execute a process of generating a price superimposition screen in which the price is superimposed on the captured image and transmitting the price superimposition screen to the terminal device.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that a person skilled in the art can understand can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and amendments that a person skilled in the art could make in accordance with the entire disclosure, including the scope of the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated into this document by reference.

Reference Signs List 2 Image processing server 3 Mobile terminal 4 Product shelf 8 Store 25 Control unit 36 Camera 100 Information processing system

Claims (14)

An information processing system,
An imaging means for generating an image of an external appearance of a product;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
a display control means for displaying on a display unit a price superimposed screen in which the price is superimposed on the captured image, and then, when a predetermined time has elapsed with the same product displayed on the price superimposed screen, displaying on the price superimposed screen a character string informing a customer that at least one product can now be selected from among the products included on the price superimposed screen ;
An information processing system comprising: The information processing system according to claim 1, wherein the display control means, when a product is selected on the price overlay screen on which the character string is displayed , obtains product information about the selected product and displays a product information screen including the product information on the display unit. The information processing system according to claim 2, wherein the display control means, when multiple products are selected on the price overlay screen with the character string displayed, displays on the display unit a product details screen that comparatively displays detailed product information for the multiple selected products. A reading means for reading code information attached to the product is provided,
The information processing system according to claim 1 , wherein the display control means acquires product details information of a product corresponding to the read code information, and displays a product details screen including the product details information on the display unit. The information processing system according to any one of claims 2 to 4, wherein the product details screen includes coupon information determined for each customer or for each customer rank. The information processing system according to any one of claims 2 to 5, wherein the product details screen includes sales information or sales quantity of the product for a specified period. The information processing system according to any one of claims 1 to 6, wherein the display control means displays a message on the display unit requesting the user to reduce the number of products included in the captured image when the size of each product included in the captured image is equal to or larger than a predetermined size, or when the number of products included in the captured image is equal to or larger than a predetermined number. The information processing system according to any one of claims 1 to 6, wherein the display control means displays on the display unit a message requesting the selection of a product included in the captured image when the size of each product included in the captured image is equal to or larger than a predetermined size, or when the number of products included in the captured image is equal to or larger than a predetermined number, and when a product is selected, displays on the display unit a product details screen including detailed product information of the selected product. A customer identification information acquisition means for acquiring customer identification information;
a price storage unit that stores different prices for each of the customers or for each rank of the customers,
9. The information processing system according to claim 1, wherein the price acquisition means acquires a price corresponding to the customer from the price storage unit based on the customer's identification information. A store identification information acquisition means for acquiring store identification information;
A price storage unit that stores different prices for each store,
10. The information processing system according to claim 1, wherein the price acquisition means acquires the price corresponding to the store from the price storage unit based on the identification information of the store. The commodity recognition means detects the number of commodities included in the captured image,
The information processing system according to any one of claims 1 to 10, wherein the display control means, when the number of products is one, displays a product details screen including product detailed information about the product on the display unit without displaying the price overlay screen and the string, and when the number of products is multiple, displays the price overlay screen on the display unit. An information processing device capable of communicating with a terminal device,
A receiving means for receiving an image of an external appearance of a product from the terminal device;
a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image;
a price obtaining means for obtaining a price of the recognized item;
a transmitting means for generating a price superimposed screen in which the price is superimposed on the captured image and transmitting the generated price superimposed screen to the terminal device, and then transmitting a display control signal to the terminal device for causing a character string to be displayed on the price superimposed screen, the character string informing a customer that at least one product can now be selected from among the products included on the price superimposed screen, when a predetermined time has elapsed with the same product displayed on the price superimposed screen ;
An information processing device comprising: 1. A computer-implemented information processing method, comprising:
receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
The information processing method generates a price overlay screen in which the price is superimposed on the captured image and transmits it to the terminal device, and then, when a predetermined time has elapsed with the same product displayed on the price overlay screen, transmits a display control signal to the terminal device to cause the price overlay screen to display a string of characters informing the customer that at least one product can now be selected from the products included on the price overlay screen . receiving an image of the exterior of the product from a terminal device;
Performing image recognition on the captured image, and recognizing the product included in the captured image;
Get the price of the recognized item,
A program that causes a computer to execute a process of generating a price overlay screen in which the price is superimposed on the captured image and transmitting it to the terminal device, and then, when a predetermined time has elapsed with the same product displayed on the price overlay screen, transmitting a display control signal to the terminal device to cause the price overlay screen to display a string of characters informing the customer that at least one product can now be selected from the products included on the price overlay screen .