JP7598902B2 - Information processing device, information processing method, and program - Google Patents

Description

Application of Article 30, paragraph 2 of the Patent Act October 14, 2021, October 15, 2021 Seven & i Holdings Co., Ltd. conducted a demonstration test at York Benimaru Katahira store using the information processing device invented by Nagasaka Tomoaki, Yamada Hitoshi, and Miyoshi Toshinori. October 26, 2021 Seven & i Holdings Co., Ltd. conducted a demonstration test at York Benimaru Daishin store using the information processing device invented by Nagasaka Tomoaki, Yamada Hitoshi, and Miyoshi Toshinori. November 12, 2021, November 13, 2021, December 14, 2021, December 15, 2021 Seven & i Holdings Co., Ltd. conducted a demonstration test at York Benimaru Tahiko Nishi store using the information processing device invented by Nagasaka Tomoaki, Yamada Hitoshi, and Miyoshi Toshinori. On July 26, 2022, Seven & i Holdings Co., Ltd. conducted a demonstration test at the York Benimaru Kanaya store using the information processing device invented by Tomoaki Nagasaka, Hitoshi Yamada, and Toshinori Miyoshi.

The present invention relates to an information processing device, an information processing method, and a program.

For example, in stores such as convenience stores and supermarkets, a display registration task is performed to register the sales locations of products. Conventionally, display registration tasks are performed by workers reading products or price tags with a dedicated reading device in a predetermined order, for example starting from the bottom left of the shelf, and the products are registered in the order they were read on a server that receives the information from the reading device.

In conventional work, for example, if a worker scanned products in an order different from the predetermined order, incorrect data would be registered. Also, if a worker forgot to scan even one product during scanning, the registration process had to be started again from the beginning.

JP 2004-30407 A

Patent Document 1 describes a system that issues display area identification information 13 for the area in which products are displayed on display fixtures in a store, and when an employee displays a product, reads the display area identification information 13 attached to the display area in which the product is displayed with a barcode reader, then reads the product identification information attached to the product with the barcode reader, and registers the display by associating the display area with the product.

The technology described in Patent Document 1 allows employees to associate display areas with products in any order, rather than in a predetermined order. However, it requires separate work to issue display area identification information for the display area and to register the issued display area identification information, and when associating the display area with the products, it is time-consuming to read the barcode of the display area identification information 13 and the barcode of the product identification information separately.

Furthermore, when reading the product identification information attached to the products, there was a risk that a product placed in the wrong position would be read, or that a sold-out product would not be read, making it impossible to perform reliable display registration.

The present invention aims to provide an information processing device, information processing method, and program that can perform display registration operations for price tags detected in any order.

An information processing device according to one aspect of the present invention includes a detection unit that detects price tags from information including multiple images, an estimation unit that estimates the display positions of multiple price tags based on the relative positions of the detected multiple price tags, and an output unit that outputs the estimation results.

According to this aspect, the information processing device can perform display registration operations for the detected price tags in any order based on the relative positions of the detected price tags.

In the above information processing device, the estimation unit may include an acquisition unit that acquires position information of the price tags detected by the detection unit, and a display position estimation unit that estimates the display positions of the multiple price tags based on the relative positions determined from the position information of the multiple price tags. According to this aspect, the information processing device can estimate the display positions of the multiple price tags using the position information of the detected price tags, so that the display information can be registered without requiring detailed advance settings regarding the display range and shelves.

In the above information processing device, the display position estimation unit may estimate a display position indicating at least the relative relationship between the multiple price tags in the up, down, left and right directions based on the position information of the multiple price tags. According to this aspect, the information processing device can indicate the relative relationship between the multiple price tags in the up, down, left and right directions as the display position.

In the above information processing device, the display position estimation unit may estimate the shelf level based on the vertical relative distance between multiple price tags and a threshold value, and estimate the left-right display position based on the left-right relative positions between multiple price tags belonging to the estimated shelf level. According to this aspect, the information processing device can estimate the shelf level with high accuracy using a minimum amount of information, using the vertical relative distance between multiple price tags and a threshold value.

The information processing device may further include a spatial recognition unit that performs spatial recognition based on information acquired by a camera that captures an image and/or a sensor that acquires predetermined information, and the acquisition unit may convert the image coordinates of the detected price tag into position information using the spatial recognition information acquired by the spatial recognition unit. According to this aspect, the information processing device can acquire accurate position information of the price tag by utilizing the camera and/or sensor that the information processing device includes.

The information processing device may further include a product identification unit that identifies the product corresponding to the detected price tag, and the output unit may register the estimated display position and information about the product identified by the product identification unit in a display master. According to this aspect, the information processing device can register a display master that associates the display position with information about the product.

The information processing device may further include a shelf identification unit that identifies a shelf to be processed based on shelf identification information and acquires shelf information for the identified shelf, and the output unit may register the display position and product information in association with the identified shelf. According to this aspect, the information processing device can register display information for multiple shelves.

The information processing device may further include an editing unit that edits the display information registered in the display master. According to this aspect, the user of the information processing device can edit the display information as needed.

In the above information processing device, the detection unit may acquire product name information and/or code information contained in the price tag, and the product identification unit may identify the product by comparing the product name information and/or code information with a product master that stores information about the product. According to this aspect, the information processing device can more accurately identify the product by comparing it with the information in the product master.

In the above information processing device, the output unit may display on the display unit a screen showing the display positions of the multiple price tags estimated by the estimation unit. According to this aspect, the user of the information processing device can reliably proceed with the display registration work while checking the processing results of the display registration.

The information processing device may further include a clustering unit that clusters the multiple price tags detected by the detection unit to estimate the display position. According to this aspect, the information processing device can estimate the display position even when the estimation unit does not function well to estimate the display position.

In the above information processing device, the estimation unit may execute a display position estimation process to estimate the display position of a price tag each time the detection unit detects a price tag, and recalculate the display positions of other estimated price tags using the display position estimation process of the detected price tag, and the output unit may output the display position of the detected price tag and the updated display positions of the other estimated price tags that have been recalculated. According to this aspect, the user of the information processing device can reliably proceed with the display registration work while checking the display registration processing results in real time.

In the above information processing device, the output unit may further output the number of faces. According to this aspect, the information processing device can manage the display information including the number of faces.

In the above information processing device, the display position may indicate at least the relative relationship between the multiple price tags in the up, down, left and right directions. According to this aspect, the information processing device can indicate the relative relationship between the multiple price tags in the up, down, left and right directions as the display position.

A method according to another aspect of the present invention includes detecting price tags from information including a plurality of images, estimating display positions of a plurality of price tags based on the relative positions of the detected plurality of price tags, and outputting the estimation result.

A program according to another aspect of the present invention causes one or more computers to detect price tags from information including multiple images, estimate the display positions of multiple price tags based on the relative positions of the detected multiple price tags, and output the estimated results.

The present invention provides an information processing device, information processing method, and program that can perform display registration operations for price tags detected in any order.

1 is a diagram showing a system configuration including an information processing device according to an embodiment of the present invention. 1 is a block diagram of an information processing apparatus according to an embodiment of the present invention; FIG. 13 is a diagram showing price tags placed on a shelf and the number of faces of products corresponding to the price tags. 10 is a flowchart showing a display information registration process of the information processing device according to one embodiment of the present invention. 10 is a flowchart showing a display position estimation process of the information processing device according to one embodiment of the present invention. 13 is a flowchart showing a clustering display position estimation process of the information processing device according to one embodiment of the present invention. FIG. 13 is a diagram showing a price tag placed on a shelf. FIG. 4 is a diagram showing an example of a screen displayed on a display unit.

The following embodiments of the present invention will be described with reference to the attached drawings. Note that the following embodiments are provided to facilitate understanding of the present invention and are not intended to limit the present invention. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention. Furthermore, those skilled in the art will be able to adopt embodiments in which the elements described below are replaced with equivalent elements, and such embodiments are also within the scope of the present invention.

(overview)
An overview of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing a system configuration including an information processing device according to an embodiment of the present invention. The system includes an information processing device 100 and a database 200, which are connected to each other via a network N so as to be able to communicate with each other. For example, a tablet terminal or a smartphone can be used as the information processing device 100.

The information processing device 100 is a device that detects price tags placed on shelves in stores such as convenience stores and supermarkets, and creates display information using the position of the detected price tag as the position of the product. The price tag includes a barcode, product name, etc., and the information processing device 100 identifies the product corresponding to the price tag by comparing the barcode or product name in the detected price tag with the product master 210 in the database 200, and registers the identified product and its display position in the display master 220 in the database 200.

The display position is information that indicates at least the relative relationship between multiple price tags in the up, down, left and right directions, and in this embodiment, includes the shelf level on a shelf and the position from the edge of multiple price tags on the same shelf level.

Note that while FIG. 1 shows the information processing device 100 and the database 200 separately, some or all of the data in the database 200 may be included in the information processing device 100.

(Functional configuration)
Fig. 2 is a block diagram of an information processing device according to an embodiment of the present invention. Although Fig. 2 shows an example of a necessary configuration assuming a single information processing device 100, the information processing device 100 can also be configured as a part of a multi-function distributed system consisting of multiple computer systems. For example, some of the functions of the information processing device 100 may be included in the database 200 or another information processing device capable of communicating with the database 200.

The information processing device 100 includes an operation unit 110, a control unit 120, a memory unit 140, an image capture unit 150, a communication unit 160, a display unit 170, and a sensor unit 180.

The operation unit 110 is configured to accept operations from a user of the information processing device 100, and can be realized by a touch panel, buttons, etc.

The control unit 120 includes an arithmetic processing unit 121 such as a CPU or MPU equivalent to a processor, and a memory 122 such as a RAM. The arithmetic processing unit 121 (processor) implements the functions and processing described below in the arithmetic processing unit 121 by expanding a program recorded in the storage unit 140 into the memory 122 and executing it based on various inputs. This program may be stored in a computer-readable non-transitory recording medium such as a CD-ROM, or distributed via a network and installed on the computer. The memory 122 functions as a work memory required for the arithmetic processing unit 121 (processor) to execute the program.

The memory unit 140 is configured with a storage device such as a hard disk, and records various programs necessary for executing the processes in the control unit 120, data necessary for executing the various programs, etc. In this embodiment, the memory unit 140 has a trained model 141.

The trained model 141 stores a trained model used to detect price tags. In this embodiment, the trained model 141 stores a trained model that receives an image as input data and outputs the area of the price tag element, which is a component of a price tag. A trained model is a model that has been trained in advance using appropriate training data for a machine learning model based on an arbitrary machine learning algorithm. However, a trained model is not limited to a model that does not undergo further training, and additional training can also be performed.

The photographing unit 150 is configured to acquire images and can be realized by a camera.

The communication unit 160 is configured to connect the information processing device 100 to the network N. For example, the communication unit 160 can be realized by a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission path such as a system bus.

The display unit 170 is configured to display various screens as well as images acquired by the image capture unit 150, and can be realized, for example, by a general display device such as a display with a touch panel.

The sensor unit 180 is configured to collect various information related to the surrounding environment of the information processing device 100, convert it into an electrical signal, and output it. For example, the sensor unit 180 can be realized by a LiDAR (Light Detection and Ranging) sensor, an acceleration sensor, a gyroscope, a geomagnetic sensor, etc.

Furthermore, as shown in FIG. 2, the calculation processing unit 121 has, as functional units, a shelf identification unit 123, a space recognition unit 124, a price tag recognition unit 125, a price tag position registration unit 126, a product identification unit 127, a display position estimation unit 128, a face number estimation unit 129, a display information registration unit 130, an output unit 131, a clustering unit 132, and an editing unit 133.

The shelf identification unit 123 identifies the shelf to be processed based on the shelf identification information, and acquires shelf information for the identified shelf. In this embodiment, the shelf identification unit 123 reads the code information attached to the shelf using the photographing unit 150, and acquires shelf information corresponding to the read code information from the shelf master 230 (not shown) in the database 200. The code information may be one-dimensional code information or two-dimensional code information.

In addition, in this embodiment, the shelf identification unit 123 identifies the shelf by reading the code information with the image capture unit 150, but in another embodiment, the shelf may be identified using other information as shelf identification information, such as receiving and reading a beacon transmitted from a beacon device with the sensor unit 180, or reading image features obtained from an image of the shelf with the image capture unit 150.

Shelf information is stored in the shelf master 230. In one embodiment, the shelf master 230 registers the shelf ID, the number of shelf levels, the shelf height, a threshold value, etc. The threshold value is a value used by the display position estimation unit 128 described below when estimating the upper and lower display positions. In this embodiment, the threshold value is set to ±10 cm when two price tags are on the same shelf level, based on the thickness of the shelf board and the height of the price tag. Note that in this embodiment, the threshold value is set when the price tags are on the same shelf level, but in another embodiment, a threshold value may be set when the price tags are not on the same shelf level, i.e., when they are in a hierarchical relationship. Also, in this embodiment, a threshold value is set for the shelf, but in another embodiment, a threshold value may be set for each shelf level.

The spatial recognition unit 124 performs spatial recognition based on information acquired by the image capture unit 150 and the sensor unit 180. In this embodiment, the spatial recognition unit 124 uses a SLAM (Simultaneous Localization and Mapping) algorithm to analyze the image captured by the image capture unit 150 to create an environmental map, and estimates the self-position on the coordinate axes of the environmental map using sensor information acquired by the sensor unit 180. In this embodiment, the spatial recognition unit 124 performs spatial recognition using the SLAM algorithm, but the SLAM algorithm is one example, and in another embodiment, spatial recognition may be performed using any other algorithm. In addition, the sensor unit 180 that can be used varies depending on the model of the information processing device 100, but spatial recognition may be performed according to the available sensor.

The price tag recognition unit 125 detects the price tag from the image and recognizes the information displayed on the price tag. In this embodiment, the price tag recognition unit 125 acquires an image from the photographing unit 150 and detects the price tag in the image. First, the price tag recognition unit 125 inputs the image to the trained model 141 and acquires the areas of the price tag elements that make up the price tag. The trained model 141 outputs the areas of the price tag card, barcode, product name, and price as the price tag elements. Next, the price tag recognition unit 125 performs a recognition process on the areas of each price tag element and acquires information such as the barcode and product name that correspond to the price tag elements. Note that in this embodiment, the price tag recognition unit 125 detects the price tag using the trained model, but the trained model is just one example, and in another embodiment, the price tag may be detected using any other mathematical model.

The price tag position registration unit 126 registers the pose information on the world coordinates of the price tag detected by the price tag recognition unit 125 as an anchor. The world coordinates are the coordinates of the price tag in three-dimensional space, and the pose information includes information on the position and orientation of the price tag. In this embodiment, the price tag position registration unit 126 uses the spatial recognition information obtained by the spatial recognition unit 124 to convert the image coordinates of the price tag detected by the price tag recognition unit 125 into world coordinates, and registers the pose information on the converted world coordinates as an anchor.

The product identification unit 127 compares the information recognized by the price tag recognition unit 125 with the product master 210 in the database 200 to identify the product corresponding to the price tag. In this embodiment, the product identification unit 127 identifies the product corresponding to the price tag from the product master 210 based on the barcode or product name information recognized by the price tag recognition unit 125.

Product information is stored in the product master 210. In one embodiment, the product master 210 stores product IDs, product names, prices, etc.

The display position estimation unit 128 estimates the display position of the price tag recognized by the price tag recognition unit 125. As described above, the display position is information indicating at least the relative relationship between multiple price tags in the up-down and left-right directions, and in this embodiment, includes the number of shelves from the bottom of a shelf (up-down display position) and the number of positions from the end of multiple price tags on the same shelf (left-right display position). The display position estimation unit 128 estimates the relative position of a price tag with respect to other price tags based on the world coordinates of each price tag obtained by the price tag position registration unit 126. In one embodiment, the display position estimation unit 128 takes the world coordinates of the first price tag as the origin, estimates the shelf level (up-down display position) of the second and subsequent price tags based on the relative distance up-down from the origin and a threshold value, and further estimates the left-right display position based on the relative left-right positions of multiple price tags belonging to the estimated shelf level.

The face number estimation unit 129 estimates the number of faces of an item corresponding to a price tag. The number of faces is the number of identical items lined up horizontally on a shelf. In this embodiment, the face number estimation unit 129 estimates the item that is closest upward from the center of gravity of the price tag in the image as the item corresponding to the price tag, and identifies items that are the same as the estimated item from the image and counts the number of faces. In this embodiment, the face number estimation unit 129 counts the number of faces for one row, and excludes items that are overlapping above from the face count. For example, in the example of Figure 3, the number of faces of the item corresponding to price tag A is 2, and item X that is overlapping above is excluded from the count.

The display information registration unit 130 registers display information. In this embodiment, the display information registration unit 130 registers display information including the display position, product information, and number of faces in the display master 220 of the database 200 in association with shelf identification information. The display information registration unit 130 also updates the display master based on the processing results of the display position estimation by the clustering unit 132 described below.

The output unit 131 outputs various screens used in the display registration process to the display unit 170. In this embodiment, the output unit 131 outputs a screen showing the display position for the registered display information to the display unit 170. The output unit 131 may also output the results of the display registration process to the communication unit 160.

The clustering unit 132 clusters the detected price tags and estimates the display positions. In this embodiment, after detecting all price tags on a shelf, the clustering unit 132 performs clustering using the number of shelf levels in the shelf information as the number of clusters, estimates the top and bottom display positions by regarding each cluster as a shelf level, and estimates the left and right display positions based on the relative left and right positions of multiple price tags belonging to the same shelf level.

The editing unit 133 edits the registered display information. In this embodiment, in response to receiving a shelf identification number, the editing unit 133 requests the output unit 131 to output a screen showing the display position for the display information in the display master 220 associated with the received shelf identification number. Thereafter, in response to receiving an editing instruction for any display information, the editing unit 133 updates the display master 220 based on the editing instruction.

(Display information registration process)
A display information registration process of the information processing device according to the embodiment of the present invention will be described in detail with reference to Fig. 4. In this embodiment, it is assumed that various information is stored in the product master 210 and shelf master 230 of the database 200 before performing the display information registration process described in Fig. 4. It is also assumed that a trained model is stored in the trained model 141 of the information processing device 100.

The process shown in FIG. 4 is executed, for example, when a user of the information processing device 100 inputs an instruction to execute the display information registration process via the operation unit 110. For example, the user inputs an instruction to execute the display information registration process by starting a display information registration application, and when the application is started, the photographing unit 150 starts and starts to continuously read images. In this embodiment, display information registration is performed for price tags placed on the shelf shown in FIG. 7.

In step S401, the shelf to be processed is identified. In this embodiment, for example, when text prompting the user to read the shelf code information is displayed on the display unit 170 and the user points the photographing unit 150 at the code information (not shown) attached to the shelf, the shelf identification unit 123 of the information processing device 100 reads the code information attached to the shelf and acquires shelf information corresponding to the read code information from the shelf master 230 in the database 200. Here, it is assumed that the shelf identification unit 123 has acquired a shelf ID of "00A", number of shelf levels of "4", shelf height of "40 cm", and threshold value of "±10 cm". In one embodiment, the shelf identification unit 123 may display on the display unit 170 a message that shelf identification has been completed and the identified shelf information.

In step S402, sensor information is acquired. In this embodiment, the spatial recognition unit 124 of the information processing device 100 acquires the sensor information from the sensor unit 180.

Next, in step S403, spatial recognition is performed. In this embodiment, the spatial recognition unit 124 uses a SLAM (Simultaneous Localization and Mapping) algorithm to analyze the images captured by the imaging unit 150 to create an environmental map, and estimates the self-position on the coordinate axes of the environmental map using sensor information acquired by the sensor unit 180. In this embodiment, spatial recognition is performed using the SLAM algorithm, but in other embodiments, spatial recognition may be performed using any other algorithm.

In step S404, the price tag is detected. In this embodiment, the price tag recognition unit 125 of the information processing device 100 inputs the image acquired from the photographing unit 150 into the trained model 141 to acquire the area of the price tag elements that make up the price tag. As described above, the trained model 141 outputs the areas of the price tag card, barcode, product name, and price as the price tag elements. Here, for example, after identifying a shelf, text prompting the user to detect a price tag is displayed on the display unit 170, causing the user to aim the photographing unit 150 at the price tag 1 in FIG. 7, and the price tag recognition unit 125 acquires the areas of the price tag card, barcode, product name, and price that correspond to the price tag 1 in FIG. 7.

In step S405, the price tag recognition unit 125 determines whether or not a price tag is present in the image. If it is determined that a price tag is not present (S405: NO), the process returns to step S402. On the other hand, if it is determined in step S405 that a price tag is present (S405: YES), the process proceeds to step S406. Here, it is assumed that it has been determined that a price tag is present, and the process proceeds to step S406.

In step S406, the price tag is recognized. In this embodiment, the price tag recognition unit 125 performs a recognition process on the area of each price tag element and obtains information corresponding to the price tag element. Here, it is assumed that the price tag recognition unit 125 obtains barcode information, product name information, and price information corresponding to price tag 1.

In step S407, the detected price tag is registered as an anchor. In this embodiment, the price tag position registration unit 126 of the information processing device 100 registers the pose information in world coordinates of the price tag detected by the price tag recognition unit 125 as an anchor. Specifically, the price tag position registration unit 126 uses the spatial recognition information obtained by the spatial recognition unit 124 to convert the image coordinates of the price tag detected by the price tag recognition unit 125 into world coordinates, and registers the pose information in the converted world coordinates as an anchor.

In step S408, the product is identified. In this embodiment, the product identification unit 127 of the information processing device 100 compares the information recognized by the price tag recognition unit 125 with the product master 210 in the database 200 to identify the product corresponding to the price tag. Here, the product identification unit 127 compares the barcode information recognized by the price tag recognition unit 125 with the product ID in the product master 210 to identify the product 1 corresponding to the price tag.

In step S409, the display position is estimated. The display position estimation process will be described in detail with reference to FIG. 5. In this embodiment, the display position estimation unit 128 of the information processing device 100 estimates the display position of the price tag recognized by the price tag recognition unit 125. In step S501, the display position estimation unit 128 acquires anchor-registered world coordinates for the price tag to be processed.

In step S502, the display position estimation unit 128 determines whether the price tag to be processed is the first price tag. If it is determined that it is the first price tag (S502: YES), the process proceeds to step S503. Here, it is assumed that the price tag to be processed is determined to be the first price tag, and the process proceeds to S503.

In step S503, the origin coordinates are registered. In this embodiment, the display position estimation unit 128 registers the world coordinates acquired in step S501 as the origin coordinates. Next, in step S504, the display position estimation unit 128 determines that the display position of the price tag being processed is the left end of the first row from the bottom (display position: 1-1), and returns to the flow in FIG. 4.

In step S410, the number of faces is estimated. In this embodiment, the face number estimation unit 129 of the information processing device 100 estimates that the product that is closest upward from the center of gravity of the price tag in the image is the product that corresponds to the price tag, and identifies products that are the same as the estimated product from the image and counts the number of faces. Here, it is assumed that the face number estimation unit 129 estimates the number of faces to be 3.

In step S411, the display information is registered. In this embodiment, the display information registration unit 130 of the information processing device 100 registers display information including the display position, product information, and number of faces in the display master 220 of the database 200, in association with the shelf identification information. Here, the display information registration unit 130 has registered display information including the display position "1-1", product information "product 1", and number of faces "3", in association with the shelf ID "00A".

In step S412, the display screen is updated. In this embodiment, the output unit 131 of the information processing device 100 outputs a screen showing the display position for the registered display information to the display unit 170. Here, it is assumed that the screen shown in FIG. 8 (A) is displayed.

In step S413, the information processing device 100 determines whether or not a display registration completion instruction has been received, and if it is determined that a completion instruction has been received (S413: YES), the process proceeds to step S414. On the other hand, if it is determined in step S413 that a completion instruction has not been received (S413: NO), the process returns to step S402. Here, it is assumed that it has been determined that a completion instruction has not been received, and the process has returned to step S402.

The user aims the image capturing unit 150 at price tag 1 in FIG. 7, followed by price tag 2 and then price tag 3, and repeats steps S402 to S413, resulting in an updated display screen being displayed in the order of FIG. 8 (A) → FIG. 8 (B) → FIG. 8 (C) in step S412. The user then aims the image capturing unit 150 at price tag 4, and steps S402 to S408 are executed.

Referring again to FIG. 5, the display position estimation process will be described in detail. In step S501, the display position estimation unit 128 acquires the anchor-registered world coordinates for the price tag being processed. Here, it is assumed that the display position estimation unit 128 has acquired the world coordinates of price tag 4.

In step S502, the display position estimation unit 128 determines whether the price tag to be processed is the first one. Here, it is determined that the price tag to be processed is not the first one (S502: NO), and the process proceeds to S505.

In step S505, the display position estimation unit 128 calculates the relative distance up and down from the origin. Here, it is assumed that the display position estimation unit 128 calculates the relative distance up and down from the world coordinates of price tag 1 to be "minus 43 cm."

In step S506, the display position estimation unit 128 estimates the shelf level (upper or lower display position) based on the relative distance up and down from the origin and a threshold value. The threshold value "±10 cm" obtained in step S401 is used as the threshold value. Here, price tag 4 is estimated to be on the shelf level below price tag 1 based on the relative distance up and down from the origin of "minus 43 cm" and the threshold value for when two price tags are on the same shelf level of "±10 cm."

In step S507, the display position estimation unit 128 estimates the left-right display position based on the relative left-right positions of multiple price tags belonging to the estimated shelf level. Here, since there are no other price tags belonging to the estimated shelf level, the display position estimation unit 128 estimates that the leftmost position is the one (display position: 1-1).

In step S508, the display position estimation unit 128 determines whether or not recalculation of other price tags is necessary. If it is determined that recalculation is not necessary (S508: NO), the process returns to the flow in FIG. 4. Here, it is assumed that it is determined that recalculation is necessary (S508: YES), and the process proceeds to S509.

In step S509, the display position estimation unit 128 recalculates the display positions of the other price tags. Here, the display position estimation unit 128 adds 1 to the top and bottom display positions of price tag 1, price tag 2, and price tag 3, recalculating them to price tag 1 (display position: 2-1), price tag 2 (display position: 2-2), and price tag 3 (display position: 2-3), respectively.

Then, the process returns to the flow in FIG. 4, where in step S410, the number of faces for price tag 4 is estimated, and in step S411, the display information is registered. Here, the display information registration unit 130 registers display information including the display position "1-1", product information "product 4", and number of faces "3" in association with the shelf ID "00A", and updates the recalculated display information for price tag 1, price tag 2, and price tag 3.

Then, in step S412, the screen shown in FIG. 8 (D) is displayed. The user aims the image capturing unit 150 at price tag 5, following price tag 4 in FIG. 7, and steps S402 to S411 are executed, and in step S412, the screen shown in FIG. 8 (E) is displayed. The user aims the image capturing unit 150 at price tag 6, following price tag 5, and steps S402 to S48 are executed. The display position estimation process for price tag 6 will be described in detail again with reference to FIG. 5.

In step S501, the display position estimation unit 128 acquires the anchor-registered world coordinates for the price tag 6 to be processed.

In step S502, the display position estimation unit 128 determines that the price tag to be processed is not the first price tag (S502: NO), and processing proceeds to S505.

In step S505, the display position estimation unit 128 calculates the relative distance above and below price tag 1 from the world coordinates as "minus 85 cm."

In step S506, the display position estimation unit 128 estimates the shelf level (upper or lower display position) based on the relative distance up and down from the origin and a threshold value. Here, price tag 6 is estimated to be on the shelf level below price tag 1 based on the relative distance up and down from the origin of "minus 85 cm" and the threshold value for when two price tags are on the same shelf level of "±10 cm."

Here, if there is a price tag on another shelf in the same vertical direction as the price tag being processed, viewed from the origin, among the price tags whose display positions have already been estimated, the shelf level is estimated based on the relative vertical distance from the price tag on the other shelf level and a threshold value. Here, price tag 6 is estimated to be on the level below price tag 5, based on the relative vertical distance of "42 cm" from the price tag on the other shelf level (price tag 5) and the threshold value of "±10 cm" when two price tags are on the same shelf level.

In step S507, the display position estimation unit 128 estimates the left-right display position based on the relative left-right positions of multiple price tags belonging to the estimated shelf level. Here, since there are no other price tags belonging to the estimated shelf level, the display position estimation unit 128 estimates that the leftmost position is the one (display position: 1-1).

In step S509, the display position estimation unit 128 determines that recalculation is necessary (S508: YES), and the display positions of the other price tags are recalculated in S509. Here, the display position estimation unit 128 adds 1 to the top and bottom display positions of price tag 1, price tag 2, price tag 3, price tag 4, and price tag 5, and recalculates them to price tag 1 (display position: 3-1), price tag 2 (display position: 3-2), price tag 3 (display position: 3-3), price tag 5 (2-1), and price tag 4 (2-2), respectively.

Returning to the flow of FIG. 4, steps S410 to S413 are executed, and the user aims the image capture unit 150 at price tags 7 to 10, and steps S402 to S413 are repeated to proceed with the display information registration process. In step S412 for price tag 10, after the screen shown in FIG. 8 (F) is displayed, it is assumed that the user inputs an instruction to complete display registration via the operation unit 110, for example.

In step S413, it is determined that a completion instruction has been received (S413: YES), and the process proceeds to step S414. In step S414, display position estimation is performed by clustering. The display position estimation process by clustering will be described in detail with reference to FIG. 6. In this embodiment, the clustering unit 132 of the information processing device 100 clusters the detected price tags to estimate the display position.

In step S601, the clustering unit 132 acquires the world coordinates of all detected price tags. Here, it is assumed that the clustering unit has acquired the world coordinates of a total of 10 price tags, price tag 1 to price tag 10.

In step S602, the clustering unit 132 acquires the known number of shelves. In this example, it is assumed that the clustering unit 132 has acquired the number of shelves "4."

In step S603, the clustering unit 132 performs clustering on the world coordinates of the price tags, with the number of shelves being the number of clusters. Here, it is assumed that 10 world coordinates are classified into 4 clusters.

In step S604, the clustering unit 132 estimates the upper and lower display positions by regarding each cluster as a shelf level. Here, the clustering unit 132 estimates that cluster 1 (price tag 10, price tag 9) is the first level, cluster 2 (price tag 6, price tag 7, price tag 8) is the second level, cluster 3 (price tag 5, price tag 4) is the third level, and cluster 4 (price tag 1, price tag 2, price tag 3) is the fourth level.

In step S605, the clustering unit 132 estimates the left and right display positions based on the relative left and right positions of multiple price tags belonging to the same shelf level, and the process returns to the flow in FIG. 4.

In step S415, the display master is updated. In this embodiment, the display information registration unit 130 updates the display master 220 based on the display position estimation processing results of the clustering unit 132.

In this embodiment, an example has been described in which the shelf identification unit 123 identifies a shelf by reading code information attached to the shelf, but in another embodiment, the user of the information processing device 100 may be prompted to input a shelf identification number, and the shelf may be identified from the input shelf identification number.

In this embodiment, the process of estimating the upper and lower display positions is illustrated in FIG. 5, but this is merely an example, and the information processing device 100 can estimate the upper and lower display positions in any other steps using the world coordinates of the price tag and a threshold value. For example, a predetermined threshold value may be used instead of the threshold value associated with the shelf, or other shelf information (shelf height) may be used in addition to the world coordinates and threshold value.

In addition, in this embodiment, an example has been described in which the face number estimation unit 129 estimates the number of faces, but in another embodiment, the user of the information processing device 100 may be asked to input the number of faces.

In addition, in this embodiment, an example has been described in which the clustering unit 132 estimates the display position after display registration is completed, but the display position estimation process by the clustering unit 132 is an optional process, and this process may be omitted or may be performed under specified conditions, such as when a threshold value is not set or when the display position estimation accuracy by the display position estimation unit 128 is poor.

(Display information editing process)
Finally, a display information editing process of the information processing device according to the embodiment of the present invention will be described. In this embodiment, it is assumed that various information is stored in the display master 220 and the shelf master 230 of the database 200 before the display information editing process is performed. Note that the display information editing process is executed, for example, when a user of the information processing device 100 inputs an instruction to execute the display information editing process via the operation unit 110.

In S1, the editing unit 133 of the information processing device 100 receives a shelf identification number. In this example, it is assumed that the editing unit 133 has received shelf ID "00A." In S2, the editing unit 133 requests the output unit 131 to output a screen showing the display position for the display information in the display master 220 associated with the received shelf identification number.

In S3, the output unit 131 outputs a screen showing the display position based on a request from the editing unit 133. Here, it is assumed that the screen shown in FIG. 8 (F) is displayed. In S4, in response to receiving an editing instruction for any display information, the editing unit 133 updates the display master 220 based on the editing instruction.

In this embodiment, an example has been described in which a user performs display information editing processing using the information processing device 100, but in another embodiment, the display information editing processing may be performed using another information processing device (not shown) that can be connected to the database 200.

As described above, according to this embodiment, the information processing device 100 can perform display registration operations for price tags detected in any order. Display information can be registered simply by pointing the photographing unit 150 at the price tags in any order, so accurate information can be registered while reducing the workload of display registration operations.

In addition, in this embodiment, the information on the detected price tag is immediately displayed on the display unit 170, so the user can proceed with the display registration process reliably while checking the display registration processing results.

Furthermore, in this embodiment, the display position up, down, left, and right can be estimated using the world coordinates of the detected price tag and a threshold value, so display information can be registered without requiring detailed advance settings regarding the display range and shelves.

100...information processing device, 110...operation unit, 120...control unit, 121...arithmetic processing unit, 122...memory, 123...shelf identification unit, 124...spatial recognition unit, 125...price tag recognition unit (detection unit), 126...price tag position registration unit (acquisition unit, estimation unit), 127...product identification unit, 128...display position estimation unit (estimation unit), 129...face number estimation unit, 130...display information registration unit (output unit), 131...output unit, 132...clustering unit, 133...editing unit, 140...storage unit, 141...trained model, 150...photography unit, 160...communication unit, 170...display unit, 180...sensor unit, 200...database, 210...product master, 220...display master, 230...shelf master, N...network

Claims (10)

An information processing device that registers display positions of a plurality of price tags by using a plurality of images acquired by pointing a camera at each of a plurality of price tags placed on a shelf in an arbitrary order in correspondence with products displayed on the shelf, the information processing device comprising:
a detection unit that detects a price tag from the image captured by the camera;
an estimation unit that executes a display position estimation process to estimate the display position of the price tag each time the detection unit detects a price tag, and recalculates the display positions of other estimated price tags using the display position estimation process of the detected price tag, wherein the display position estimation process acquires position coordinates of the detected price tag using spatial recognition information based on information acquired by the camera and a sensor that acquires predetermined information, estimates a shelf level based on the position coordinates of the other estimated price tags, and estimates left-right display positions based on the left-right relative positions between multiple price tags belonging to the estimated shelf level;
an output unit that outputs the detected display position of the price tag and the recalculated updated display positions of the other estimated price tags;
An information processing device comprising: A product identification unit that identifies a product corresponding to the detected price tag is further provided.
The output unit registers the estimated display position and information on the product identified by the product identification unit in a display master.
The information processing device according to claim 1 . A shelf identification unit is further provided which identifies a shelf to be processed based on the shelf identification information and acquires shelf information of the identified shelf,
The output unit registers the display position and information on the product in association with the specified shelf.
The information processing device according to claim 2 . The information processing device according to claim 3, further comprising an editing unit that edits the display information registered in the display master. The detection unit acquires product name information and/or code information included in the price tag,
the product identification unit identifies the product by comparing the product name information and/or code information with a product master that stores information related to the product;
The information processing device according to claim 3 . The information processing device according to claim 1, wherein the output unit includes displaying a screen on a display unit that shows the display positions of the price tags estimated by the estimation unit. The information processing device according to claim 1, further comprising a clustering unit that clusters the multiple price tags detected by the detection unit and estimates the display position. A face number estimation unit is further provided for estimating a product corresponding to the detected price tag based on the position of the detected price tag in the image acquired by the camera, and identifying a product identical to the estimated product from the image and counting the number of faces,
The information processing device according to claim 1 , wherein the output unit further outputs the face number. A method executed by an information processing device for registering display positions of a plurality of price tags by using a plurality of images acquired by pointing a camera at each of a plurality of price tags placed on a shelf in an arbitrary order in correspondence with products displayed on the shelf, the method comprising:
detecting a price tag from the image captured by the camera;
Each time a price tag is detected, a display position estimation process is executed to estimate the display position of the price tag, and the display positions of other estimated price tags are recalculated by the display position estimation process of the detected price tag, wherein the display position estimation process acquires position coordinates of the detected price tag using spatial recognition information based on information acquired by the camera and a sensor that acquires specified information, estimates a shelf level based on the position coordinates of the other estimated price tags, and estimates left and right display positions based on the left and right relative positions between multiple price tags belonging to the estimated shelf level;
and outputting the detected display position of the price tag and the recalculated updated display positions for the other estimated price tags. one or more computers that register display positions of a plurality of price tags by using a plurality of images acquired by pointing a camera at each of a plurality of price tags installed on a shelf in an arbitrary order in correspondence with products displayed on the shelf;
detecting a price tag from the image captured by the camera;
Each time a price tag is detected, a display position estimation process is executed to estimate the display position of the price tag, and the display positions of other estimated price tags are recalculated by the display position estimation process of the detected price tag, wherein the display position estimation process acquires position coordinates of the detected price tag using spatial recognition information based on information acquired by the camera and a sensor that acquires specified information, estimates a shelf level based on the position coordinates of the other estimated price tags, and estimates left and right display positions based on the left and right relative positions between multiple price tags belonging to the estimated shelf level;
and outputting the detected display position of the price tag and the recalculated updated display positions for the other estimated price tags.

Images