JP7646302B2 - Management devices and shelves - Google Patents

Description

The present invention relates to a product shelf on which products are displayed and a management device that helps manage the products displayed on the product shelf.

In recent years, unmanned stores, where customers pay at an unmanned POS themselves or where there is no POS at all, have become more common (see, for example, Non-Patent Document 1). Instead of reducing manpower, unmanned stores have implemented fraud prevention and automated payment systems by placing a wide variety of sensors in the store.

Sensors installed in unmanned stores include, for example, cameras (see, for example, Patent Document 1) and weight sensors (see, for example, Patent Document 2).

Special table 2019-523198 publication JP 2019-034819 A

The top part of Figure 1 shows a conventional product display management device, in which each product is provided with its own weight sensor. However, this type of system limits the freedom of product placement, making it difficult for convenience stores and other stores to adopt.

The present invention was made in consideration of these problems, and its purpose is to provide technology that can automate the detection of increases and decreases in product numbers while ensuring the freedom and mobility of product placement on product shelves.

One aspect of the present invention relates to a management device for detecting an increase or decrease in the number of multiple products placed on a shelf of a product shelf, the management device comprising: a first acquisition means for acquiring a change in output from multiple weight sensors provided at different positions so as to support the shelf, a second acquisition means for acquiring information on a subject from a camera that includes at least a part of the shelf within its angle of view, and a detection means for detecting how much of each product has increased or decreased based on the acquired changes in output from the multiple weight sensors and the acquired information on the subject, and when the position of the center of gravity of the increased or decreased product determined from the acquired changes in output from the multiple weight sensors is within a predetermined distance from a predetermined position where no products are displayed, the detection means detects how much of each product has increased or decreased using a position on the shelf related to a predetermined action of a person identified from the acquired information on the subject.

The "camera" may be an RGB camera, a ToF (Time of Flight) camera, or may include these two types of cameras, or may be a camera in which these two types of cameras are integrated into one camera. The ToF camera is an infrared camera that can acquire depth information in a planar manner.
"Object information" includes an image if it is acquired from an RGB camera, includes depth information if it is acquired from a ToF camera, and includes both an image and depth information if it is acquired from two types of cameras. Thus, object information includes an image and/or depth information.

In addition, any combination of the above components, or mutual substitution of the components or expressions of the present invention between devices, methods, systems, computer programs, and recording media storing computer programs, are also valid aspects of the present invention.

The present invention makes it possible to automate the detection of increases and decreases in product numbers while ensuring the freedom and mobility of product placement on product shelves.

FIG. 2 is a schematic diagram showing an example of a product display mode realized by the management device according to the first embodiment. 1 is a schematic diagram showing a product shelf managed by a management device according to a first embodiment; FIG. 3 is a side view of a portion including a right weight sensor in FIG. 2 . 1 is a schematic diagram showing an upper shelf, a left weight sensor, a right weight sensor, and a product placed on the upper shelf. FIG. 13 is a schematic diagram showing a state in which three products placed on the upper shelf are picked up. FIG. 11 is a schematic diagram showing a state in which a first product and a second product placed on an upper shelf are picked up. FIG. FIG. 2 is a hardware configuration diagram of a management device according to an embodiment. FIG. 8 is a block diagram showing the functions and configuration of the management device of FIG. 7. 9 is a data structure diagram showing an example of a customer information storage unit in FIG. 8 . 9 is a data structure diagram showing an example of an inventory management holding unit in FIG. 8 . 9 is a data structure diagram illustrating an example of a sensor information storage unit in FIG. 8 . 10 is a flowchart showing the flow of a series of processes in the product placement registration unit of FIG. 8 . 9 is a flowchart showing a series of processing steps in the management device of FIG. 8 . FIG. 11 is a block diagram showing the functions and configuration of a management device according to a second embodiment. 15 is a flowchart showing a series of processing steps in the management device of FIG. 14 . FIG. 13 is a side view of a product shelf according to a third embodiment. FIG. 13 is a side view of a product shelf according to a modified example of the third embodiment. 13 is a schematic diagram showing a distance measuring sensor attached to the upper surface of a shelf board. FIG. 13 is a schematic diagram showing a plurality of distance measuring sensors attached to the sides of a shelf board. FIG. FIG. 13 is a data structure diagram illustrating an example of an inventory management holding unit according to a modified example. 1 is a schematic diagram showing an upper shelf, a left weight sensor, a right weight sensor, and a product placed on the upper shelf. FIG. FIG. 22 is a schematic diagram showing the state in which the same number of products as in FIG. 21 are picked up from a different position. 1 is a schematic diagram showing an upper shelf, a left weight sensor, a right weight sensor, and a product placed on the upper shelf. FIG. FIG. 24 is a schematic diagram showing the state in which the same number of products as in FIG. 23 are picked up from a different position.

In the following, identical or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicate explanations will be omitted where appropriate. In addition, some parts that are not important for the explanation will be omitted in each drawing.

(First embodiment)
Fig. 1 is a schematic diagram showing an example of a product display mode realized by a management device according to a first embodiment. In the upper part of Fig. 1 showing the conventional technology, dedicated containers 4a, 4b, 4c, and 4d are placed on a product shelf 2 for each product. The container 4a includes a weight sensor 6a that measures the weight of the product placed in the container 4a, and a separator 8a. The other containers have a similar configuration. In this way, in the product display of the conventional technology, a weight sensor is prepared for each type of product.

In the conventional product display shown in Figure 1, registering product locations is cumbersome. It is also difficult to flexibly rearrange products according to stock levels. In particular, with the conventional technology, as shown in Figure 1, when stock levels decrease, products are displayed in a "toothless" manner. This looks bad and tends to be disliked by stores.

In view of these problems with the conventional technology, the management device according to this embodiment realizes automation of product registration and realizes flexibility in product display as shown in the lower part of Figure 1. In this embodiment, by automating product registration and enabling rearrangement, it becomes possible to move products to the center in a face-up state even when inventory is running low. This makes it possible to maintain a good appearance on the product shelves.

Furthermore, in this embodiment, while allowing for flexible product rearrangement as described above, weight sensors, cameras, and distance sensors are used to detect how many of each product a customer has picked up from the shelves. This allows for labor-saving, unmanned, or automated checkout in stores, and/or labor-saving, unmanned, or automated inventory management.

Figure 2 is a schematic diagram showing a product shelf 10 managed by a management device according to the first embodiment. The product shelf 10 comprises an upper shelf 12 and a lower shelf 14. Multiple types of products are placed on the upper shelf 12. The upper shelf 12 is provided with separators 16a, 16b, and 16c for separating the products by type. The lower shelf 14 is configured in the same way as the upper shelf 12.

A store in which the product shelves 10 are installed is provided with a surveillance camera 24 that captures within its viewing angle the entire upper shelf 12 and the front of the lower shelf 14. The surveillance camera 24 is positioned above the front of the product shelf 10 so as to overlook the front of the product shelf 10. A number of price tags 18a, 18b, 18c corresponding to a number of types of products are attached to the front (front face) of the upper shelf 12, which fits within the viewing angle of the surveillance camera 24, and a number of price tags 18d, 18e corresponding to a number of types of products are attached to the front of the lower shelf 14, which fits within the viewing angle of the surveillance camera 24.

The price tag 18a includes a one-dimensional code 20a that encodes a product ID that identifies the corresponding product ("bread stew" in the example of FIG. 1). The one-dimensional code 20a is attached in a direction that allows it to be viewed by the surveillance camera 24. For example, the one-dimensional code 20a is provided on a surface different from the display surface on which the product name and price are printed. In particular, the one-dimensional code 20a is provided on the upper surface when the display surface is the front. This makes the display surface easy to see for customers, and the one-dimensional code 20a easy to capture on the surveillance camera 24. Note that a QR code, barcode, color code, stealth code, or blinking pattern may be used instead of the one-dimensional code 20a, and more generally, the price tag only needs to have code information that encodes information that identifies the corresponding product in a visible manner or in a manner that can be recognized by a camera.

The price tag 18a and the corresponding separator 16a are arranged to have a predetermined positional relationship. For example, in the embodiment shown in FIG. 2, the left end of the price tag 18a is aligned with the separator 16a. This positional relationship makes it possible to estimate the position of the separator 16a from the position of the price tag 18a. The other price tags have the same configuration as the price tag 18a.

The upper shelf 12 is further provided with a pair of weight sensors (left weight sensor 22a, right weight sensor 22b) that measure the total weight of the upper shelf 12 itself and the items placed on it. The lower shelf 14 is also provided with a similar pair of weight sensors 22c, 22d.

Figure 3 is a side view of a portion including the right weight sensor 22b in Figure 2. A shelf support member 10b, such as a shelf support bracket, is fixed to the support post 10a of the product shelf 10, on which the right weight sensor 22b is provided, and on which the upper shelf board 12 is placed. In other words, the right weight sensor 22b and the shelf support member 10b are arranged and/or configured to support the upper shelf board 12 together. The left weight sensor 22a is configured in the same manner as the right weight sensor 22b.

The left weight sensor 22a and the right weight sensor 22b may be any type of sensor capable of detecting changes in weight. The configuration of such weight sensors is well known and will not be described in detail in this specification.

The sum of the weight measurement by the left weight sensor 22a and the weight measurement by the right weight sensor 22b is (total weight of products placed on the upper shelf 12 + total weight of separators placed on the upper shelf 12 + weight of the upper shelf 12 itself). When separators are not being attached or detached, the amount of change in the total is the amount of change in the weight of products placed on the upper shelf 12.

In the following description, it is assumed that the weight of the separator can be ignored, and that the weight of the shelf itself is offset by zero-point adjustment in each weight sensor. However, those skilled in the art who have read this specification will understand that the technical ideas related to the embodiments described in this specification can be applied even when the weight of the separator cannot be ignored or when the weight of the shelf itself is taken into consideration.

As described above, the sum of the measurement value of the left weight sensor 22a (hereafter referred to as ML) and the measurement value of the right weight sensor 22b (hereafter referred to as MR) is the total weight of the products placed on the upper shelf 12, and does not depend on how the products are arranged on the upper shelf 12. However, the values of ML and MR differ depending on how the products are arranged on the upper shelf 12. The same is true for the difference; the difference between ML and MR indicates the weight of the product that has increased or decreased, but does not indicate where it was placed or where it was placed. However, the position of the product that has increased or decreased can be identified by comparing the amount of change in ML (hereafter referred to as ΔML) with the amount of change in MR (hereafter referred to as ΔMR).

4 is a schematic diagram showing the upper shelf 12, the left weight sensor 22a, the right weight sensor 22b, and a product 30 placed on the upper shelf 12. The distance from the point where the left weight sensor 22a and the upper shelf 12 contact each other to the center of gravity G of all the products placed on the upper shelf 12 (in this case, the position of the product 30) is L, and the distance from the center of gravity G to the point where the right weight sensor 22b and the upper shelf 12 contact each other is R. From the balance of the moment around the center of gravity G,
ML x L = MR x R
Therefore,
M L:M R = R:L
holds true.

When the product 30 is picked up from the state shown in FIG. 4, the amount of change in ML (ΔML) and the amount of change in MR (ΔMR) are as follows:
ΔML:ΔMR=R:L
holds true.

Similarly, when no product is placed on the upper shelf 12 and a product 30 is newly placed thereon, resulting in the state shown in FIG. 4, ΔML and ΔMR are also given as ΔML:ΔMR=R:L.
holds true.

According to the principle of superposition of moments, a state in which multiple products are placed in an arbitrary arrangement on the upper shelf 12 is equivalent to a state in which an object having the total weight of the multiple products is placed at the center of gravity G of the multiple products. Therefore, as in the above, for the position (L, R) of the center of gravity G of such multiple products,
M L:M R = R:L
In the case where such a plurality of products are picked up substantially simultaneously, ΔML and ΔMR are as follows:
ΔML:ΔMR=R:L
Similarly, when multiple products are newly placed on the upper shelf 12 from a state where no products were placed on the upper shelf 12, ΔML and ΔMR are also expressed as ΔML:ΔMR=R:L.
holds true.

In Figure 4, we considered the case where no products other than the product being increased or decreased are placed on the upper shelf 12, but the relationship between ΔML and ΔMR does not change even if other products are placed on the upper shelf 12. According to the principle of moment superposition, the presence of other products does not affect ΔML and ΔMR that accompany the increase or decrease of the product being increased or decreased, so even when other products are present, the inverse ratio of the ratio between ΔML and ΔMR indicates the position of the center of gravity G of the product that has increased or decreased.

In this way, the inverse ratio of the ratio of the amount of change in ML (ΔML) to the amount of change in MR (ΔMR) indicates the position of the center of gravity G of the increased or decreased product. If the arrangement of the products on the upper shelf 12 (information indicating which product is in which position) is known, it is possible to determine which products have increased or decreased (e.g., picked up or newly placed) from the center of gravity position determined by the above inverse ratio. Since the total weight of the increased or decreased products is separately known from ΔML + ΔMR and the weight per product is also known, if the total weight of the increased or decreased products is N times (N is a natural number) the weight per product in the position determined by the inverse ratio, it is possible to determine that the number of products in the position determined by the inverse ratio has increased or decreased by N. If not, the image of the surveillance camera 24 and/or the output of the distance sensor described below is referenced to determine a combination of increased or decreased products such that the center of gravity is located at the position determined by the inverse ratio.

Figure 5 is a schematic diagram showing the state when three products 30 placed on the upper shelf 12 are picked up. The products 30 depicted in solid line boxes are the products 30 to be picked up, and the other products are shown in dashed line boxes. From the inverse ratio of ΔML and ΔMR, the positions of the center of gravity G of the three picked up products 30 are identified as shown in Figure 5. For convenience of explanation, the point where the left weight sensor 22a and the upper shelf 12 contact is set to z = 0, and the z axis is defined to extend from that point to the point where the right weight sensor 22b and the upper shelf 12 contact. From the known arrangement of the products 30, it can be seen that several products 30 were placed at z = z1, z2, z3, and z4 before picking up. In this known arrangement of products 30, no products 30 were placed at the position of the center of gravity G determined from the inverse ratio, so a combination of products 30 that would give such a position of the center of gravity G is found from among the products 30 placed at z = z1, z2, z3, z4.

First, the location of the picked-up product is narrowed down using images from the surveillance camera 24 or other means such as a distance sensor. For example, if a customer's arms are detected at z=z1 and z=z3 almost simultaneously in the state shown in FIG. 5, it is clear that products 30 placed at z=z1 and z3 have been picked up, although the number of products is unknown. Next, the picked-up weight of the product 30 placed at z=z1 and the picked-up weight of the product 30 placed at z=z3 are estimated from the inverse ratio of the distance D1 between z=z1 and the position of the center of gravity G and the distance D2 between z=z3 and the position of the center of gravity G.

For example, if D1:D2 = 1:2, then (number of items picked up at z=z1): (number of items picked up at z=z3) = 2:1. If the number of items 30 picked up is found to be 3 based on the change in total weight, then it can be seen that two items 30 were picked up at z=z1 and one item 30 was picked up at z=z3.

In the example of FIG. 5, the weights of all the products 30 are the same, but even if the weights are different, the picked-up products can be identified by the same calculation. FIG. 6 is a schematic diagram showing the state when the first product 32 and the second product 34 placed on the upper shelf 12 are picked up. The weight of the first product 32 is 2m, and the weight of the second product 34 is m. The position of the center of gravity G is identified from the inverse ratio of ΔML and ΔMR. When the customer's arms are detected at z=z1 and z=z3 almost simultaneously by the surveillance camera 24 or the distance measuring sensor, it is known that the first product 32 and the second product 34 placed at z=z1 and z3, respectively, have been picked up, although the number of products is unknown. Next, the pickup weight of the product 30 placed at z=z1 and the pickup weight of the product 30 placed at z=z3 are estimated from the inverse ratio of the ratio of the distance D1 between z=z1 and the position of the center of gravity G and the distance D2 between z=z3 and the position of the center of gravity G. For example, if D1:D2 = 1:2, then (pickup weight at z=z1):(pickup weight at z=z3) = 2:1. If the change in total weight is 3m, then the pick-up weight at z=z1 is 2m, and the pick-up weight at z=z3 is m. Since the weight of each of the first products 32 at z=z1 is 2m, the number of products picked up is 1, and since the weight of each of the second products 34 at z=z3 is m, the number of products picked up is 1.

In the examples of Figures 5 and 6, it is assumed that the position of the product can be expressed as z = z1, etc. However, in reality, there are limitations to identifying the position, and for example, in the example of Figure 2, it is assumed that the product is located at the price tag (it is located between the separators with an error of a few centimeters from the price tag). Under this assumption, the position of the product is the position of the price tag. Even if this assumption is made, it will be clear to those skilled in the art that the discussion of Figures 5 and 6 is valid. Alternatively, calculations can be made using probability, taking into account that products have a size.

Returning to FIG. 2, the distance measurement sensor array 28 is attached to the front side of the underside of the upper shelf 12, and has a detection range downward. The distance measurement sensor array 28 has multiple distance measurement sensors 28a arranged at equal intervals in a direction parallel to the longitudinal direction of the upper shelf 12. The separators of the lower shelf 14 are positioned between adjacent distance measurement sensors 28a to avoid interference with the distance measurement sensors 28a.

The detection range of each distance measurement sensor 28a overlaps with the front of the lower shelf 14. In particular, each distance measurement sensor 28a is positioned and configured so that it can detect an arm if it enters the space below it between the upper shelf 12 and the lower shelf 14. The management device retains the position of each distance measurement sensor 28a. The management device identifies the position of the distance measurement sensor 28a that detected the object as the position of the lower shelf 14 where the customer extended their arm.

The distance sensor 28a may be a known optical distance sensor in which the voltage changes depending on the distance. In this embodiment, the distance sensor 28a is used as an object detection sensor for detecting objects, but in other embodiments (described later), distance information obtained from the distance sensor is also used.

The management device is connected wirelessly or by wire to the surveillance camera 24, the left weight sensor 22a, the right weight sensor 22b, and the distance sensor array 28. Because the management device knows the positional relationship between the separator and the price tag, it can dynamically register the position of the product by detecting the position of the price tag from the image from the surveillance camera 24.

The management device detects the arm 26a of the customer 26 with the surveillance camera 24, and detects the pickup of a product by combining the detection result with the change in the output of the left weight sensor 22a and the right weight sensor 22b. The detection of the arm 26a is based on a known pose estimation technique or a method using depth information. More specifically, the method of detecting the position of the arm 26a includes (1) a method of detecting that an object (arm) has approached a position of a specific shelf from the outside using depth information from a ToF camera, (2) a method of using pose estimation using an RGB camera, and (3) a method of using these two methods in combination. The example of FIG. 2 shows a surveillance camera 24 (often set on the ceiling) installed in the store, but in addition to this, for example, another camera may be installed on the top of the product shelf 10 (described later in other embodiments). Each of the surveillance camera 24 and the other camera may be a camera in which an RGB camera and a ToF camera are integrated, or may have only one of either type. The position of arm 26a may be detected using images and/or depth information from surveillance camera 24 and images and/or depth information from another camera.

The following describes a case where the surveillance camera 24 is an RGB camera and images acquired from the surveillance camera 24 are used. However, it will be understood by those skilled in the art who have read this specification that in other embodiments, only depth information acquired from a ToF camera may be used, or image and depth information acquired from an integrated RGB/ToF camera may be used.

Figure 7 is a hardware configuration diagram of a management device 100 according to an embodiment. The management device 100 includes a memory 104, a processor 106, a communication interface 108, a display 102, and an input interface 110. Each of these elements is connected to a bus 112 and communicates with each other via the bus 112.

The memory 104 is a storage area for storing data and programs. The data and programs may be stored in the memory 104 permanently or temporarily. The processor 106 executes the programs stored in the memory 104 to realize various functions in the management device 100. The communication interface 108 is an interface for transmitting and receiving data to and from the outside of the management device 100. For example, the communication interface 108 includes interfaces for communicating with the surveillance camera 24, the left weight sensor 22a, the right weight sensor 22b, and the distance sensor array 28. The display 102 is a device for displaying various information, such as a liquid crystal display or an organic EL (Electroluminescence) display. The input interface 110 is a device for receiving input from a user. The input interface 110 includes, for example, a touch panel provided on the display 102, various input keys, and the like.

Figure 8 is a block diagram showing the functions and configuration of the management device 100 of Figure 7. Each block shown here can be realized in hardware terms by elements and mechanical devices such as a computer CPU, and in software terms by computer programs, etc., but here we show functional blocks realized by the cooperation of these. Therefore, those skilled in the art who have read this specification will understand that these functional blocks can be realized in various ways by combining hardware and software.

The management device 100 detects an increase or decrease in the number of products placed on the shelves of the product shelf 10. The management device 100 includes an image acquisition unit 122, a product placement registration unit 124, a weight change acquisition unit 120, an increase or decrease detection unit 128, a position identification unit 126, a checkout processing unit 130, an inventory management unit 136, a customer information storage unit 132, an inventory management storage unit 134, and a sensor information storage unit 138.

Figure 9 is a data structure diagram showing an example of the customer information storage unit 132 of Figure 8. The customer information storage unit 132 stores information on customers in the store. The customer information storage unit 132 stores a customer ID that identifies the customer, coordinates within the store that indicate the customer's current location, and information on unpaid products currently held by the customer, in association with each other. The customer's coordinates are determined by analyzing images obtained from the surveillance camera 24 or other surveillance cameras.

Figure 10 is a data structure diagram showing an example of the inventory management storage unit 134 in Figure 8. The inventory management storage unit 134 stores information on various products displayed on the product shelf 10, their inventory, and their arrangement. The inventory management storage unit 134 stores a product ID that identifies the product, the product name, the weight per unit of the product, the shelf board ID that identifies the shelf board on which the product is placed, the position of the separator that separates the products, the range of the products on the shelf board, and the number of remaining products, in association with each other. The position of the separator is represented by the z coordinate that is defined along the length of the shelf board with the left edge of the shelf board as the origin (see Figure 2). The range of the products is specified by the range of the z coordinate. The number of remaining products is the number of the product currently displayed on the shelf board. This number is manually input when the product is first taken out of the shelf, and is updated by the inventory management unit 136 every time an increase or decrease is detected by the increase/decrease detection unit 128.

Figure 11 is a data structure diagram showing an example of the sensor information storage unit 138 of Figure 8. The sensor information storage unit 138 stores the correspondence between distance measurement sensors and products. The sensor information storage unit 138 stores a sensor ID that identifies the distance measurement sensor, the shelf board ID of the shelf board on which the distance measurement sensor detects the customer's arm, the position where the distance measurement sensor is attached, and the product ID of the product corresponding to the distance measurement sensor, in association with each other. The position where the distance measurement sensor is attached is represented by the z coordinate described above. A product corresponding to a distance measurement sensor is a product whose existence range includes the position of the distance measurement sensor.

Returning to FIG. 8, the image acquisition unit 122 acquires images from a surveillance camera 24 that includes at least a portion of the upper shelf 12 and at least a portion of the lower shelf 14 within its angle of view.

The product placement registration unit 124 identifies the placement of products on the upper shelf 12 or the lower shelf 14 by analyzing the images acquired by the image acquisition unit 122 at a predetermined time interval, for example, every minute, every 10 minutes, or every hour. Alternatively, when a store employee changes the placement of the shelves, they may explicitly issue a registration instruction by pressing a switch or taking a specific pose and having the pose recognized by pose estimation. The product placement registration unit 124 registers the identified placement in the inventory management holding unit 134, or updates the inventory management holding unit 134 with the identified placement.

The product placement registration unit 124 updates the sensor information storage unit 138 with the identified placement. For example, if the ranges of "butter bread" (product ID "BTT") and "curry bread" (product ID "CUR") change from "7 < z < 14" and "14 < z < 30" to "9 < z < 16" and "16 < z < 21", respectively, as a result of moving the products on the lower shelf 14 to the center, the product placement registration unit 124 changes the product ID corresponding to sensor ID "A003" from "CUR" to "BTT" in the sensor information storage unit 138 in FIG. 11.

Figure 12 is a flowchart showing a series of processing steps in the product placement registration unit 124 of Figure 8. The product placement registration unit 124 acquires an image of the product shelf 10 acquired by the image acquisition unit 122 (S702). The product placement registration unit 124 determines one of the price tags in the acquired image as the price tag to be processed, and identifies the corresponding product by reading the one-dimensional code of the price tag to be processed (S704). The product placement registration unit 124 identifies the position of the price tag to be processed in the acquired image, that is, the shelf board ID of the shelf board to which the price tag to be processed is attached and the position of the price tag represented by the z coordinate defined for the shelf board (S706). The product placement registration unit 124 estimates the position of the corresponding separator from the position of the price tag to be processed identified in step S706 using a known positional relationship. The product placement registration unit 124 registers the estimated result of the separator position in the "shelf board ID" and "separator position" of the inventory management holding unit 134 in association with the product identified in step S704.

The product placement registration unit 124 determines whether all price tags in the acquired image have been processed (S710). If unprocessed price tags remain (N in S710), the process returns to step S704, and the unprocessed price tags are selected for processing. If all price tags have been processed (Y in S710), the product placement registration unit 124 estimates the presence range of each product from the separator position of the product registered in the inventory management holding unit 134, and registers it in the inventory management holding unit 134 (S712). For example, if the separator position of a product is z = 10 and the separator position of the product next to it is z = 15, the product placement registration unit 124 estimates the presence range of the product as 10 < z < 15. Figure 10 shows an example of the presence range estimated in this way.

Returning to FIG. 8, the weight change acquisition unit 120 receives the output of a number of weight sensors that are provided at different positions so that each sensor supports a shelf board. The weight change acquisition unit 120 calculates the amount of change in the output from the received output. For example, the weight change acquisition unit 120 receives measurement values from each of the left weight sensor 22a and right weight sensor 22b that support the upper shelf board 12, and calculates the amount of change in each measurement value. The weight change acquisition unit 120 similarly acquires the amount of change in the measurement value for the pair of weight sensors 22c, 22d that support the lower shelf board 14.

The position identification unit 126 identifies a position on the shelf related to a specific action of the customer by analyzing the image acquired by the image acquisition unit 122. In this embodiment, the position identification unit 126 analyzes the image using a known pose estimation technique and detects that the customer has extended his/her arm toward a product placed on the upper shelf 12 or the lower shelf 14. The position identification unit 126 also identifies the shelf to which the arm has been extended and the position on that shelf. In the example of FIG. 2, the position identification unit 126 detects that the customer 26 has extended his/her arm 26a to position z=20 on the upper shelf 12.

The position identification unit 126 also identifies a position on the lower shelf 14 related to the person's action based on information acquired from the distance measurement sensor 28a of the distance measurement sensor array 28. In particular, the position identification unit 126 identifies the position of the distance measurement sensor 28a that detected the object as the position on the lower shelf 14 beyond the customer's outstretched arm. The position identification unit 126 may identify the sensor ID of the distance measurement sensor 28a that detected the object, or may identify the position of the distance measurement sensor 28a by referring to the sensor information storage unit 138.

The increase/decrease detection unit 128 detects how much of each product has increased or decreased based on the amount of change in the output of the weight sensors acquired by the weight change acquisition unit 120 and the arrangement of the products on the shelf boards stored in the inventory management storage unit 134. In particular, the increase/decrease detection unit 128 detects which products have increased or decreased by comparing the position of the center of gravity of the increased or decreased product, which is determined from the amount of change in the output of the weight sensors acquired by the weight change acquisition unit 120, with the arrangement of the products on the shelf boards. The detection principle in the increase/decrease detection unit 128 is as described above with reference to Figures 3 to 6. When the increased or decreased product cannot be identified based on the amount of change in the output of the weight sensors alone, or to verify the product identified based on such amount of change, the position identified by the position identification unit 126 is used.

The settlement processing unit 130 executes settlement processing when a customer leaves the store. For example, when a customer performs settlement processing at the store's POS, the settlement processing unit 130 may send information about the customer's unpaid items registered in the customer information storage unit 132 to the POS. Alternatively, after it is confirmed that the customer has left the store, the settlement processing unit 130 may execute electronic payment based on the information about the customer's unpaid items registered in the customer information storage unit 132. Electronic payment may be realized using known payment technologies such as credit card payment or electronic money payment.

The inventory management unit 136 updates the inventory management holding unit 134 with the detection results from the increase/decrease detection unit 128. The inventory management unit 136 may also register the increased or decreased products and their quantities detected by the increase/decrease detection unit 128 in a master inventory management table (not shown).

The operation of the management device 100 having the above configuration will now be described.
13 is a flow chart showing a series of processing steps in the management device 100 of FIG. 8. The weight change acquisition unit 120 acquires the amount of change in the measurement value of the left weight sensor and the amount of change in the measurement value of the right weight sensor (S802). The increase/decrease detection unit 128 calculates the inverse ratio of the ratio of the two acquired amounts of change (S804). The increase/decrease detection unit 128 refers to the inventory management storage unit 134 and identifies products whose existence range includes the position on the shelf specified by the calculated inverse ratio (S806). At this time, if the position on the shelf specified by the calculated inverse ratio is the position of a separator or its vicinity, the increase/decrease detection unit 128 may proceed to step S812 described below.

The increase/decrease detection unit 128 judges whether the total amount of change in the measured values of the two weight sensors is N times the weight of each item identified in step S806 (N is a natural number) (S808). If it is judged to be N times (Y in S808), the increase/decrease detection unit 128 identifies the item identified in step S806 as an increased or decreased item, and identifies the obtained N as the number of increased or decreased items (S810). The increase/decrease detection unit 128 performs verification by comparing the increased or decreased items obtained in step S810 with the increased or decreased items identified from the position of the customer's arm identified by the position identification unit 126 (described later in step S812) (S822). If the verification results in a mismatch, the increase/decrease detection unit 128 executes a predetermined exception process, such as requesting the operator to confirm the image.

If it is determined in step S808 that the number is not N times greater (N in S808), the increase/decrease detection unit 128 identifies multiple products that have increased or decreased from the position of the customer's arm identified by the position identification unit 126 (S812). Since there is no distance measurement sensor for the upper shelf 12 but the image of the surveillance camera 24 is good, the position of the customer's arm identified from the image of the surveillance camera 24 is used. The increase/decrease detection unit 128 identifies the products that have increased or decreased from the identified position of the customer's arm and the positions of the products registered in the inventory management storage unit 134. For example, the increase/decrease detection unit 128 identifies the presence range of each product placed on the shelf beyond the outstretched arm by referring to the inventory management storage unit 134, and determines which presence range includes the identified position of the customer's arm. The increase/decrease detection unit 128 identifies the products that include the identified position of the customer's arm in their presence range as the products that have increased or decreased.

For the lower shelf 14, the sensor ID or sensor position of the distance measurement sensor 28a that detected the object (arm) is used, or both the detection result of the distance measurement sensor 28a and the analysis result of the image from the surveillance camera 24 are used.

When using the detection result of the distance measurement sensor 28a, the increase/decrease detection unit 128 refers to the sensor information storage unit 138 and identifies the product on the lower shelf 14 that corresponds to the sensor ID of the distance measurement sensor 28a that detected the object as the product that has increased or decreased. In the example of Figure 11, when the distance measurement sensor with sensor ID "A001" detects an object, the product with product ID "BTT" on the shelf with shelf ID "PAN2" is identified as the product that has increased or decreased.

The increase/decrease detection unit 128 calculates the weight of each product that has increased or decreased from the position on the shelf determined by the inverse ratio (the position of the center of gravity of the increased or decreased product), the position of each product that has increased or decreased, and the sum of the two changes (S814). The increase/decrease detection unit 128 determines the number of products that have increased or decreased from the weight of each product that has increased or decreased and the weight per unit of the product (S816). The principle of the calculation in steps S814 and S816 is as described with reference to Figures 5 and 6.

The inventory management unit 136 updates the remaining quantity of the corresponding item in the inventory management storage unit 134 based on the increased or decreased item and its quantity obtained in step S810, or the increased or decreased item and its quantity obtained in step S816 (S818).

When the increase/decrease detection unit 128 detects a decrease in the number of products, i.e., when a customer picks up a product, the increase/decrease detection unit 128 updates the customer information storage unit 132 with the detection result (S820). The increase/decrease detection unit 128 processes the image acquired from the surveillance camera 24 to identify the customer who picked up the product. For example, the management device 100 performs ID authentication when the customer enters the store, and then continues to track the location of the customer. In the example of FIG. 9, the management device 100 continues to update the current location of the customer corresponding to the customer ID of the customer. Therefore, when the pickup of a product is detected, it is possible to know who is standing in front of the shelf. The increase/decrease detection unit 128 uses a change in weight as a trigger to perform a process of matching the arm with the customer who owns the arm by combining pose estimation, depth information, etc. The increase/decrease detection unit 128 registers in the customer information storage unit 132 that the customer identified in this manner has picked up the number of items identified in step S810 or step S812 calculated in step S810 or step S816. For example, if it is detected that a customer identified by customer ID "CO1" has picked up "one" "apple pie," the increase/decrease detection unit 128 adds "apple pie: 1" to the information on unpaid items in the entry for customer ID "CO1" in the customer information storage unit 132 (or, if "apple pie: 1" has already been registered, it increments the number by 1 to make it "apple pie: 2").

As a specific example, assume that after steps S802 and S804, it is found that the position on the shelf identified by the inverse ratio is z=7 on the upper shelf 12, and the total weight reduction is 240g. In step S806, z=7 is included in the existence range of "bread stew" in FIG. 10, "2<z<10", so the "bread stew" is identified. In step S808, the total weight reduction of 240g is twice the weight of one "bread stew" of "120g", so the process proceeds to step S810 and it is detected that "two" "bread stew" have been reduced. If the arm of the customer with customer ID "CO1" is detected near z=7 as a result of analyzing the image from the surveillance camera 24, the verification in step S822 is successful, and the remaining number of "bread stew" in the inventory management holding unit 134 is updated from "4" to "2" in step S818. In step S820, "Bread Stew: 2" is added to the unpaid product information for customer ID "CO1" in the customer information storage unit 132.

In the above-described embodiment, examples of the storage unit are a hard disk and a semiconductor memory. Furthermore, based on the description in this specification, it will be understood by those skilled in the art who have read this specification that each unit can be realized by a CPU (not shown), an installed application program module, a system program module, a semiconductor memory that temporarily stores the contents of data read from a hard disk, and the like.

The management device 100 according to this embodiment uses the surveillance cameras 24 and one-dimensional codes to automatically register product placement, and detects increases or decreases in product numbers based on a comparison of changes in the measured values of multiple weight sensors. This allows store staff to freely rearrange products while eliminating the need to re-register products when they are rearranged, and also automates inventory management and checkout when customers pick up products. This further reduces the number of staff required in a store.

In addition, with the management device 100 according to this embodiment, there is no need to provide a weight sensor for each product; product placement is registered via the surveillance camera 24, and an increase or decrease in the number of products is detected based on the change in the measured values of a pair of left and right weight sensors. Therefore, there is no need to place unsightly weight sensors on the shelves, so flexibility in product display can be maintained or increased, vertical display efficiency can be maintained or increased, and the appearance of the product shelf 10 can be maintained.

In addition, in detecting an increase or decrease in an item in the management device 100 according to this embodiment, the primary focus is on identifying the item by comparing the change in the measured values of multiple weight sensors, and the secondary focus is on detecting the position of the customer's arm by analyzing the image from the surveillance camera 24. This makes it possible to more reliably detect the pick-up of an item even in the event of occlusion, such as when the arm is obscured by the customer's head or torso in the image from the surveillance camera 24 and cannot be seen, or is obscured by another customer, or when the line of sight of the surveillance camera 24 is blocked and the arm cannot be seen.

In addition, the management device 100 according to this embodiment further uses a distance measurement sensor 28a to identify the position of the customer's arm. This can therefore further increase the accuracy of detecting an increase or decrease in the number of products. Furthermore, even if the arm is not visible in the image from the surveillance camera 24 due to occlusion, the distance measurement sensor 28a can detect the arm more reliably.

In addition, the management device 100 according to this embodiment is provided with separators that separate products. Therefore, for example, fraudulent behavior such as switching products at the back of the shelf can be prevented. In the example of FIG. 2, if there were no separators, a customer could reach out to the cheese bread on the top shelf 12, pass the cheese bread straight through, bend their arm to the left at the back, grab the bread stew next to it, and pull it out. In this case, the customer would be able to buy bread stew (240 yen) for the price of the cheese bread (150 yen). If the weight difference between the bread stew and the cheese bread is large, such fraudulent behavior can be detected, but if the weight difference is small, it is relatively difficult to detect. Therefore, by providing separators that separate the products, such fraudulent behavior can be prevented.

Second Embodiment
In the first embodiment, the case where the increased or decreased products are identified based on the ratio of the change in the measured value of a pair of weight sensors is described. In the second embodiment, the increased or decreased products are identified by a distance measuring sensor, and the increased or decreased number is determined based on the change in the total weight obtained from the weight sensor. In the following, the case where the increase or decrease in the products displayed on the lower shelf 14 in FIG. 2 is detected by the distance measuring sensor array 28 and the pair of weight sensors 22c, 22d is described. However, it will be clear to those skilled in the art who have read this specification that the increase or decrease in the products displayed on the upper shelf 12 can also be detected by providing a separate distance measuring sensor for the upper shelf 12.

Figure 14 is a block diagram showing the functions and configuration of a management device 200 according to the second embodiment. Each block shown here can be realized in hardware terms by elements and mechanical devices such as a computer CPU, and in software terms by computer programs, etc., but here we have depicted a functional block realized by the cooperation of these. Therefore, those skilled in the art who have read this specification will understand that these functional blocks can be realized in various ways by combining hardware and software.

The management device 200 includes an image acquisition unit 122, a product placement registration unit 124, a total weight change acquisition unit 202, an increase/decrease detection unit 204, a sensor output acquisition unit 206, a settlement processing unit 130, an inventory management unit 136, a customer information storage unit 132, an inventory management storage unit 134, and a sensor information storage unit 138.

The total weight change acquisition unit 202 acquires the change in weight of multiple types of products placed on the lower shelf 14 of the product shelf 10 from the left weight sensor 22c and the right weight sensor 22d, which measure the weight of the lower shelf 14. The total weight change acquisition unit 202 calculates the change in the total weight of the products by summing the change in the measurement value of the left weight sensor 22c and the change in the measurement value of the right weight sensor 22d.

The sensor output acquisition unit 206 acquires detection results from the distance measurement sensors 28a of the distance measurement sensor array 28. The sensor output acquisition unit 206 monitors the output of each distance measurement sensor 28a, and when the output of a certain distance measurement sensor 28a indicates the detection of an object, it identifies the sensor ID of that distance measurement sensor 28a.

The increase/decrease detection unit 204 detects how much of each product has increased or decreased based on the change in total weight acquired by the total weight change acquisition unit 202 and the detection results acquired by the sensor output acquisition unit 206. The increase/decrease detection unit 204 refers to the sensor information storage unit 138 and identifies the product on the lower shelf 14 corresponding to the sensor ID identified by the sensor output acquisition unit 206 as the product that has increased or decreased. In the example of FIG. 11, when the distance measurement sensor with sensor ID "A001" detects an object, the product with product ID "BTT" on the shelf with shelf ID "PAN2" is identified as the product that has increased or decreased. The increase/decrease detection unit 204 refers to the inventory management storage unit 134 to acquire the weight of each identified product, and calculates the number of increases or decreases by dividing the change in total weight by that weight.

The operation of the management device 200 having the above configuration will now be described.
Fig. 15 is a flow chart showing a series of processes in the management device 200 in Fig. 14. The management device 200 acquires the detection results of the distance measurement sensor 28a from the distance measurement sensor array 28 (S302). The management device 200 identifies the products that have increased or decreased based on the acquired detection results (S304). The management device 200 acquires the amount of change in the total weight of the products on the lower shelf 14 based on the measured values of the pair of weight sensors 22c, 22d (S306). The management device 200 calculates the increase or decrease in the number of products identified in step S304 based on the acquired amount of change (S308).

According to the management device 200 of this embodiment, there is no need to provide a weight sensor for each product, and the increase or decrease in the number of products is detected based on the change in the measured values of the distance sensor and a pair of weight sensors on the left and right. Therefore, there is no need to place unsightly weight sensors on the shelves, so the flexibility of product display can be maintained or increased, the display efficiency in the vertical direction can be maintained or increased, and the appearance of the product shelf 10 can be maintained.

In addition, the management device 200 in this embodiment uses the distance sensor 28a and a pair of weight sensors 22c, 22d to detect the increase or decrease in the number of products, so that the increase or decrease in the number of products can be detected more reliably even if occlusion occurs.

(Comparison between systems)
(1) Camera + 1D code + weight sensor total quantity - This is possible if only one item is being picked up.
・It is not anticipated that different products may be picked up at the same time. (2) Camera + 1D code + total weight sensor + distance sensor ・It can handle even if the number of products picked up increases to two.
・A case where two items A and one item B of the same weight are taken, and a case where one item A and two items B are taken, are not anticipated. (3) Ratio of camera + 1D code + total weight sensor amount + weight sensor change amount ・Can handle even if the number of items taken increases to two.
・It is not possible to distinguish between combinations of products that have the same moment. (4) Camera + 1D code + total weight sensor + ratio of weight sensor change + distance sensor ・It can handle even if the number of products to be picked increases to two.
It is possible to distinguish between taking two items A and one item B of the same weight, and taking one item A and two items B (based on the ratio of changes in the weight sensor measurement value).
- It is possible to distinguish combinations of products with the same moment (from the distance sensor value)
- Although there is a probability that detection may be missed due to sensor noise, the accuracy is theoretically 100% (experimentally 99%).

Third Embodiment
The third embodiment relates to the arrangement and configuration of distance measuring sensors on a product shelf.
16 is a side view of a product shelf 300 according to the third embodiment. The product shelf 300 includes a support column 302, a bottom plate 304, a first shelf plate 306, a second shelf plate 308, a third shelf plate 310, a fourth shelf plate 312, a fifth shelf plate 314, a distance measurement sensor 316, a camera 320, and a camera support frame 322.

The first shelf 306, the second shelf 308, the third shelf 310, the fourth shelf 312, and the fifth shelf 314 are all attached and fixed to the support 302. The first shelf 306, the second shelf 308, and the third shelf 310 form a first shelf group 324 that has the same length in the depth direction. The fourth shelf 312 is longer in the depth direction than the third shelf 310 above it, and protrudes forward. The fifth shelf 314 is longer in the depth direction than the fourth shelf 312 above it, and protrudes forward. The fourth shelf 312 and the fifth shelf 314 form a second shelf group 326 that protrudes gradually forward the lower they are.

The camera 320 is supported by a camera support frame 322 and positioned so as to overlook the front of the product shelf 300. Due to the protruding configuration of the second shelf board group 326, the front of each shelf board belonging to that group is clearly visible in the angle of view of the camera 320. In contrast, the first shelf board group 324 has the same length in the depth direction, so the visibility of the front of the shelves is poorer than that of the second shelf board group 326. In particular, the front 310a of the third shelf board 310, which is located further down, is affected by occlusion caused by the first shelf board 306 and the second shelf board 308.

The distance sensor 316 is attached to the front of the underside of the first shelf 306, which is the top of three consecutive shelves, the first shelf 306, the second shelf 308, and the third shelf 310, and has a detection range below. These three consecutive shelves 306, 308, 310 and the distance sensor 316 are configured so that the detection range of the distance sensor 316 overlaps with the front 310a of the third shelf 310, which is the bottom of the three consecutive shelves.

The distance sensor 316 emits an infrared laser 328 in a substantially vertically downward direction. Since the second shelf 308 is configured to have substantially the same length as the first shelf 306 in the depth direction, the second shelf 308 does not interfere with the infrared laser 328, and the infrared laser 328 also passes through the front 310a of the third shelf 310. If the distance of an object detected by the distance sensor 316 is shorter, the object is determined to be in front of the second shelf 308, and if the distance of an object detected by the distance sensor 316 is longer, the object is determined to be in front 310a of the third shelf 310. The distance sensor 316 can also detect an object in front of the fourth shelf 312.

In this way, with the product shelf 300 according to this embodiment, a single distance sensor 316 can detect objects on multiple shelves 308, 310, and 312.

The technical idea of this embodiment can be applied to any case where a distance measurement sensor is attached to a product shelf. For example, the distance measurement sensor of the first and second embodiments may be arranged and configured as in this embodiment.

In this embodiment, the distance measurement sensor 316 is attached to the underside of the first shelf 306, but this is not limited to the above. For example, as shown by the dashed line in FIG. 16, the distance measurement sensor 318 may be attached to the camera support frame 322 so that the detection range of the distance measurement sensor 318 overlaps with the front of the first shelf 306.

In the present embodiment, the second shelf group 326 having a protruding configuration is arranged below the product shelf 300, but this is not limited to the above. FIG. 17 is a side view of a product shelf 400 according to a modified example of the third embodiment. The product shelf 400 includes a support 302, a bottom plate 304, a first shelf 406, a second shelf 408, a third shelf 410, a fourth shelf 412, a fifth shelf 414, a distance sensor 416, a camera 320, and a camera support frame 322.

The first shelf 406, second shelf 408, third shelf 410, fourth shelf 412, and fifth shelf 414 are all attached and fixed to the support 302. The second shelf 408 is longer in the depth direction than the first shelf 406 above it, and protrudes forward. The first shelf 406 and second shelf 408 form a third shelf group 426 that protrudes gradually further forward the lower they are. The third shelf 410, fourth shelf 412, and fifth shelf 414 are arranged below the third shelf group 426, and form a fourth shelf group 424 that has the same length in the depth direction.

Due to the protruding configuration of the third shelf group 426, the front of each shelf belonging to that group is well reflected in the angle of view of the camera 320. In contrast, the fourth shelf group 424 has the same length in the depth direction and protrudes further forward than the third shelf group 426, so the front of the shelves is not as well reflected compared to the third shelf group 426.

The distance measurement sensor 416 is attached to the front underside of the third shelf 410 (the third from the top of the five shelves) of the three consecutive shelves, the third shelf 410, the fourth shelf 412, and the fifth shelf 414, and has a detection range below. These three consecutive shelves 410, 412, 414 and the distance measurement sensor 416 are configured so that the detection range of the distance measurement sensor 416 overlaps with the front of the fifth shelf 414, the lowest of the three consecutive shelves.

The above describes the configuration and operation of the management device and product shelves according to the embodiment. This embodiment is merely an example, and those skilled in the art will understand that various modifications are possible in the combination of each component and each process, and that such modifications are also within the scope of the present invention.

In the first and second embodiments, the position of the separator is estimated from the position of the price tag, and the product existence range is estimated from the position of the separator, but this is not limited to the above. For example, if no separator is provided, the position that bisects the space between adjacent price tags may be recognized as the boundary of the product existence range. Alternatively, the size of the existence range may be set in advance for each product. Alternatively, the automatic registration of product arrangement by the product arrangement registration unit 124 may be eliminated, and the existence range of each product may be registered manually. Even in this case, automatic increase/decrease detection by the increase/decrease detection unit 128 is possible.

In the first to third embodiments, the case where the distance measuring sensor is provided on the underside of the shelf board has been described, but this is not limited to this. FIG. 18 is a schematic diagram showing a distance measuring sensor 502 attached to the upper surface of a shelf board 500. The distance measuring sensor 502 has an upward detection range and detects an object in front of the shelf board 500 to which the distance measuring sensor 502 is attached. FIG. 19 is a schematic diagram showing multiple distance measuring sensors 512 attached to the sides of a shelf board 510. The multiple distance measuring sensors 512 are attached to a frame extending in the height direction of the product shelf and have a detection range in the longitudinal direction of the shelf board 510. In the configuration of FIG. 19, the height position of each distance measuring sensor 512 is known, so the position of the detected object on the shelf board 510 is measured together with the detected distance and the known height. More generally, the distance measuring sensor may be arranged and configured in any manner as long as it can detect the position of an arm extending toward the shelf board.

In the first and second embodiments, a distance measuring sensor is used as the object detection sensor, but a sensor other than a distance measuring sensor may be used as long as it is capable of detecting the presence of an object.

In the first embodiment, when no products are displayed at the position of the center of gravity determined from the ratio of the changes in the measured values of the left and right weight sensors 22a and 22b, the surveillance camera 24 and/or the distance sensor are used to narrow down the candidates, but this is not limited to the above. For example, the number of products picked up may be calculated from the change in the total weight, the center of gravity position may be calculated for each possible combination of the number of products, and the calculated center of gravity position may be compared with the center of gravity position calculated from the ratio. For example, in the state of FIG. 5, if ΔML+ΔMR indicates the value for three products 30, it is understood that three products 30 have been picked up. Therefore, three products 30 placed at z=z1, z2, z3, and z4 are arbitrarily selected and the center of gravity position is calculated. If the center of gravity position obtained as a result of the calculation matches the center of gravity G position calculated by the inverse ratio, it is understood that the three products 30 used in the calculation have been picked up.

(Modification of detection logic)
In the first embodiment, the center of gravity of the increased or decreased product is calculated from the inverse ratio of the ratio of the change in the measured values of the left and right weight sensors, and the center of gravity is compared with the arrangement of the products on the shelf to determine the increased or decreased product. However, the present invention is not limited to this. For example, the increased or decreased product may be determined directly from the change in the measured values of the weight sensors without calculating the center of gravity of the increased or decreased product. In the management device according to this modification, for each product displayed on the upper shelf 12, the change in the measured values of the left weight sensor 22a and the right weight sensor 22b due to the increase or decrease of one product (hereinafter referred to as δML and δMR, respectively) is calculated in advance and registered in the inventory management storage unit. FIG. 20 is a data structure diagram showing an example of the inventory management storage unit 250 according to the modification. The inventory management storage unit 250 stores a product ID that identifies the product, a product name, the weight of each product, a shelf ID that identifies the shelf on which the product is placed, the positions of separators that separate the products, the range in which the product is located on the shelf, the remaining number of products, and (δML, δMR), in association with each other.

When the changes ΔML and ΔMR in the measurements of the left weight sensor 22a and the right weight sensor 22b are detected, the management device according to this modified example compares the set of detected changes (ΔML, ΔMR) with the set (δML×K, δMR×K) (K is a natural number) obtained from the set of changes per product (δML, δMR) registered in the inventory management holding unit 250. If there is a product that matches as a result of the comparison, the management device determines that the number of products has increased or decreased by K, updates the remaining number in the inventory management holding unit 250, and updates the customer information holding unit 132.

If there is no matching product, the management device determines that multiple products placed in different locations have increased or decreased. The management device combines the sets of change amounts per product (δML, δMR) registered in the inventory management holding unit 250 for different products, and compares them with the detected set of change amounts (ΔML, ΔMR). If there is no match, the management device repeats the same comparison with a different combination. The management device repeats this process until a match is found, and identifies the products that have increased or decreased and their numbers based on the combination of products that match.

Figure 21 is a schematic diagram showing the upper shelf 12, the left weight sensor 22a, the right weight sensor 22b, and a product 30 placed on the upper shelf 12. For ease of explanation, the point where the left weight sensor 22a and the upper shelf 12 contact is taken as z = 0, the z axis is defined extending from that point toward the point where the right weight sensor 22b and the upper shelf 12 contact, and the point where the right weight sensor 22b and the upper shelf 12 contact is taken as z = 25. Furthermore, all five products 30 placed on the upper shelf 12 are assumed to have the same mass of "1".

In FIG. 21, it is assumed that a customer picks up one product 30 located at position z=10 and one product 30 located at position z=20 at approximately the same time.
For product 30 at position z=10,
δML=1×15/(10+15)=0.6, δMR=1-ML=0.4
It becomes.
For product 30 at position z=20,
δML=1×5/(20+5)=0.2, δMR=3-ML=0.8
It becomes.
Therefore, in this case, ΔML and ΔMR are respectively ΔML=0.6+0.2=0.8 and ΔMR=0.4+0.8=1.2.

Fig. 22 is a schematic diagram showing a state in which the same number of products 30 as in Fig. 21 are picked up from different positions. In Fig. 22, assume that a customer picks up one product 30 at position z=5 and one product 30 at position z=10 at almost the same time.
For product 30 at position z=5,
δML=1×20/(5+20)=0.8, δMR=1−ML=0.2
It becomes.
Therefore, in this case, ΔML and ΔMR are calculated as ΔML=0.8+0.6=1.4 and ΔMR=0.2+0.4=0.6, respectively.

Referring to Figures 21 and 22, the total change in weight in both cases is the equivalent of two products 30 (2), but the values of ΔML and ΔMR are different. Therefore, by looking at the set of changes (ΔML, ΔMR), it is possible to distinguish between the states in Figures 21 and 22, where the change in total weight can only detect the simultaneous pick-up of the same two products 30. That is, if (ΔML, ΔMR) = (0.8, 1.2), it can be determined that one product 30 with z = 10 and one product 30 with z = 20 were picked up at the same time. If (ΔML, ΔMR) = (1.4, 0.6), it can be determined that one product 30 with z = 5 and one product 30 with z = 10 were picked up at the same time.

In this modified example, it becomes impossible to identify which product has been picked up in the following situations. In such cases, the image from the surveillance camera 24 and the distance sensor 28a are used to identify the product that has been picked up, as in the embodiment.

Fig. 23 is a schematic diagram showing the upper shelf board 12, the left weight sensor 22a, the right weight sensor 22b, and products 30 placed on the upper shelf board 12. In Fig. 23, assume that a customer picks up one product 30 at position z=10 and one product 30 at position z=15 at approximately the same time.
For product 30 at position z=15,
δML=1×10/(5+20)=0.4, δMR=1-ML=0.6
It becomes.
Therefore, in this case, ΔML and ΔMR are calculated as ΔML=0.6+0.4=1 and ΔMR=0.4+0.6=1, respectively.

Fig. 24 is a schematic diagram showing a state in which the same number of products 30 as in Fig. 23 are picked up from different positions. In Fig. 24, assume that a customer picks up one product 30 at position z=5 and one product 30 at position z=20 at almost the same time.
In this case, ΔML and ΔMR are calculated as ΔML=0.8+0.2=1 and ΔMR=0.2+0.8=1, respectively.

Thus, even if (ΔML, ΔMR) = (1, 1) is obtained, there is a possibility that one product 30 with z = 10 and one product 30 with z = 15 were picked up at the same time (Figure 23), and there is also a possibility that one product 30 with z = 5 and one product 30 with z = 20 were picked up at the same time (Figure 24), and these cannot be distinguished from ΔML and ΔMR. In such cases, in this modified example, as in the embodiment, the image from the surveillance camera 24 and/or the detection results of the distance sensor array 28 are used to identify which product was picked up.

10 Product shelf, 12 Upper shelf, 14 Lower shelf, 22a Left weight sensor, 22b Right weight sensor, 24 Surveillance camera, 26 Customer, 100 Management device.

Claims (9)

A device for detecting an increase or decrease in the number of products placed on a shelf of a product shelf, comprising:
a first acquisition means for acquiring an amount of change in output of a plurality of weight sensors provided at different positions so as to support the shelf board;
A second acquisition means for acquiring information of a subject from a camera that includes at least a part of the shelf board within an angle of view;
a detection means for detecting how much each product has increased or decreased based on the acquired change in output from the weight sensors and the acquired information on the subject;
The detection means is a management device that detects how much of each item has increased or decreased by using a position on the shelf related to a specified human action, identified from the acquired information on the subject, when the position of the center of gravity of the increased or decreased item, determined from the change in output from multiple weight sensors, is within a specified distance from a predetermined position where no items are displayed . The shelf further includes a first identification unit that identifies a position on the shelf that is related to a human action based on information acquired from an object detection sensor that is associated with the shelf,
The management device according to claim 1 , wherein the detecting means uses the position identified by the first identifying means to detect how much an item has increased or decreased. A plurality of object detection sensors are disposed in association with the shelf board;
The management device according to claim 2 , wherein the first identification unit identifies a position of an object detection sensor that has detected an object as a position on the shelf that is related to a human action. The shelf board is provided with separators for separating products by type,
The management device according to claim 3 , wherein the separator is disposed between adjacent object detection sensors. A second identification means is further provided for identifying an arrangement of the plurality of products on the shelf board from the acquired information of the subject,
The management device according to any one of claims 1 to 4, wherein the detection means detects which products have increased or decreased by comparing the position of the center of gravity of the increased or decreased products, which is determined from the change in output of multiple weight sensors obtained, with the specified arrangement. A plurality of codes corresponding to the plurality of products are attached to at least a portion of the shelf board that falls within the angle of view of the camera, and each of the codes has code information that encodes information that identifies the corresponding product in a visible manner;
The management device according to claim 5 , wherein the second identification means identifies the position of the product from code information included in the image captured by the camera. The management device according to claim 6, wherein the camera is disposed above the front of the product shelf so as to overlook the front of the product shelf, and the code information is attached in a direction that is within the angle of view of the camera. a third identification means for identifying a position on the shelf board related to a predetermined action of a person by analyzing the acquired information of the subject;
The management device according to claim 5 , wherein the detection means uses the position identified by the third identification means to detect how much the number of products has increased or decreased. A means for acquiring a change in weight of a plurality of types of products placed on a shelf of a product shelf from a sensor that measures the weight of the shelf;
A means for acquiring a detection result from an object detection sensor disposed in association with the shelf board;
a means for detecting how much each product has increased or decreased based on the acquired weight change and the acquired detection result;
The detection means is a management device that detects how much of which product has increased or decreased, using the position on the shelf identified from the acquired detection results, when the position of the center of gravity of the product that has increased or decreased, determined from the acquired weight change, is within a specified distance from a predetermined position where no products are displayed.

Images