JP7784641B2 - Inventory management system and inventory management method - Google Patents

Description

This disclosure relates to an inventory management system and an inventory management method.

Technology related to storage cabinets for storing objects has been proposed in the past. Patent Document 1 discloses a refrigerator that allows for inventory management within the cabinet while reducing the user's workload, and prevents a decrease in effective storage space within the cabinet and an increase in manufacturing costs.

JP 2015-81762 A

The present disclosure provides an inventory management system that can determine that an item is subject to inventory management based on the user's intentional actions.

An inventory management system according to one aspect of the present disclosure includes a photographing unit that photographs a video of the opening of a storage unit as viewed from above; a first detection unit that performs image processing on the video to detect whether a stationary object other than the storage unit is visible in a first area of the video; a second detection unit that performs image processing on the video to detect whether a user's hand or arm positioned in front of the opening is visible in a second area of the video; and a determination unit that determines that a stationary object other than the storage unit is an object of inventory management when a stationary object other than the storage unit is detected to be visible in the first area during a period when the user's hand or arm is not visible in the second area.

An inventory management system according to one aspect of the present disclosure can determine that an item is subject to inventory management based on the user's intentional actions.

FIG. 1 is a block diagram showing the functional configuration of a warehousing and shipping management system according to an embodiment. FIG. 2 is an external view of a refrigerator and an incoming/outgoing management device provided in the incoming/outgoing management system according to the embodiment. FIG. 3 is a first diagram showing a moving image captured by the image capturing section. FIG. 4 is a second diagram showing the moving image captured by the image capturing section. FIG. 5 is a third diagram showing a moving image captured by the image capturing section. FIG. 6 is a fourth diagram showing a moving image captured by the image capturing section. FIG. 7 is a state transition diagram of four states used in managing food inventory. FIG. 8 is a block diagram showing the functional configuration of the information processing unit. FIG. 9 is a flowchart of an operation example 1 of the inventory management system according to the embodiment. FIG. 10 is a diagram showing an example of food management information. FIG. 11 is a sequence diagram of a second example of the operation of the inventory management system according to the embodiment. FIG. 12 is a diagram showing an example of a display screen showing the contents of food management information.

The following describes the embodiments in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims are described as optional components.

Please note that each figure is a schematic diagram and is not necessarily an exact representation. In each figure, substantially identical components are designated by the same reference numerals, and duplicate explanations may be omitted or simplified.

(Embodiment)
[composition]
First, the configuration of the inventory management system according to the embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of the inventory management system according to the embodiment.

The inventory management system 10 shown in Figure 1 is a system for managing food stored in a refrigerator 20. The inventory management system 10 comprises a refrigerator 20, an inventory management device 30, a server device 40, and an information terminal 50.

Refrigerator 20 is an example of a storage cabinet, and is installed in a user's home or the like to refrigerate food. Figure 2 is an external view of refrigerator 20 (and inventory management device 30). As shown in Figure 2, refrigerator 20 comprises a main body 21, a door 22 covering the opening of main body 21, and multiple shelves 24 provided in a storage chamber 23 within main body 21. When door 22 is opened, refrigerator 20 is ready to place food on each of the multiple shelves 24 (i.e., store food in storage chamber 23). In the example of Figure 2, refrigerator 20 has two doors 22 covering storage chamber 23, but it may also have only one door 22 covering storage chamber 23.

The inventory management device 30 is attached, for example, to the top surface of the refrigerator 20, and performs information processing to manage food stored in the refrigerator 20. The inventory management device 30 is, for example, retrofitted to an existing refrigerator 20, but may also be built into the refrigerator 20. In other words, the inventory management device 30 may be a device separate from the refrigerator 20, or may be a device manufactured integrally with the refrigerator 20. As shown in FIG. 1, the inventory management device 30 includes an imaging unit 31, an opening/closing sensor 32, an information processing unit 33, a memory unit 34, a communication unit 35, and an alarm unit 36.

The camera unit 31 captures video from above the refrigerator 20 showing food being taken in and out of the refrigerator 20. That is, the camera unit 31 captures video of the refrigerator 20 with the door 22 open (video of the opening of the refrigerator 20) as viewed from above the refrigerator 20. The camera unit 31 is implemented, for example, by a camera with a fisheye lens, but may also be implemented by a camera with a normal lens. The video includes multiple images captured continuously, and each of the multiple images is also referred to as a frame.

The photographing unit 31 is attached to the housing of the inventory management device 30, but may also be attached to the refrigerator 20 separately from the housing of the inventory management device 30. In other words, the photographing unit 31 may be retrofitted to the refrigerator 20. In this case, the photographing unit 31 has a structure such as a magnet, suction cup, or clip that allows it to be retrofitted to the refrigerator 20.

The opening/closing sensor 32 is located at the top of the refrigerator 20 and detects whether the door 22 is open or closed (how open it is) by measuring the distance from the opening/closing sensor 32 to the door 22. The opening/closing sensor 32 is realized, for example, by an infrared distance measuring sensor. The opening/closing sensor 32 can individually detect whether the right door 22 and the left door 22 are open or closed by separately measuring the distance from the opening/closing sensor 32 to the right door 22 and the distance from the opening/closing sensor 32 to the left door 22.

The opening/closing sensor 32 may be realized by an angle sensor or the like instead of an infrared distance sensor. The opening/closing sensor 32 may also be provided by the refrigerator 20, in which case the inventory management device 30 does not need to be provided with the opening/closing sensor 32.

The information processing unit 33 performs information processing to manage food stored in the refrigerator 20. The information processing unit 33 is realized, for example, by a microcomputer, but may also be realized by a processor or dedicated circuit. The information processing unit 33 has, as functional components, a food detection unit 37, a hand/arm detection unit 38, and a determination unit 39. The functions of the food detection unit 37, hand/arm detection unit 38, and determination unit 39 are realized, for example, by the microcomputer or the like that constitutes the information processing unit 33 executing a computer program stored in the memory unit 34. The functions of the food detection unit 37, hand/arm detection unit 38, and determination unit 39 will be described in detail below.

The memory unit 34 is a storage device that stores the computer program executed by the information processing unit 33 and various information required for the above information processing (such as the food management information described below). The memory unit 34 is realized, for example, by a semiconductor memory.

The communication unit 35 is a communication circuit that enables the inventory management device 30 to communicate with the server device 40 via the wide area communication network 60. The communication unit 35 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard used by the communication unit 35.

The notification unit 36 notifies the user that a food item has been recognized as a target for inventory management, that an error has occurred related to the image capture unit 31, etc. The notification unit 36 may be implemented, for example, by a speaker or buzzer, and notify by sound, but may also be implemented by a light-emitting element such as an LED (Light Emitting Diode), and notify by light emission.

The server device 40 is a computer located outside the facility where the refrigerator 20 is installed, and specifically, is a cloud server. The server device 40 performs information processing to present information about the food stored in the refrigerator 20. The server device 40 includes a communication unit 41, an information processing unit 42, and a memory unit 43.

The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the inventory management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used by the communication unit 41.

The information processing unit 42 performs information processing to present information about food stored in the refrigerator 20. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or dedicated circuit. The information processing unit 42 has a presentation unit 44 as a functional component. The function of the presentation unit 44 is realized, for example, by the microcomputer or the like that constitutes the information processing unit 42 executing a computer program stored in the memory unit 43. The function of the presentation unit 44 will be described in detail below.

The memory unit 43 is a storage device that stores the computer program executed by the information processing unit 42 and various information required for the above information processing. The memory unit 43 is realized, for example, by a semiconductor memory.

The information terminal 50 is an information terminal owned by the user. The user uses the information terminal 50 to check the food stored in the refrigerator 20 (the contents of the food management information) while away from home. The information terminal 50 is, for example, a portable information terminal such as a smartphone or tablet terminal, but may also be a stationary information terminal such as a personal computer.

[Determination of four states]
Next, the inventory management operation of food items in the refrigerator 20 performed by the inventory management system 10 will be described. When a user wants to store or remove food items from the refrigerator 20 for inventory management, the user temporarily holds the food items above a specific area on the bottom surface of the storage compartment 23. That is, the user intentionally holds the food items above the specific area. The inventory management device 30 can detect that the user has held the food items above the specific area by processing the video captured by the image capture unit 31. FIG. 3 shows an example of the video captured by the image capture unit 31.

Figure 3 is a diagram showing the entire shooting range of the shooting unit 31. The video shown in Figure 3 shows the opening of the refrigerator 20 and the storage room 23 located behind the opening. The video also shows a specific area P. The specific area P is an area on the bottom surface of the storage room 23 (the inner surface of the refrigerator 20) where a specific pattern is drawn, and is a real area. Although a user (person) is not shown in Figure 3, as shown in Figures 4 to 6, etc., described below, the video also shows a user located in front of the opening.

In the explanation of image processing for moving images described below, the image processing is performed on a predetermined range A0 corresponding to a portion of the shooting range of the image capture unit 31, rather than the entire shooting range of the image capture unit 31. In other words, in the explanation of image processing described below, the term frame, image, or moving image will refer to the portion of the predetermined range A0, rather than the entire shooting range of the image capture unit 31.

A first area A1 and a second area A2 are set within the specified range A0. The first area A1 and the second area A2 are virtual areas used in image processing and are shown by dashed lines in Figure 3. The first area A1 is set to include the specific area P. The first area A1 and the specific area P are set at locations suitable for the user to hold over an item they wish to use for inventory management.

The shapes and arrangements of the specific area P and the first area A1 are merely examples, and the shapes and arrangements of the specific area P and the first area A1 may be determined empirically or experimentally as appropriate by the designer of the inventory management system 10. It is also possible that an explicit specific area P is not provided, and the first area A1 is set on at least a portion of the bottom surface of the storage room 23 (a surface of a single color, such as white).

When a user stores food in the inventory, the video captured by the camera unit 31 is expected to change as shown in Figures 3 to 6.

Figure 3 shows a video before the user places food in the storage. This state is defined as the STANDBY state. In the STANDBY state, the user's hands or arms, or the food the user is holding (hereinafter referred to as the user's arms, etc.), are not visible within the specified range A0.

Figure 4 shows a video in which a user is moving food above a specific area; this state is defined as a READY state. The READY state is a state in which the user's arm or other object is visible in the first area A1 of the video and the user's arm or other object is moving.

Figure 5 shows a video in which a user holds food still above a specific area; this state is defined as the SET state. The SET state is a state in which the user's arm (in this case, food) is displayed in the first area A1 of the video and the user's arm is still (or, more accurately, moving very little).

Figure 6 shows a video of a user storing food in storage room 23 (placing it on shelf 24); this state is defined as the INOUT state. The INOUT state is a state in which the user's arm, etc. is visible in the first area A1 of the video and the user's arm, etc. is moving. After the INOUT state, when the user has finished placing the food, the state returns to STANDBY state. The STANDBY state, READY state, SET state, and INOUT state described above are represented in a state transition diagram as shown in Figure 7. Figure 7 is a state transition diagram of the STANDBY state, READY state, SET state, and INOUT state.

Although no video example is shown, when a user takes food out, the state transitions in the same order as when food is put in: STANDBY state, READY state, SET state, INOUT state, and STANDBY state.

[Image processing for determining four states]
The inventory management device 30 can manage food by determining the above four states based on the difference in pixel values between frames included in the video (presence or absence of movement) and the difference in pixel values between the first reference image and frames included in the video (presence or absence of the user's arm, etc.). The method for determining the four states will be described below with reference to FIG. 8. FIG. 8 is a block diagram showing the functional configuration of the information processing unit 33 included in the inventory management device 30. FIG. 8 also shows the imaging unit 31 and the open/close sensor 32.

In the following explanation of Figure 8 (explanation of image processing for determining the four states), the terms frame, image, video, etc. refer to a portion of the predetermined range A0 rather than the entire shooting range of the shooting unit 31. During image processing, the image is temporarily stored in the memory unit 34, but this point will not be explained further below.

As shown in FIG. 8, the information processing unit 33 has a food detection unit 37 and a judgment unit 39. The food detection unit 37 has a frame-to-frame difference calculation unit 37a, a reference image acquisition unit 37b, and a reference image difference calculation unit 37c.

First, we will explain how the inter-frame difference calculation unit 37a determines whether an object in a video is moving. The inter-frame difference calculation unit 37a calculates the difference between the pixel values of pixels included in a frame captured by the image capture unit 31 at time t-1 and the pixel values of pixels included in a frame at the current time t. For example, the inter-frame difference calculation unit 37a calculates the difference between the pixel values of corresponding pixels between a frame at time t-1 and a frame at the current time t. If the number of pixels included in a frame is m (m is a natural number), m differences are calculated. The inter-frame difference calculation unit 37a determines that an object in the video (such as a user's arm) is moving if n/m is equal to or greater than a predetermined ratio, and determines that an object in the video (such as a user's arm) is not moving if n/m is less than the predetermined ratio. The predetermined threshold and predetermined ratio may be determined empirically or experimentally as appropriate.

Next, we will explain how the reference image acquisition unit 37b and the reference image difference calculation unit 37c determine whether an object such as the user's arm is captured in the first area A1 of the video. The reference image acquisition unit 37b acquires the first reference image captured by the image capture unit 31. The first reference image is an image in which no objects other than the refrigerator 20 are captured in the first area A1, as shown in Figure 3.

For example, when the opening/closing sensor 32 detects that at least one of the two doors 22 is open, the reference image acquisition unit 37b causes the photographing unit 31 to photograph a first reference image and acquires the first reference image. In other words, the reference image acquisition unit 37b acquires a new first reference image each time the door 22 is opened. This makes it possible to acquire a first reference image that shows the latest interior of the refrigerator 20. Since it is a rule that food should not be placed on the specific area P when storing food in the refrigerator 20, the image captured immediately after the door 22 is opened does not show any objects other than the refrigerator 20 in the first area A1 that includes the specific area P, and can therefore be used as the first reference image.

It is not necessary that the first reference image be acquired when the door 22 is opened; for example, the first reference image may be acquired by the image capture unit 31 when the inventory management device 30 is newly installed (at initial setup), and this first reference image may continue to be used thereafter. The first reference image may also be acquired at any timing; for example, the first reference image acquired when the door 22 is opened may be updated periodically while the refrigerator door 22 is open.

When the first reference image is captured at any timing, the reference image acquisition unit 37b acquires (uses) the first reference image after confirming that no objects other than the refrigerator 20 are visible in the first area A1 of the captured image. For example, if the specific area P is an area in which a predetermined pattern is depicted, the reference image acquisition unit 37b can determine that no objects other than the refrigerator 20 are visible in the first area A1 if the predetermined pattern is detected in the first area A1 by pattern matching. Furthermore, if an explicit specific area P is not provided and the first area A1 is set as part of the bottom surface (a surface of a single color, such as white) of the storage compartment 23, the reference image acquisition unit 37b can determine that no objects other than the refrigerator 20 are visible in the first area A1 if no edges are detected in the first area A1 by edge detection.

The reference image difference calculation unit 37c calculates the difference between the pixel values of pixels included in the first region A1 of the first reference image acquired by the reference image acquisition unit 37b and the pixel values of pixels included in the first region A1 of the frame at the current time t. For example, the reference image difference calculation unit 37c calculates the difference between the pixel values of corresponding pixels between the first region A1 of the first reference image and the first region A1 of the frame at the current time t. If the number of pixels included in the first region A1 is p (p is a natural number), p differences are calculated. The reference image difference calculation unit 37c determines that the number of differences that are greater than a predetermined threshold, q (q is a natural number), of the p differences, is q (q is a natural number). If q/p is equal to or greater than a predetermined ratio, it determines that a user's arm, etc. is present in the first region A1. If q/p is less than the predetermined ratio, it determines that a user's arm, etc. is not present. The predetermined threshold and the predetermined ratio may be determined empirically or experimentally as appropriate.

Next, we will explain how the determination unit 39 determines the four states. The determination unit 39 determines which of the four states the current state is based on the determination result of the inter-frame difference calculation unit 37a (presence or absence of an object in the video), the determination result of the reference image difference calculation unit 37c (presence or absence of the user's arm, etc. in the first area A1), and the state transition diagram (Figure 7).

Immediately after the door 22 is opened, the user's arm or other object is not visible in the video (predetermined range A0), so there is no movement of objects visible in the video, and the user's arm or other object is not located in the first area A1. Therefore, the determination unit 39 determines that the refrigerator is in the STANDBY state. After the STANDBY state, when the user places their arm or other object inside the refrigerator 20, the user's arm or other object visible in the video moves, and the user's arm or other object is located in the first area A1, so the determination unit 39 determines that the refrigerator has transitioned to the READY state.

Subsequently, when the user intends to make food a management target and stops the device above the specific area P, the movement of the user's arm, etc. shown in the video stops (becomes less) and the user's arm, etc. (in this case, the food) is located in the first area A1, so the judgment unit 39 judges that the device has transitioned to the SET state.

When the determination unit 39 determines that the SET state has continued for a certain period of time, it determines that the state is in the INOUT state. The INOUT state is a state in which the user is performing the action of putting food in or taking food out, and the user's arm, etc. shown in the video is moving and is positioned in the first area A1. On the other hand, when the determination unit 39 determines that the SET state has not continued for a certain period of time (the duration of the SET state has not reached the certain period of time), it determines that the state has returned to the READY state.

After the INOUT state, when the user pulls their hand out of the refrigerator 20, the user's arm or other object is not visible in the video (predetermined range A0), meaning that there is no movement of the objects visible in the video, and the user's arm or other object is not located in the first area A1. Therefore, the determination unit 39 determines that the refrigerator is in the STANDBY state.

As described above, the information processing unit 33 has a food detection unit 37 that performs image processing on the video captured by the imaging unit 31 to detect whether a stationary object other than the refrigerator 20 is captured in the first area A1 of the video (specifically, whether food has stopped in the first area A1). This allows the information processing unit 33 (determination unit 39) to determine that a stationary object other than the refrigerator 20 is an object subject to inventory management when it is detected that the object is captured in the first area A1. More specifically, the information processing unit 33 can determine that a food object subject to inventory management has been stored (or removed) when the state transitions from the STANDBY state to the READY state, the SET state, the INOUT state, and then the STANDBY state in this order.

[Configuration to suppress erroneous determination]
Incidentally, when a user stores food that is not subject to inventory management, the user does not place the food above specific area P, so the food does not transition from the READY state to the SET state, and the state remains determined to be READY, and the user places their hand inside refrigerator 20 as shown in Figure 6. Here, if the user stands still with their hand inside refrigerator 20 as shown in Figure 6, determination unit 39 may erroneously determine that the state has transitioned from the READY state to the SET state.

Therefore, the information processing unit 33 sets a second area A2 in the video (predetermined range A0), and when the user's arm or the like is visible in the second area A2, it does not determine that the device is in the SET state even if the user's arm or the like in the video stops moving and the user's arm or the like is located in the first area A1. In other words, the determination unit 39 determines whether the device has transitioned to the SET state by adding the requirement that the user's arm or the like in the video stops moving and the user's arm or the like is located in the first area A1, as well as the requirement that the user's arm or the like is not visible in the second area A2.

As shown in Figures 3 to 6, the second area A2 is located further back than the first area A1 in the depth direction of the refrigerator 20 (the vertical direction in Figures 3 to 6). The length of the second area A2 in the depth direction of the refrigerator 20 is shorter than the length of the first area A1 in the depth direction, and the length of the second area A2 in the width direction of the refrigerator 20 (the horizontal direction in Figures 3 to 6) is longer than the length of the first area A1 in the width direction. The second area A2 is set in a location where it is expected that the user's hands or arms will not be visible when the user holds an item that they wish to manage inventory and positions the item in the first area A1.

Note that this second area A2 is just one example, and the shape and arrangement of the second area A2 can be determined empirically or experimentally as appropriate by the designer of the inventory management system 10.

As shown in Figure 8 above, the information processing unit 33 has a hand/arm detection unit 38 as a component for determining whether the user's arm or other object is visible in the second area A2. The hand/arm detection unit 38 has a reference image acquisition unit 38a and a reference image difference calculation unit 38b. Below, we will explain how the reference image acquisition unit 38a and the reference image difference calculation unit 38b determine whether an object such as the user's arm is visible in the second area A2 of the video. The reference image acquisition unit 38a acquires the second reference image captured by the imaging unit 31. The second reference image is an image in which no objects other than the refrigerator 20 are visible in the second area A2, as shown in Figure 3.

For example, when the opening/closing sensor 32 detects that at least one of the two doors 22 is open, the reference image acquisition unit 38a causes the photographing unit 31 to photograph a second reference image and acquires the second reference image. In other words, the reference image acquisition unit 38a acquires a new second reference image each time the door 22 is opened. This makes it possible to acquire a second reference image that shows the latest interior of the refrigerator 20.

It is not necessary that the second reference image be acquired when the door 22 is opened; for example, the second reference image may be acquired by the photographing unit 31 when the inventory management device 30 is newly installed (at initial setup), and this reference image may continue to be used thereafter. The second reference image may also be acquired at any timing; for example, the reference image acquired when the door 22 is opened may be updated periodically while the refrigerator door 22 is open. The first reference image may also be used as the second reference image; in this case, acquisition of the second reference image may be omitted.

The reference image difference calculation unit 38b calculates the difference between the pixel values of pixels included in the second region A2 of the second reference image acquired by the reference image acquisition unit 38a and the pixel values of pixels included in the second region A2 of the frame at the current time t. For example, the reference image difference calculation unit 38b calculates the difference between the pixel values of corresponding pixels between the second region A2 of the second reference image and the second region A2 of the frame at the current time t. If the number of pixels included in the second region A2 is x (x is a natural number), x differences are calculated. The reference image difference calculation unit 38b determines that the number of differences greater than a predetermined threshold, y (y is a natural number), of the x differences, is y (y is a natural number). If y/x is equal to or greater than a predetermined ratio, it determines that a user's arm, etc. is present in the second region A2. If y/x is less than the predetermined ratio, it determines that a user's arm, etc. is not present. The predetermined threshold and the predetermined ratio may be determined empirically or experimentally as appropriate.

In this way, the hand/arm detection unit 38 performs image processing on the video captured by the image capture unit 31 to detect whether a hand or arm is visible in the second area A2 of the video. This allows the determination unit 39 to determine that a stationary object other than the refrigerator 20 is an object subject to inventory management when it is detected that a stationary object other than the refrigerator 20 is visible in the first area A1 during a period when a hand or arm is not visible in the second area A2 (specifically, food has stopped flowing in the first area A1). This prevents the erroneous determination that the refrigerator has transitioned from the READY state to the SET state.

[Operation example 1]
Next, a food inventory management operation will be described as a first specific example of the operation of the inventory management system 10. FIG.

When the information processing unit 33 of the inventory management device 30 determines that at least one of the two doors 22 is open based on the detection result of the opening/closing sensor 32 (S11), it activates the imaging unit 31 and starts capturing video using the imaging unit 31 (S12).

Next, the information processing unit 33 performs food detection processing (S13). In the food detection processing, the information processing unit 33 determines whether food is being stored or removed by determining the above four states.

In addition, when the user stops the food above the specific area P and transitions from the SET state to the INOUT state, the information processing unit 33 may use the notification unit 36 to notify the user that the food has been recognized as a target for inventory management. For example, if the notification unit 36 is realized by a speaker, the notification unit 36 outputs a sound or voice message indicating that the food has been recognized as a target for inventory management. If the notification unit 36 is realized by a light-emitting element, the notification unit 36 emits light to notify that the food has been recognized as a target for inventory management.

This allows the user to easily understand that the food has been successfully frozen (the food can now be included in inventory management).

In addition, the information processing unit 33 temporarily stores in the memory unit 34, for example, a frame when transitioning from the SET state to the INOUT state.

Then, when the information processing unit 33 determines that both doors 22 are closed based on the detection results of the open/close sensor 32 (S14), it performs an update process for the food management information (S15). Figure 10 is a diagram showing an example of food management information, in which food IDs and images of the food are stored in association with each other. The food management information is information that lists the foods in the refrigerator 20 at the time of step S11 above, in other words, inventory list information.

In step S15, the information processing unit 33 cuts out the portion of the frame temporarily stored in step S13 in which the food appears (the portion near the first area A1) as an image of the food. When new food is received in steps S11 to S14, the information processing unit 33 assigns an ID to the cut-out image of the food and adds it to the food management information. On the other hand, when food is removed from the food management information in steps S11 to S14, the information processing unit 33 deletes from the food management information the image of the food that is most similar to the cut-out image of the food, out of one or more food images included in the food management information. In this case, the information processing unit 33 performs a process of determining the similarity of the images using various existing algorithms.

Whether food has been stored or removed is determined, for example, in the INOUT state, based on the presence or absence of a user's arm, etc. in a third area in the video (not shown; for example, an area near shelf 24). The determination unit 39 determines that food has been stored if a user's arm, etc. is present in the third area, and determines that food has been removed if a user's arm, etc. is not present in the third area. The method using the reference image described above can be used to determine the presence or absence of a user's arm, etc. in the third area.

As described above, the inventory management system 10 can manage the inventory of food in the refrigerator 20.

In addition, if an error occurs with the imaging unit 31, such as when it is impossible to capture video using the imaging unit 31, the information processing unit 33 may use the notification unit 36 to notify that an error has occurred with the imaging unit 31. For example, if the notification unit 36 is realized by a speaker, the notification unit 36 outputs a voice message or the like indicating that an error has occurred. If the notification unit 36 is realized by a light-emitting element, the notification unit 36 notifies that an error has occurred by emitting light.

In this way, if an error occurs related to the imaging unit 31, the user will be notified and will be able to take measures to deal with the error.

[Operation example 2]
Next, a description will be given of an operation of presenting food stored in the refrigerator 20 as a second operation example of the receiving/delivery management system 10. FIG. 11 is a sequence diagram of the second operation example of the receiving/delivery management system 10.

The communication unit 35 of the inventory management device 30 transmits the food management information to the server device 40 (S21). The communication unit 41 of the server device 40 receives the food management information, and the presentation unit 44 stores the received food management information in the memory unit 43 (S22). The communication unit 35 of the inventory management device 30 transmits the food management information to the server device 40, for example, each time the food management information is updated. This causes the latest food management information to be stored in the memory unit 43 of the server device 40. Note that the food management information may be transmitted from the inventory management device 30 to the server device 40 periodically, such as once at a predetermined time.

On the other hand, when the user performs a specified operation on the information terminal 50 while away from home or elsewhere to check the food stored in the refrigerator 20 (the contents of the food management information), the information terminal 50 accepts this specified operation (S23) and, based on the accepted specified operation, sends a presentation request to the server device 40 (S24).

The communication unit 41 of the server device 40 receives the presentation request. Based on the received presentation request, the presentation unit 44 transmits presentation information to the information terminal 50 to present the contents of the food management information stored in the memory unit 43 (S25). More specifically, the transmission of the presentation information is performed by the communication unit 41.

The information terminal 50 receives the presentation information and, based on the received presentation information, displays a display screen on the display unit of the information terminal 50 showing the contents of the food management information (S26). Figure 12 is a diagram showing an example of a display screen showing the contents of the food management information. On the display screen of Figure 12, an image of the food included in the food management information (an image cut out from a video captured by the image capture unit 31) is superimposed on a background image simulating the contents of the refrigerator 20.

In this way, users of the inventory management system 10 can check the food stored in their home refrigerator 20 while they are away from home.

[Modification]
In the above embodiment, a configuration for suppressing erroneous determination of an item to be subject to inventory management has been described. The inventory management system 10 may also be configured to exclude a food item that has been determined to be a target of inventory management from the target items.

As described above, when a food item is determined to be a target item, the notification unit 36 issues a notification to that effect. At this time, the inventory management system 10 may exclude from the list of target items the food item that has been determined to be a target item for inventory management, on the condition that the user performs a predetermined gesture in the first area A1 (such as holding their hand in a peace sign) within a predetermined period of time after the notification.

[Other Modifications]
In the above embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

Furthermore, refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be implemented as other storage cabinets that do not have refrigeration functions. Furthermore, the items (in other words, objects) stored in the storage cabinet are not limited to food, and may be other items. It is also conceivable that refrigerator 20 may store items other than food.

[Effects, etc.]
Below, examples of techniques that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these techniques will be described.

Technology 1 is an inventory management system 10 that includes a capture unit 31 that captures video of the opening of a storage cabinet viewed from above; a food detection unit 37 that performs image processing on the video to detect whether a stationary object other than the storage cabinet is visible in a first area A1 of the video; a hand/arm detection unit 38 that performs image processing on the video to detect whether a user's hand or arm positioned in front of the opening is visible in a second area A2 of the video; and a determination unit 39 that determines that a stationary object other than the storage cabinet is an object subject to inventory management when a stationary object other than the storage cabinet is detected to be visible in the first area A1 during a period when the user's hand or arm is not visible in the second area A2. The food detection unit 37 is an example of a first detection unit, and the hand/arm detection unit 38 is an example of a second detection unit.

The inventory management system 10 can determine that an item that the user places in the first area A1 is an object of inventory management. In other words, the inventory management system 10 can determine that an item is an object of inventory management based on the user's intentional actions. Furthermore, the second area A2 is set, for example, in a location where the user's hand or arm is not expected to be visible when the user intentionally places an item that the user wants to make an object of inventory management in the first area A1. By setting the second area A2 in this manner, the inventory management system 10 can prevent erroneous determination of an object of inventory management.

Technology 2 is a warehouse management system 10 of Technology 1, in which the hand/arm detection unit 38 acquires a second reference image captured in advance by the photographing unit 31, and detects whether a hand or arm is captured in the second area A2 based on the difference between the pixel values of pixels in the second area A2 in the acquired second reference image and the pixel values of pixels in the second area A2 in the image included in the video.

Such an inventory management system 10 can detect whether a hand or arm is visible in the second area A2 by comparing the second reference image with the current image (frame).

Technology 3 is an inventory management system 10 of Technology 2, in which the hand/arm detection unit 38 can change the second reference image.

Such an inventory management system 10 can change (update) the second reference image.

Technology 4 is a warehouse management system 10 of Technology 3 in which a door 22 is provided at the opening, and the hand/arm detection unit 38 uses the image captured by the photographing unit 31 when the door 22 is opened as a second reference image, and updates the second reference image each time the door 22 is opened.

Such an inventory management system 10 can update the second reference image each time the door 22 is opened.

Technology 5 is an inventory management system 10 according to any one of technologies 1 to 4, in which the food detection unit 37 acquires a first reference image captured in advance by the photographing unit 31, and detects the presence or absence of stationary objects other than storage units in the first area A1 based on the difference between the pixel values of pixels in the first area A1 in the acquired first reference image and the pixel values of pixels in the first area A1 in an image included in the video, and the difference in pixel values between multiple images included in the video.

Such an inventory management system 10 can detect the presence or absence of stationary objects by comparing a first reference image with the current image (frame) and by comparing multiple images (frames) contained in a video.

Technology 6 is an inventory management system 10 of Technology 5, in which the food detection unit 37 can change the first reference image.

Such an inventory management system 10 can change (update) the first reference image.

Technology 7 is an inventory management system 10 of Technology 6 in which a door 22 is provided at the opening, and the food detection unit 37 uses the image captured by the photographing unit 31 when the door 22 is opened as a first reference image, and updates the first reference image each time the door 22 is opened.

Such an inventory management system 10 can update the first reference image each time the door 22 is opened.

Technology 8 is an inventory management system 10 of any of technologies 1 to 7, in which the first area A1 is an area including the inner surface of the storage facility shown in the video.

Such an inventory management system 10 can use the area in the video showing the interior of the storage unit as the first area.

Technology 9 is an inventory management system 10 of any of technologies 1 to 8, in which a specific area P with a predetermined pattern drawn on it is provided on the inner surface of the storage facility shown in the video, and the first area A1 includes the specific area P.

Such specific areas P are useful as markers for the location where the user should place items that they want to manage in inventory.

Technology 10 is an inventory management system 10 of any of technologies 1 to 9, in which the second area A2 is located further back than the first area A1 in the depth direction of the storage facility.

If the second area A2 is set in this manner, when a user holds an item that they want to manage in inventory and positions the item in the first area A1, the user's hand or arm is not likely to be visible in the second area A2. Therefore, the inventory management system 10 can prevent erroneous determination of the item to be managed in inventory.

Technology 11 is an inventory management system 10 of any of technologies 1 to 10, in which the length of the second area A2 in the depth direction of the storage facility is shorter than the length of the first area A1 in the depth direction.

Such an inventory management system 10 can prevent incorrect identification of inventory management objects based on the second area A2, which has a shorter depth length than the first area A1.

Technology 12 is an inventory management system 10 of any of technologies 1 to 11, in which the widthwise length of the storage facility in the second area A2 is longer than the widthwise length of the first area A1.

Such an inventory management system 10 can prevent incorrect determination of inventory management objects based on the second area A2, which has a longer width than the first area A1.

Technology 13 is a warehousing/receiving management system 10 of any of technologies 1 to 12, further comprising an alarm unit 36 that notifies when an error occurs related to the photographing unit 31.

Such an inventory management system 10 can notify the user that an error has occurred related to the photographing unit 31.

Technology 14 is an inventory management system 10 of any of technologies 1 to 13, in which the storage facility is a refrigerator 20.

The inventory management system 10 can determine whether an item is subject to inventory management in the refrigerator 20.

Technology 15 is a computer-implemented inventory management method. The inventory management method includes a photographing step of photographing a video of the opening of a storage unit viewed from above, a first detection step of performing image processing on the video to detect whether a stationary object other than the storage unit is captured in a first area A1 of the video, a second detection step of performing image processing on the video to detect whether a user's hand or arm positioned in front of the opening is captured in a second area A2 of the video, and a determination step of determining that a stationary object other than the storage unit is an object of inventory management when a stationary object other than the storage unit is detected to be captured in the first area A1 during a period when the user's hand or arm is not captured in the second area A2.

This type of inventory management method can determine that an item is subject to inventory management based on the user's intentional actions.

Technology 16 is a program for causing a computer to execute the inventory management method described in Technology 15.

Such programs can determine that an item is subject to inventory management based on the user's intentional actions.

(Other embodiments)
Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

For example, in the above embodiment, the inventory management system is realized by multiple devices, but it may also be realized by a single device. For example, the inventory management system may be realized as a single device equivalent to the inventory management device of the above embodiment, or as a single device equivalent to a server device. When the inventory management system is realized by multiple devices, the components (especially functional components) of the inventory management system may be allocated in any way to the multiple devices. For example, some or all of the processing described in the above embodiment as being executed by the inventory management device may be executed by a server device.

Furthermore, for example, the communication method between devices in the above embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between devices. Furthermore, the information transmission path described in the above embodiment is not limited to the transmission path shown in the sequence diagram.

For example, in the above embodiment, the processing performed by a specific processing unit may be performed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

Furthermore, in the above-described embodiments, each component may be realized by executing a software program appropriate for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

In addition, general or specific aspects of the present disclosure may be realized as a system, device, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM. In addition, general or specific aspects of the present disclosure may be realized as any combination of a system, device, method, integrated circuit, computer program, and recording medium.

For example, the present disclosure may be realized as a method for managing the inflow and outflow of objects executed by a computer, or as a program for causing a computer to execute the method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of this disclosure.

The inventory management system disclosed herein is useful as a device that can manage items in storage facilities.

10. Inventory management system 20. Refrigerator (storage)
21 Main body 22 Door 23 Storage room 24 Shelf 30 Intake/out management device 31 Photography unit 32 Open/close sensor 33, 42 Information processing unit 34, 43 Memory unit 35, 41 Communication unit 36 Notification unit 37 Food detection unit (first detection unit)
38 Hand/arm detection unit (second detection unit)
39 Determination unit 40 Server device 44 Presentation unit 50 Information terminal 60 Wide area communication network

Claims (16)

a camera unit that captures a video of the opening of the storage cabinet as seen from above;
a first detection unit that performs image processing on the video to detect whether a stationary object other than the storage unit is captured in a first area of the video;
a second detection unit that performs image processing on the video to detect whether a hand or arm of a user positioned in front of the opening is captured in a second region of the video;
and a determination unit that determines that a stationary object other than the storage unit is an object to be managed in inventory when a stationary object other than the storage unit is detected to be captured in the first area during a period when the user's hand or arm is not captured in the second area. The second detection unit
acquiring a second reference image captured in advance by the imaging unit;
2. The inventory management system according to claim 1, wherein whether a hand or arm is present in the second area is detected based on the difference between the pixel values of pixels in the second area in the acquired second reference image and the pixel values of pixels in the second area in the image included in the video. The inventory management system according to claim 2 , wherein the second detection unit is capable of changing the second reference image. A door is provided at the opening,
The inventory management system according to claim 3, wherein the second detection unit uses an image captured by the photographing unit when the door is opened as the second reference image, and updates the second reference image every time the door is opened. The first detection unit
acquiring a first reference image captured in advance by the imaging unit;
2. The inventory management system of claim 1, which detects the presence or absence of stationary objects other than the storage unit in the first area based on the difference between the pixel values of pixels in the first area in the acquired first reference image and the pixel values of pixels in the first area in the image included in the video, and the difference in pixel values between multiple images included in the video. The inventory management system according to claim 5 , wherein the first detection unit is capable of changing the first reference image. A door is provided at the opening,
The inventory management system according to claim 6, wherein the first detection unit uses an image captured by the photographing unit when the door is opened as the first reference image, and updates the first reference image every time the door is opened. The inventory management system according to any one of claims 1 to 7, wherein the first area is an area including the inner surface of the storage unit shown in the video. A specific area on which a predetermined pattern is drawn is provided on the inner surface of the storage cabinet shown in the video,
The inventory management system according to claim 8 , wherein the first area includes the specific area. The inventory management system according to any one of claims 1 to 7, wherein the second area is located further back than the first area in a depth direction of the storage shed. The inventory management system according to any one of claims 1 to 7, wherein the length of the second area in the depth direction of the storage is shorter than the length of the first area in the depth direction. The inventory management system according to any one of claims 1 to 7, wherein the length of the second area in the width direction of the storage cabinet is longer than the length of the first area in the width direction. The inventory management system according to any one of claims 1 to 7, further comprising a notification unit that notifies that an error related to the photographing unit has occurred. The inventory management system according to any one of claims 1 to 7, wherein the storage is a refrigerator. A computer-implemented inventory management method, comprising:
a photographing step of photographing a video of the opening of the storage shed viewed from above;
a first detection step of detecting whether a stationary object other than the storage unit is captured in a first region of the video by performing image processing on the video;
a second detection step of detecting whether a hand or arm of a user positioned in front of the opening is captured in a second region of the video by performing image processing on the video;
and a determining step of determining that a stationary object other than the storage cabinet is an object to be managed in inventory, provided that the stationary object is detected to be captured in the first area during a period in which the user's hand or arm is not captured in the second area. A program for causing the computer to execute the inventory management method described in claim 15.